[Twisted-Python] __slots__ help (Re: Memory size of Deferreds)
Marcin Kasperski
Marcin.Kasperski at softax.com.pl
Tue May 20 07:44:59 MDT 2008
It is not that difficult to see what is inside the deferred (spawn
some debugger and see):
(this may be a class variable)
debug (boolean)
(those surely are instance variables)
callbacks (a list, initially empty, later containing tuples of tuples)
called (int)
paused (int)
result (anything)
timeoutCall (None or object)
_runningCallbacks (boolean)
_debugInfo (object of type DebugInfo)
Bare names of attributes take about 60 bytes (this is what could be
reclamied with __slots__), some of them are references, we have a list
(which inside contains complicated structures), ....
It could be an interesting experiment to define __slots__ for Deferred
class and see what is the memory impact.
Hmm, hmm. I did an experiment. Here is one version of the testing code:
Cls = Deferred
count = 250000
import os
pid = os.getpid()
cmd = "ps -F %d" % pid
a = [Cls() for _ in xrange(count)]
os.system(cmd)
b = [Cls() for _ in xrange(count)]
os.system(cmd)
c = [Cls() for _ in xrange(count)]
os.system(cmd)
and here is one using __slots__:
from twisted.internet.defer import timeout, Deferred
class SlotsDeferred(object):
__slots__ = ['debug', 'callbacks', 'called', 'paused', 'result', 'timeoutCall', '_runningCallbacks', '_debugInfo']
# whole content of original Deferred class copied as-is here
Cls = SlotsDeferred
count = 250000
import os
pid = os.getpid()
cmd = "ps -F %d" % pid
a = [Cls() for _ in xrange(count)]
os.system(cmd)
b = [Cls() for _ in xrange(count)]
os.system(cmd)
c = [Cls() for _ in xrange(count)]
os.system(cmd)
And here are the results:
(standard Deferred)
$ python testdef.py
UID PID PPID C SZ RSS PSR STIME TTY STAT TIME CMD
marcink 23109 20536 92 15180 56992 1 15:40 pts/18 S+ 0:00 python testdef.py
UID PID PPID C SZ RSS PSR STIME TTY STAT TIME CMD
marcink 23109 20536 99 28374 109000 1 15:40 pts/18 S+ 0:02 python testdef.py
UID PID PPID C SZ RSS PSR STIME TTY STAT TIME CMD
marcink 23109 20536 99 41568 161020 1 15:40 pts/18 S+ 0:05 python testdef.py
(Deferred with __slots__ added)
$ python testdef.py
UID PID PPID C SZ RSS PSR STIME TTY STAT TIME CMD
marcink 23127 20536 0 7834 28068 1 15:41 pts/18 S+ 0:00 python testdef.py
UID PID PPID C SZ RSS PSR STIME TTY STAT TIME CMD
marcink 23127 20536 99 13683 51160 0 15:41 pts/18 S+ 0:01 python testdef.py
UID PID PPID C SZ RSS PSR STIME TTY STAT TIME CMD
marcink 23127 20536 99 19532 74256 1 15:41 pts/18 S+ 0:02 python testdef.py
As one can see, the memory usage is ~ halved.
So ... maybe it would make sense to add __slots__ to deferreds, after all?
--
----------------------------------------------------------------------
| Marcin Kasperski | In any large change, 1/3 will think it is
| http://mekk.waw.pl | great, 1/3 will think it is stupid, and 1/3
| | will wait (Reed)
----------------------------------------------------------------------
More information about the Twisted-Python
mailing list