[Twisted-Python] Re: Memory size of Deferreds
Martin Geisler
mg at daimi.au.dk
Wed May 21 01:56:30 MDT 2008
MichaĆ Pasternak <michal.dtz at gmail.com> writes:
>> def add(x, y):
>> sum = gatherResults([x, y])
>> sum.addCallback(lambda results: results[0] + results[1])
>> return sum
>>
>> Multiplication is similar, but requires network communication --
>> that is why it returns a Deferred.
>
> I don't think that there's anything wrong with this example in terms
> of being written using Twisted Python.
Okay, thanks for looking at it!
>> The important point is that as much as possible is done in
>> parallel, that all operations run as soon as their operands are
>> available. Twisted made this extremely easy thanks to the
>> DeferredList and Deferred abstraction.
>
> This is the point where I think, that you answered your question. If
> you do stuff in parallel, you use more Deferreds and more memory.
Yes, I guess you are right... :-) The design tries to do everything at
once with as much parallism as possible. But since bandwidth is
limited, I guess I could achieve the same throughput by scheduling
things in a slower rate. Maybe using something like this:
class map_parallel(Deferred):
def __init__(self, iterable, func, max):
Deferred.__init__(self)
self.iter = iterable
self.func = func
self.running = 0
for _ in range(max):
self.running += 1
self.run_next(None)
def run_next(self, arg):
try:
e = self.iter.next()
e.addCallback(self.func)
e.addCallback(self.run_next)
except StopIteration:
# The iterator is exhausted.
self.running -= 1
if self.running == 0 and not self.called:
# All callbacks have finished.
self.callback(None)
# Pass the argument through unchanged.
return arg
where map_parallel will map a function over a list of Deferreds, but
it will only run up to max function applications in parallel. I will
experiment with it, and if it works well, then it would be cool to
have a class like this in Twisted.
> I will repeat myself now: I would rather try to find other places to
> optimize VIFF, than reducing memory footprint of Deferred. I don't
> have a CS degree, I'm not a mathematician, but from what you wrote
> it seems that maybe it would be worth to try implementing something
> stack-based (like RPN), than the tree approach you seem to be doing
> now - of course if having a stack is something you can have using
> secure multiparty calculations.
Thanks for the ideas! You could certainly do what you suggest,
changing the schedule like that is okay from a security point of view.
And if I can delay allocating memory until needed by doing so, then it
will definitely be interesting to try.
--
Martin Geisler
VIFF (Virtual Ideal Functionality Framework) brings easy and efficient
SMPC (Secure Multi-Party Computation) to Python. See: http://viff.dk/.
More information about the Twisted-Python
mailing list