[Twisted-web] making streams optional in web2

James Y Knight foom at fuhm.net
Thu Feb 9 11:40:49 MST 2006 

On Feb 9, 2006, at 6:21 AM, glyph at divmod.com wrote:
> A few days ago, JP showed me some benchmarks of web2 vs. CherryPy  
> hosting a trivial WSGI application.
>
> It turns out it's 4x slower, clocking in only ~250 req/sec on a  
> middle-spec Opteron, as opposed to CherryPy's ~1000.  From previous  
> benchmarks, I happen to know that Apache can do ~3000 req/sec on  
> that same hardware, making web2 well over 10x slower.
>
> I didn't look at the profile too closely, but preliminarily it  
> seemed clear that the major bottleneck at this point was the  
> creation and invocation of tons of Deferreds in  
> _NotifyingProducerStream, while generating the response.
>
> Considering that streams aren't used at all in the actual HTTP  
> protocol implementation (and thank goodness for that, given this  
> discovery) it strikes me that the use of them in the response- 
> generating API ought to be more optional, for cases where  
> efficiency is important.  Like, for example, DAV, which also seems  
> to be making embarrassingly heavy use of the streams API - I  
> haven't benchmarked that yet, but I'm sure it will be an adventure  
> if I do :).

Benchmarking is a dangerous activity.

I only get 131req/sec from ab on my mac from the following IResource  
resource, vs 114 from the wsgi resource, vs 706 from apache. Vs 74  
from cherrypy's tut01_helloworld.py. So, shrug, I'm sure my benchmark  
sucks, and must be measuring something completely different than  
yours, given the wide variance in relative numbers. Anyhow, I'm happy  
that someone is benchmarking web2, because I haven't done so any time  
recently, but what you write above comes quite close to random flaming.

Here's the simple resources:

def simple_wsgi_app(environ, start_response):
     start_response("200 OK", [])
     return ['Hello world!']

class SimpleResource(object):
     implements(iweb.IResource)
     def renderHTTP(self, req):
         return http.Response(200, None,  "Hello world!")
     def locateChild(self, req, seg):
         return self, ()

Here's all the construction of Deferreds from the native response.  
Only one is from the streams module. Looks like some of the others  
could be removed without changing any APIs, if that actually helps  
things. Going through the WSGI interface adds one additional deferred  
construction in addition to these 4.

(Pdb) break twisted.internet.defer.Deferred.__init__
Breakpoint 1 at /Users/jknight/Tw/trunk/twisted/internet/defer.py:163

(Pdb) bt
   /Users/jknight/Tw/trunk/twisted/internet/selectreactor.py(139) 
_doReadOrWrite()
-> why = getattr(selectable, method)()
   /Users/jknight/Tw/trunk/twisted/internet/tcp.py(350)doRead()
-> return self.protocol.dataReceived(data)
   /Users/jknight/Tw/trunk/twisted/protocols/basic.py(232)dataReceived()
-> why = self.lineReceived(line)
   /Users/jknight/Tw/trunk/twisted/web2/channel/http.py(664) 
lineReceived()
-> self.chanRequest.lineReceived(line)
   /Users/jknight/Tw/trunk/twisted/web2/channel/http.py(142) 
lineReceived()
-> self.processRequest()
   /Users/jknight/Tw/trunk/twisted/web2/channel/http.py(384) 
processRequest()
-> self.request.process()
   /Users/jknight/Tw/trunk/twisted/web2/server.py(256)process()
-> d = self._getChild(self.site.resource, self.postpath)
   /Users/jknight/Tw/trunk/twisted/web2/server.py(283)_getChild()
-> return defer.maybeDeferred(
   /Users/jknight/Tw/trunk/twisted/internet/defer.py(116)maybeDeferred()
-> return succeed(result)
   /Users/jknight/Tw/trunk/twisted/internet/defer.py(49)succeed()
-> d = Deferred()
 > /Users/jknight/Tw/trunk/twisted/internet/defer.py(163)__init__()
-> self.callbacks = []

(Pdb) bt
   /Users/jknight/Tw/trunk/twisted/internet/selectreactor.py(139) 
_doReadOrWrite()
-> why = getattr(selectable, method)()
   /Users/jknight/Tw/trunk/twisted/internet/tcp.py(350)doRead()
-> return self.protocol.dataReceived(data)
   /Users/jknight/Tw/trunk/twisted/protocols/basic.py(232)dataReceived()
-> why = self.lineReceived(line)
   /Users/jknight/Tw/trunk/twisted/web2/channel/http.py(664) 
lineReceived()
-> self.chanRequest.lineReceived(line)
   /Users/jknight/Tw/trunk/twisted/web2/channel/http.py(142) 
lineReceived()
-> self.processRequest()
   /Users/jknight/Tw/trunk/twisted/web2/channel/http.py(384) 
processRequest()
-> self.request.process()
   /Users/jknight/Tw/trunk/twisted/web2/server.py(256)process()
-> d = self._getChild(self.site.resource, self.postpath)
   /Users/jknight/Tw/trunk/twisted/web2/server.py(283)_getChild()
-> return defer.maybeDeferred(
   /Users/jknight/Tw/trunk/twisted/internet/defer.py(191)addCallback()
-> callbackKeywords=kw)
   /Users/jknight/Tw/trunk/twisted/internet/defer.py(182)addCallbacks()
-> self._runCallbacks()
   /Users/jknight/Tw/trunk/twisted/internet/defer.py(307)_runCallbacks()
-> self.result = callback(self.result, *args, **kw)
   /Users/jknight/Tw/trunk/twisted/web2/server.py(312)_handleSegment()
-> return self._getChild(newres, newpath)
   /Users/jknight/Tw/trunk/twisted/web2/server.py(281)_getChild()
-> return defer.succeed(res)
   /Users/jknight/Tw/trunk/twisted/internet/defer.py(49)succeed()
-> d = Deferred()
 > /Users/jknight/Tw/trunk/twisted/internet/defer.py(163)__init__()
-> self.callbacks = []

(Pdb) bt
   /Users/jknight/Tw/trunk/twisted/internet/selectreactor.py(139) 
_doReadOrWrite()
-> why = getattr(selectable, method)()
   /Users/jknight/Tw/trunk/twisted/internet/tcp.py(350)doRead()
-> return self.protocol.dataReceived(data)
   /Users/jknight/Tw/trunk/twisted/protocols/basic.py(232)dataReceived()
-> why = self.lineReceived(line)
   /Users/jknight/Tw/trunk/twisted/web2/channel/http.py(664) 
lineReceived()
-> self.chanRequest.lineReceived(line)
   /Users/jknight/Tw/trunk/twisted/web2/channel/http.py(142) 
lineReceived()
-> self.processRequest()
   /Users/jknight/Tw/trunk/twisted/web2/channel/http.py(384) 
processRequest()
-> self.request.process()
   /Users/jknight/Tw/trunk/twisted/web2/server.py(258)process()
-> d.addCallback(self._cbFinishRender)
   /Users/jknight/Tw/trunk/twisted/internet/defer.py(191)addCallback()
-> callbackKeywords=kw)
   /Users/jknight/Tw/trunk/twisted/internet/defer.py(182)addCallbacks()
-> self._runCallbacks()
   /Users/jknight/Tw/trunk/twisted/internet/defer.py(307)_runCallbacks()
-> self.result = callback(self.result, *args, **kw)
   /Users/jknight/Tw/trunk/twisted/web2/server.py(355)_cbFinishRender()
-> d = defer.Deferred()
 > /Users/jknight/Tw/trunk/twisted/internet/defer.py(163)__init__()
-> self.callbacks = []

(Pdb) bt
   /Users/jknight/Tw/trunk/twisted/internet/selectreactor.py(139) 
_doReadOrWrite()
-> why = getattr(selectable, method)()
   /Users/jknight/Tw/trunk/twisted/internet/tcp.py(350)doRead()
-> return self.protocol.dataReceived(data)
   /Users/jknight/Tw/trunk/twisted/protocols/basic.py(232)dataReceived()
-> why = self.lineReceived(line)
   /Users/jknight/Tw/trunk/twisted/web2/channel/http.py(664) 
lineReceived()
-> self.chanRequest.lineReceived(line)
   /Users/jknight/Tw/trunk/twisted/web2/channel/http.py(142) 
lineReceived()
-> self.processRequest()
   /Users/jknight/Tw/trunk/twisted/web2/channel/http.py(384) 
processRequest()
-> self.request.process()
   /Users/jknight/Tw/trunk/twisted/web2/server.py(258)process()
-> d.addCallback(self._cbFinishRender)
   /Users/jknight/Tw/trunk/twisted/internet/defer.py(191)addCallback()
-> callbackKeywords=kw)
   /Users/jknight/Tw/trunk/twisted/internet/defer.py(182)addCallbacks()
-> self._runCallbacks()
   /Users/jknight/Tw/trunk/twisted/internet/defer.py(307)_runCallbacks()
-> self.result = callback(self.result, *args, **kw)
   /Users/jknight/Tw/trunk/twisted/web2/server.py(359)_cbFinishRender()
-> d.callback(response)
   /Users/jknight/Tw/trunk/twisted/internet/defer.py(229)callback()
-> self._startRunCallbacks(result)
   /Users/jknight/Tw/trunk/twisted/internet/defer.py(294) 
_startRunCallbacks()
-> self._runCallbacks()
   /Users/jknight/Tw/trunk/twisted/internet/defer.py(307)_runCallbacks()
-> self.result = callback(self.result, *args, **kw)
   /Users/jknight/Tw/trunk/twisted/web2/http.py(419)writeResponse()
-> d = stream.StreamProducer(response.stream).beginProducing 
(self.chanRequest)
   /Users/jknight/Tw/trunk/twisted/web2/stream.py(725)beginProducing()
-> finishedCallback = self.finishedCallback = Deferred()
 > /Users/jknight/Tw/trunk/twisted/internet/defer.py(163)__init__()
-> self.callbacks = []



James

More information about the Twisted-web mailing list