[Twisted-Python] Questionaire: How twisted are you ? (Was: Evangelism notes...)

[Robin Bryce]

In the spirit of just having a go and testing out my twisted knowledge 
on this list I thought I'd start a questionaire, with answers, inspired by:

> I would love to see high level conceptual documentation to give me a
> better idea of  basic concepts  like the "reactor", "application",
> "service", "transport","protocol", "factory" - what they are supposed
> to do and how they are  to fit together.

I am by no stretch of the imagination qualified to advertise these 
answers as authoratative. Buyer beware, and if you don't like mine give 
me a fat -ve 10 for my twisted score and show me your answer ;-)

Authoritative intro material is at:

http://twistedmatrix.com/projects/core/documentation/howto/basics.html
http://twistedmatrix.com/projects/core/documentation/howto/glossary.html

Q1. How do you run a twisted program ?

In as many ways as there are pebbles on a beach ;-)

There are lots of great examples in the docs that get you to the point 
where you have a python file that will create an application instance at 
global scope. Once you've got this far the simplest way to run your 
twisted program is to add the following to that file:

from twisted.internet import reactor
reactor.run()

And then at the command prompt

python myapplication.py

Once you're comfortable with the basics of Services, the Application, 
Factory's and Protocols you're going to start caring about things like 
persistence, daemonization, privilege scheding, plugins etc. At that 
point it's time to look at 'twistd'. If you remove the lines from 
myapplication.py that were added above then you can run your application 
directly from the prompt with:

twistd -oy myapplication.py

Then you will get the almost the same result. The primary difference is 
that your now running your program as a daemon.

twistd is a program that loads twisted applications from many different 
formats, including .py, and starts the application instance it finds there.

Q2. What is an application ?

Most twisted programs begin with an application instance. Twisted 
defines a helper function for creating an application instance so that 
it includes all the bits of functionality that allow it to be driven by 
the twisted framework. This function is found in the module 
twisted.application.service and is called 'Application'.

Most twisted programs define a python file that makes a call to the 
'Application' function at global scope. Just doing this, and nothing 
else, will not get you very interesting results, for that you need to 
look at adding 'Services'.

If you've already got a python file that defines an application instance 
this way - in addition to some example code - and you just want to know 
the simplest way to run your program, then the answer you're after is 
almost certainly:

# add these lines to myprogram.py
from twisted.internet import reactor
reactor.run()

$ python myprogram.py

But see Q1. for a little more on this.

This is the most suitable way if you want to single step through your 
program in a debugger to find out how twisted works. There are many 
other ways to run your program but it's safe to ignore them until you 
decide you want them.

Q3. What's a Service ?

'Services' collect together the bits that make it possible for cool 
networking stuff to happen in a single thread without blocking[1]. You 
don't have to use them to use the twisted framework but they can help. 
If you are using an application instance then you almost certainly want 
to use Services as well.

You add Service's to your application instance (see Q2 for creating 
one). Each service typically manages a discrete network service. The 
twisted framework starts and stops all Services in an application in a 
well defined way when your program is executed. This includes provision 
for giving your services a chance to do stuff before privilege scheding 
happens.

To define what your service is you will usually want to subclass 
twisted.application.service.Service and override any, or all of:

privilegedStartService, startService, stopService.

During a single run of your program each of these methods is called 
exactly once and in exactly that order. If you need to bind to 
privileged ports then privilegedStartService is where you need to do it.

The bits that are used to piece together Service behavior are 
principally: Factory's, Protocols, and Defereds. The parameters involved 
in binding these bits together are things like hostnames, portnumbers, 
database passwords, and (arguably) the networking protocol the service 
is dealing with. Collecting these things together into a service can be 
done by hand, often, it can be easier, more flexible, and much more 
extensible if Adapters are used to do the binding.

[1] Note carefully the word 'possible' in relation to 'without 
blocking'. The twisted framework can not guarantee your program will not 
block. Twisted just gives you a set of concurrency primitives to help 
you write your program so it doesn't block. To understand how to write a 
twisted program so it never blocks you need to learn to love 'Defered's. 
Concurrency primitives in python essentially boil down to a choice 
between callbacks or generators. The twisted framework supports both (as 
of 2.0) but the callback style is the most established. Generators 
didn't arrive   in python until 2.2.

Q4. What's a MultiService ?

At some point you will almost certainly decide that subclassing 
MultiService is going to help you. It's not. To quote moshez:

"duuuuuude. don't. inherit. multiservice."

[And certainly don't inherit from both MultiService and Componentized at 
the same time ;-)]

MultiService is an internal class that is used by the framework to 
manage the order in which all services, added to the application 
instance by you, are started and stoped. The fraze "added to the 
application instance" is potentially misleading but from the birds eye 
perspective this is essentially what you do when you add Services.

You almost certainly do want to subclass Service. But don't be fooled 
into subclassing MultiService.

Q4 What's a Factory ?

Q5 What's a Protocol ?

Q6 How do I get my Factory to open a connection ?

Q7. How do I get my Factory to serve a connection ?

Q8. How do I cancel my attempt to open a connection ?

Q9. What's a plugin ?

Q10. Ok, I'm feeling brave, what is an Application realy and why can't 
it be subclassable ?

You can't subclass it because it is a function. But that doesn't tell 
you anything useful.

The value returned by the function 
twisted.application.service.Appliction is an instance of 
twisted.python.components.Componentized.

Componentized is a class that provides interface aggregation facilities 
in a persistable way. When you call Application you are creating an 
instance of Componentized that aggregates instances of the following 
together:

* twisted.application.service.MultiService
* twisted.application.service.Process
* twisted.persisted.sob.Persitent

The twisted framework uses interfaces and adapters to reduce the 
tendency for functional dependencies to snarl up the implementation of 
the framework. To do this it needs your application to support interface 
aggregation and to provide the basic interfaces that enable it to drive 
your program the Twisted way.

In these terms the function Application produces an instance that 
supports the interfaces IService, IMultiService, IPersitable and 
IProcess. MultiService implements IService and IMultiService[1], 
Persistent implements IPersistable, and Process implements IProcess.

These are the basic guarantees about the application instance that 
enable the twisted framework to drive your application.

You can't specialize of the above instances either - unless you are 
prepared to mimic the behavior of Application. You're only route for 
specializing your application is by adding services. ie., via some 
variation of:

IService(MySerivce()).setServiceParent(IMultiService(application))


It's not unusual or unreasonable to write a replacement for Application. 
A perfectly acceptable example is:

def MyApplication():

       a = MyService()
       a.setName("myapplication")
       ret = service.MultiService()
       a.setServiceParent(ret)

       return ret

Reading up on adapters & interfaces in the python / Zope3 sense can give 
you an extra perspective when looking at the twisted docs on Adapters 
and interfaces. Twisted 2.0 actually uses Zope3 interfaces. And if 
you're reading this answer then there is a good chance taking the time 
to do this will help you.

http://www.python.org/peps/pep-0246.html
http://mail.zope.org/pipermail/zope3-dev/2005-January/013064.html

[1]It is required that the IService interface and the IMultiService 
interface are implemented by the same underlying object (or at least it 
is by the application unit tests).

You really probably don't want to read any more of this answer. For one 
thing Componentized looks like it's going to be superseded by Faceted 
which does similar things but much much better.

 From the doc string of Componentized:

"I am a mixin to allow you to be adapted in various ways persistently."

Componentized caches addaptions in a persitable way. Your applications 
root service, the MultiService instance above is effectively a 
persistably cached adaption from None to IMultiService and to IService.

Well you did ask ;->

When you do:

IService(application)

You are retreiving the IService interface for you application that 
happens to be implemented by twisted.application.service.MultiService

Depending on the feedback I'm keen to see/provide answers to Q4-Q8, 
Don't feel confident at all about Q9.

Cheers,

Robin

Mary Gardiner wrote:
> On Sun, May 08, 2005, Eugene Coetzee wrote:
> 
>>It certainly helps, although it may be a bit minimalistic. 
> 
> 
> A review would be appreciated :) Also, any suggestions on how to make
> the glossary easier to find? 
> 
> -Mary
> 
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python at twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python