[Twisted-Python] Re: In Defense of Taps

[Glyph Lefkowitz]

On 12 Feb 2003 15:43:25 -0000, Moshe Zadka <m at moshez.org> wrote:
> >  1. Not very many mature projects use Twisted yet.
> >  2. Everybody hates TAPs unless I've personally explained why they shouldn't.

> We should at least take those results as far as our single best opportunity
> to do advocation: PyCon in general, and Itamar's tutorial in particular.  I
> think we all believe that (at the very least) twisted.internet is ready for
> production-code. We all believe that TAPs are useful. I think we should take
> the opportunity in all talks to emphasise how useful these things are.

This is a good opportunity, yes.  Thanks to Steve's post, I am less
discouraged.  For technical reasons, it is easier for us to make this pitch to
the people interested in twisted.internet than in other areas of Twisted
though: more below.

> Also, I'd attempt to take another conclusion from it: people used to complain
> Twisted is not documented enough, but I think now the complaint is that
> people are not sure where to start from.

This is a serious problem both on the website and in the documentation.  There
is also the issue of whether certain kinds of developers can even depend upon
Twisted to do what they need to do.  We have annotations in the docstrings for
various modules, but there is no at-a-glance reference card for what modules
within Twisted are really up to snuff.

The first thing is to decide what the classes of people who will be looking at
Twisted are.  Here, I think you've correctly identified two of them: web
developers and level network protocol programmers.  What are the other groups?
Simulation programmers?  Game programmers?  Corporate sysadmins?  Workflow
designers?  Interactive agent programmers?  (This is a question to the mailing
list at large: who _are_ you people, anyway?  There are almost 300 people
subscribed, and we've only got 5 users listed on the website!)

> Part of it is a real problem: the skills a Woven developer should have are
> (almost) complete disjoint than the skills a programmer imlpementing a new
> kind of server (say, SFS):
                  ^^^^^^^^

I would like to emphasize this extremely subtle hint in case you didn't get it
the first time, liiwi :-)

> the first would just use a quick mktap web --path/twisted-web package to set
> up the server, and start writing .rpy's, while the later would spend most of
> his time minding the protocol, and only ending with writing an mktap plugin.

I think that your conclusion is correct, but there is another, technical
problem here.  We have a plugin type which we suggest network programmers to
use (and the flaws of this plugin type have been previously discussed).  RPYs
are a partial solution, and Donovan and I have been back and forth over better
and worse ways to structure applications.  This still hasn't completely
settled.  It bothers me that RPYs are so different from the TAP plugin
architecture, and wedded to a filesystem hierarchy that has nothing to do with
your Python code.  It's still really challenging (though by no means
impossible) to take any 2 existing Twisted web apps and put them together in
the same server.

In the absence of the COIL uber-plugin API that we are eventually going to
need, perhaps there should be some more work in the near term going into
defining domain-specific plugin APIs for different aspects of Twisted?  I
avoided this for a long time because I didn't want to proliferate more bad
integration schemes, but by now it's probably causing more trouble than it's
worth.  At least, I think we could easily come up with a half-solution that's
better than the current half-solution.

twisted.web would be the most obvious area to do this first.  While we're at
it, we could clean up the API for bot creation in twisted.words and make it
compatible to some extent with twisted.im.  (Donovan and Kevin respectively,
I'd like your opinions here...)

> What is my conclusion? I consider only the second programmer "primary Twisted
> audience": not specifically web programmers, but network programmers. For
> network programmers, the following HOWTOs are useful: [snip]

That's a good index for network programmers, and I think it mirrors the way the
book has been structured so far.  I think there is also a need to educate web
programmers about the networking aspects of web stuff within Twisted, and how
it can be used to integrate externally.  This will become even more of an issue
as the LivePage stuff that Donovan is working on right now matures: Woven will
be able to change the way that people think about "web apps".

> It would be nice if we could make special indexes for special kinds
> of programmers, pointing them to a specific reading of the documentation.
> How to do it is left as an exercise to the reader (but I will note that
> on the TeX side, we're pretty much covered so it really more of a question
> of decisions than of technology).

As we try to construct these different paths, let's not leave out documents,
but rather re-order them so that different interest groups will be exposed to
things they might not have considered, but those different areas of interest
will be small detours that diverge from their main path, and not obstacles to
their progress.

> I encourage the "newbies" on the list to speak up: what are *you* planning
> to do with Twisted, and which documents would help you with that.

Yes.  A lot of the Twisted development discussion is ephemeral and
experimental, and therefore takes place over IRC, so those of you out there in
the user community may have gotten the impression that us developers don't like
getting into extended discussions in e-mail.  I hope these last few days have
disabused you of this notion :-).

I've said this before in different ways, and I'm sure I'll say it again: If
you're writing applications for Twisted, it is in your best interest to help us
grow and enhance the platform.  One of the primary ways you can do that is
through _feedback_.  I am consistently surprised by how popular this project is
and how many people have heard of it, because there is often so little
commentary on what users want or need.

That said, if you give us feedback that takes a substantial amount of time to
implement, don't expect immediate results :-).  Designing a new system,
especially one as general as what I've been talking about the last few weeks (a
general configuration dashboard and plugin API for all of Twisted) is
challenging and takes a while.  Still, it is made all the more difficult for
having to guess what people actually want and use Twisted for: it is now a
framework with applications in domains where none of the core developers have
any experience.

-- 
 |    <`'>    |  Glyph Lefkowitz: Traveling Sorcerer   |
 |   < _/ >   |  Lead Developer,  the Twisted project  |
 |  < ___/ >  |      http://www.twistedmatrix.com      |
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: </pipermail/twisted-python/attachments/20030212/4a22f6e7/attachment.sig>