[Twisted-web] Some help with Nevow and databse I/O

Valentino Volonghi aka Dialtone dialtone at divmod.com
Fri Feb 24 13:29:23 MST 2006


On Fri, 24 Feb 2006 10:11:21 -0800, Eric Gaumer <egaumer at pagecache.org> wrote:
>Hello everyone,

Hi

>I have two questions that are somewhat interrelated. In my test cases I have
>authentication (using nevow session guard) setup using a SHA stored via the
>shelve module.

This is technically wrong.

Let me explain why:
Yes, you are using nevow session guard to have authentication but no, it is not guard that is setup to use a SHA stored via the shelve module. The thing that is setup in that way is twisted.cred.

In fact guard is only glue that is used to get credentials from a request and pass them to twisted.cred.

Why do I tell you this? Because it means that you don't have to change your application in order to change the source of authentication data.

And yet people say that cred/guard is crappy... Oh man. :)

>The true customer info is stored in a MySQL database. I can pull that data
>doing the following:
>
>    def data_query(self, context, data):
>        return self.dbpool.runQuery('SELECT name, email, id, TransferType,
>		basic_charge, over_charge, permitted_transfer FROM Customers')
>
>I've read Abe's book on twisted and on page 54 he states "Nevow is designed
>from the ground up for Twisted, which means you can use Deferreds everywhere"

>My question is do I have to use the adbapi module or can I use deferreds to
>handle database queries?

adbapi module returns deferreds for each of the 3 methods:

DBPool.runQuery
DBPool.runOperation
DBPool.runInteraction

>I've read the docs on deferreds but I'd be lying if I said I can fully wrap my
>brain around all aspects of the concept or how it's implemented.

class Deferred(list):
    def addCallback(self, fun, *args, **kwargs):
        self.append((fun, args, kwargs))

    def callback(self, initial_value):
        for fun, args, kwargs in self:
            initial_value = fun(initial_value, *args, **kwargs)
        return initial_value

Easy isn't it?

>I don't understand why I would need separate threads in the first place. I
>know that the query would block but isn't that transaction being done via
>sockets anyway? Why can't I use the same mechanism (deferred) I would use, to
>say, defer work waiting on a response from a mail server? Can't I submit a
>query, have the reactor go do something else, and have a callback run when
>the data becomes available? Why do I need to spawn threads to avoid blocking?

deferreds don't make blocking code non-blocking, they are just a different mechanism to register callbacks. Instead of doing:

button.handle_event('clicked', list_of_callbacks)

you do

button.clicked().addCallback(callback1).addCallback(callback2)

You need to spawn a thread because there is hardly any database adapter that is written to be async, and those that are written to be async are either buggy or not complete or not really usable, plus having deferreds after every query in python (which doesn't support deferreds natively) makes coding a lot harder (you can deal with some deferreds here and there, but not with 10 deferreds each function).

If you want to do N operations at the same time just use runInteraction instead of runQuery (but this doesn't seem to be the case).

>I can live with using the adbapi but wonder if it's possible to simply defer
>that work and use a callback mechanism (I'm using the original twisted.web
>not twisted.web2).

As I said above, authentication has almost nothing to do with guard and a lot to do with twisted.cred. Cred chapter in Abe's book is one of the best ones in the book thus I suggest you to read it.

>I was pretty much heading down this road but I don't really understand how
>else to do it. I've browsed many examples and have never really come across
>one that involves session.guard, newvow, and db access all in the same
>application. Anyone have any experience they could share?

I can give you an example of how to deal with authentication using a real database:
http://svn.berlios.de/viewcvs/weever/trunk/src/users/auth.py?view=markup
This code uses the 'old' twisted interfaces and is a bit rusty, nevertheless it shows the basic concepts and how to authenticate against a database.

I'm working on a pet project these days which is still very early in development ( http://vise.teknico.net/ ) but I'll have some code cleaned up and ready to be showed in not so long and of course it does authentication against a database using SQLAlchemy as an abstraction layer (it is integrated with twisted in an interesting way).

-- 
Valentino Volonghi aka Dialtone
Now Running MacOSX 10.4
Blog: http://vvolonghi.blogspot.com
New Pet: http://www.stiq.it



More information about the Twisted-web mailing list