Brian's weblog : /twisted/index.html

Zooko's in town, and already I feel 20% smarter. I roped him into a discussion about the Promise syntax I'm developing for Foolscap, and he suggested an alternative that has some good properties.

I'll illustrate with an example where promise-pipelining actually does you some good. (many of the use cases I've been thinking of involve some sort of publish/subscribe scheme, and in those cases you win almost nothing with pipelining). I'm imagining a theoretical Buildbot status interface using newpb, and a tools that wants to connect to the buildmaster and retrieve the results of the latest build for a given Builder. The oldpb code would look like this:

    # Example 1
    def checkResults(results):
        if results == SUCCESS:
            print "yay!"
    def oops(failure):
        print "boo"
    #
    s = getStatus()
    d = s.callRemote("getBuilder", "python-2.4-full")
    d.addCallback(lambda builder: builder.callRemote("getBuild", -1))
    d.addCallback(lambda build: build.callRemote("getResults"))
    d.addCallback(checkResults)
    d.addErrback(oops)

The syntax I've currently got in Foolscap would make it look like this:

    # Example 2
    s = getStatus()
    b = send(s).getBuilder("python-2.4-full")
    b1 = send(b).getBuild(-1)
    r = send(b1).getResults()
    when(r).addCallback(checkResults).addErrback(oops)

The big win with the promise pipelining is that all 3 calls (4 if you include getStatus) take place in one round trip, whereas the oldpb approach requires 3 or 4 separate roundtrips. As MarkM has said, the pipes are getting wider but not shorter, and eventually the round-trip latency will be the biggest bottleneck.

The syntax that Zooko suggested would make this all look much more like the (blocking) synchronous form:

    # Example 3
    s = getStatus()
    b = s.getBuilder("python-2.4-full")
    b1 = b.getBuild(-1)
    r = b1.getResults()
    r._then(checkResults)._except(oops)

Or you could chain it all into a single column, which my editor wouldn't like (you'd have to add some outer parenthesis to keep it indenting happily) but which python will still accept:

    # Example 4
    getStatus().getBuilder("python-2.4-full")
      .getBuild(-1)
      .getResults()
      ._then(checkResults)
      ._except(oops)

which is a lot easier to read than the same collapsed form with my send() syntax:

    # Example 5
    when(send(send(send(getStatus()).getBuilder("python-2.4-full")).getBuild(-1)).getResults()).addCallback(checkResults).addErrback(oops)

Now, a syntax which looks synchronous is great for programmers who aren't familiar with asynchronous control flows: they can look at example 3 or 4 and, except for the funny _then clause, it all looks exactly like what they expect from xmlrpclib or other blocking RPC mechanisms. The problem with this syntax is that they might forget that they're actually dealing with Promises, and try to do something like:

    results = b.getResults()
    if results == SUCCESS:
        print "yay!"

and forget that 'results' is actually a Promise, and the only things you can do with a promise is to send messages to it, or invoke _then or _except. In some cases this could just raise an exception:

    counter = b.getCounter()
    print counter + 1
    # TypeError: unsupported operand types(s) for +: 'instance' and int

And in other cases (like 'results is SUCCESS') it might fail silently, always returning False. Whereas the send() syntax would make it obvious that you're dealing with a Promise.

One thing I like about Zooko's approach is that I can have the _then and _except methods be simplified wrappers for the more general purpose _when or _when_resolved method, the one that returns a Deferred:

    results = b.getResults()
    d = results._when()
    d.addCallback(checkResults)

That way *I* can use Deferreds for my control flow, while the newcomers for whom Deferreds still seem magical can use a somewhat-familiar _then(callback) approach. (without this, we'd be walking backwards in time to the beginning of the evolutionary path that has resulted in Deferreds as a general-purpose callback management tool).

In addition, these two syntaxes aren't necessarily mutually exclusive. I could have one kind of Promise that implements the __getattr__ magic necessary to make Zooko's syntax work, but if you call send() on one, it sets a flag to disable that magic, so that you end up using the send/when syntax.

There was more to the discussion but it's all in a notebook in the other room and I'm too sleepy to express it all right now.

Promises

Aaaagh! Promises are hurting my brain.

I'm trying to figure out how to provide a useful subset of E's reference mechanics in newpb/foolscap. Specifically, one of the clever things that E does is to provide Promise Pipelining, a limited form of remote code execution, in which I can ask you for an object and tell you to deliver a message to that object in a single round trip (rather than the usual two). So I want to be able to do something like:

# target, record, and results are all Promise objects
target = tub.getReferenceAsPromise(sturdyref)
record = send(target).getRecord(args)
results = send(record).getField(otherargs)
def printResults(r):
  print r
when(results).addCallback(printResults) # when() returns a Deferred

You can also include Promises as arguments:

record = send(target).getRecord(args)
send(laserprinter).printRecord(record)

So I'd like to provide this feature in python/foolscap, both because using Promises as a programming technique holds a lot of promise (as it were) for being a cleaner asynchronous style, and because it opens up the possibility of doing pipelining (which is an actual performance win).

The challenge is that E has very different reference mechanics than python. In E, any reference could be a Promise. (specifically, each reference is in any one of 5 states: LocalPromise, RemotePromise, Near, Far, and Broken). Whereas in python, references are always Near, and we have to fake everything else with wrapper objects.

My current approach is to have the Promise class be the wrapper and have it handle everything except Near references. The basic Promise is created with a matching resolver:

promise, resolver = foolscap.makePromise()
resolver(result) # resolves the promise

But the most common way to get one is to do an eventual send to something:

from foolscap import send
class Adder:
  def add(arg):
    return arg+1
a = Adder()
promise = send(a).add(4)

There are only two things you can do with a promise: send it more messages, and wait for it to resolve. The former is done with send (which accepts either a promise or a regular object, and always does an eventual-send), the latter is done with when:

from foolscap import when
d = when(promise)
d.addCallback(printResults)

The when always returns a Deferred that will fire with the resolution of the Promise. So send moves us from the synchronous world to the asynchronous+promise world, while when and addCallback move us back to the synchronous one. (when by itself moves us from the asynchronous+promise world to the asynchronous+Deferred world).

So far so good. But here are some problems:

can Promises be resolved multiple times? I don't think so. The state transitions between LocalPromise and RemotePromise don't count.

how should this interact with eventual-send? certainly when you do send(a).foo(), and a is a normal reference (not a Promise), that a.foo() call should not happen in the same call stack (i.e. Vat turn, i.e. Reactor tick). But should all the events be sent in a big batch as soon as the promise is resolved? Or should they be sent out one at a time somehow? I suppose if Promises can only ever be resolved once, this is not as complicated as I'd first thought.

Should we try to support immediate calls to resolved Promises? In E, if you have a Near reference, you can do both immediate and eventual sends. In python, it would look like:
```
p = send(obj).foo(args)
# later.. p is probably resolved
send(p).bar() # eventual send
p.baz() # immediate send
```
Hm, maybe that isn't such a great idea.

newpb-0.0.2 released

I finally got some twisted time this weekend, so I fixed ticket #1999 and moved newpb out of the Twisted subdirectory entirely, renaming it to Foolscap in the process. I also released version 0.0.2, so there's a complete tarball ready to install and play with.

Having it live outside the Twisted tree has a number of advantages. Twisted is mature enough to have moved to a slower development model that preserves stability at the expense of making new development easy. Each potential change to the codebase must be reviewed before being applied to the trunk, so all development takes place on branches and must serve to fix a specific ticket. Very little of the newpb development falls under this model, and there are a distinct scarcity of people able to review newpb code. By moving it outside the Twisted tree, I can continue to work on it in a more suitable development model.

In addition, moving it outside the twisted. package makes it much easier to test and deploy. When it lived in twisted/pb/*.py, you had to actually install it before using it, into the same directory as the rest of Twisted. Now that it lives in foolscap/*.py instead, you can run it from the source tree. This will make things easier for everybody.

The new name is a bit of a compromise, though. I'm not entirely satisfied with "Foolscap". It has some good properties (google thinks it is fairly unique, it has "cap" which might make you think of capabilities, it has "oo" which might make you think of objects, there's the visual of a twisted foolscap of paper, the jester's hat-and-bells could make a nice logo). But it also has some bad ones (MarkM points out that there's enough negative baggage around the word "capabilities" that you might not want "cap" in your protocol name, using the word "fool" gives some negative connotations, the promise-pipelining aspects are really more interesting than the capabilities ones, and anyways "foolscap" doesn't really flow off the tongue in a glib manner). But it needed a name to live outside Twisted, and now it has one. That might change, but Foolscap should get us through the next couple of months.

I've been staring at E's CapTP protocol a lot, thanks to help from Mark Miller, trying to understand what their goals are, how they accomplish them, and what pieces would be useful to implement in Foolscap. What I learned last week was how the CapTP 3-Vat introduction system works. I think I can implement it in Foolscap, but I'm trying to decide if it's worth it. CapTP does some funny tricks to make sure that messages which introduce two Vats are delivered in the correct order relative to other messages between those Vats (this is called E-Order in MarkM's papers). I assume this is a good property to maintain (my general approach is to assume that everything MarkM does has a good reason behind it, and that if I work at it long enough I may learn that reason for myself, but for now just shut up and implement it).

But a lot of CapTP is tied up in Promises, and I'm still getting my head around how to provide something in python that resembles a Promise and is still useable. We don't have a lot of the language features that E does, in particular the way that an E object holding a reference to a Promise will eventually discover (after the promise has been resolved) that they're holding a reference to some other object. We don't have that sort of silent slot mutation in Python, so I'm trying to figure out what would be a meaningful equivalent. So far the Promise syntax is looking something like:

 p2 = send(p1).foo(args)
 #  equivalent of E's:  p2 = p1 <- foo(args)

Of course you can also use send() on non-promises if you just want to do an eventual-send. This is a more precise way to accomplish what I've been (crudely) doing with reactor.callLater(0,..) all these years. I'm also writing a sendOnly for when you want to throw away the return value. E has compiler support for this, it knows whether the results of the send are used or not, and can switch between send and sendOnly automatically. Python does not have such a context sensor, so we have to do it by hand.

Then, when you want to interface back to the synchronous world, you use when() to turn the promise into a Deferred, to which you can then attach some code to run:

 def _stuff(value):
     print value
 d = when(p2)
 d.addCallback(_stuff)

Trying to get this to work with the actual eventual-send queue and make the result Promises work correctly is making my head spin. I need to sit down with Zooko on this stuff, he'll understand it well enough to help me get my brain around it.

concurrency

Had a great chat with Donovan today, about newpb and E and secure python and concurrency management. It turns out we have some of the same ideas about interesting things to do with these kinds of tools. He pointed me at a language named Io that's doing some neat stuff with lightweight coroutines, and had some interesting thoughts on coroutines in python (making protocol-parsing code look a good bit simpler than the purely data-driven model that twisted Protocol classes tend to have).

Twist-E

Spent another great day down at HP, talking about implementing E and web-calculus concepts within Twisted and newpb. Tyler Close was kind enough to spend the entire afternoon with me, explaining how his web-calculus works and the design decisions behind it. I'm really excited about implenting this stuff in newpb: I think we can make a system that's both secure *and* highly usable. Some of the ideas I came away with that I want write up before I forget:

Promises: In addition to Deferred, we can build a Promise. The usage syntax would look like:

 p = tub.getReference(url)
 p.authorize(credentials).subscribe(self)
 when(p.getReady()).addCallback(lambda res: p.trigger())
 p2 = Promise(d1) # turn "deferred which fires with an instance" into a Promise
 p3 = p2.invoke()
 d2 = when(p3)
 d2.addCallback(stuff)

The Promise object is basically a wrapper around any Deferred that expects to fire with an instance. It has a __getattr__ which lets it pretend to implement any method. Such methods just queue the call and its arguments, then finish immediately, returning a new Promise. Something like:

class Promise:
  def __getattr__(self, methname):
    if self.resolved:
        m = getattr(self.resolution, methname)
        assert callable(m)
        return m
    def newmethod(*args, **kwargs):
        self.calls.append((methname, args, kwargs))
        # except more cleverness in case the method is invoked after the
        # promise is resolved
    return newmethod

When the Deferred fires, all pending calls are invoked on the instance it fired with. Each call also returns a Promise, possibly already fulfilled, with the results of that call, so that p.meth1().meth2() is the asynchronous equivalent of o.meth1().meth2(), or func2(func1(o)). 'p.meth1(); p.meth2()' means that meth2 must be invoked *after* meth1: I'm not sure what other kind of sequencing promises to make (should we wait until meth1 has finished before invoking meth2?).

If the Deferred errbacks instead, then the Promise is "smashed", which is like an errback. No further method calls are made, any dependent Promises are smashed too.

The idea is to make the asynchronous domain be the normal case, and mark the boundary with the synchronous domain specially. when() would be a function that turns a Promise into a Deferred, with which the transition could be scheduled:

def when(p):
  if not isinstance(p, Promise):
    return defer.succeed(p.resolution)
  if p.resolved:
    return defer.succeed(p.resolution)
  else:
    d = defer.Deferred()
    p.waiting.append(d)
    return d

He pointed out that E currently has two separate method invocation syntaxes: 'o.foo()' requires a local reference, and may or may not return a Promise. 'p <- foo()' can accept either a local reference or a Promise, and always returns a Promise. (actually I'm not sure I'm getting this right, but the implication was that there were two forms, one for local and one for remote, whereas Tyler felt that there should only be one).

Then, later, we'll create the RemotePromise, which is a Promise that's associated with a RemoteReference. rp.foo(args) is equivalent to d.addCallback(lambda res: res.callRemote("foo", args)) . When Promises are serialized, they get a clid and show up as another Promises on the far end. You push the waiting as far away as possible, apparently this is the way to reduce the probability of deadlocks.

My main concern with this syntax is that it may confuse the synchronous-domain developers that we (as Twisted) have been trying to gently nudge into the world of asynchronous programming. We're not blocking, but the code looks a lot like that's what's happening. But, once you've stopped thinking that the lack of a .callLater implies immediate execution, the p.meth(args) syntax really is a lot cleaner. You just assume that everything could be a promise, and you use when() if you need to assure that you have an immediate value.

One problem with reference counting is that your peer can force you to retain an object for arbitrarily long times, by just never sending you the decref (and Gifts make things even worse). Tyler's hunch is that distributed reference counting is the wrong approach, and it is more practical to manage object lifetime with the Vat/Tub. Break application processing into units, create a Tub for each unit, when the unit is finished, destroy the Tub. All objects that pass through a Tub are registered (under an unguessable name) in that Tub, so they remain accessible for the lifetime of the Tub, and then become inaccessible when the Tub is destroyed.

To use this well, it must be easy to create new Tubs and destroy them later. These Tubs must be able to share listener ports, which can distinguish the desired Tub by its keyid. To accomplish this with newpb, I think we may need a module-level registry of Listeners, so that two Tubs that are asked to listen on the same port will register with the same Listener. (it might also make sense to use newtub = oldtub.makeTub(), and have the Listener be inherited). We should pay attention to the possibility of sharing a TCP connection to an existing Tub, but keep in mind that separate TLS keys will require separate TCP connections.

Secure PB URLs want a key as the primary specifier, followed by a list of location hints, followed by a Tub-scoped name.

PBY url: pby://key@1.2.3.4,foo.com,[::1],loc2,loc3/name
 key is base32(sha1(tub.pubkey))
 unix socket is trickier
 non-authenticated url still requires Tub ID

He also feels that DoS prevention (one of the three reasons for Constraints, the other two being semantic typechecking assertions and API documentation) is difficult to implement and hard to get right, and unlikely to do the complete job that you'd want out of it. He said MarkM burned a lot of cycles trying to build DoS prevention techniques into CapIDL, and it would be worth asking him for his thoughts.

He said one deployment pattern would be to put security proxies in a set of separate processes, which perform deserialization, check arguments, etc, and then pass the results on to the real object. The security proxies would be CPU/memory limited, and there would be one per connection, so that if someone started to abuse their connection, only they would suffer. Once you get to a service large enough to be worried about DoS attacks, you'd want this architecture anyway because then you can distribute it out to multiple machines. I was skeptical about how to go about implementing this sort of proxy: how much CPU time do you give it? If it takes 1ms to deserialize a message that then consumes 1s of server time, do you have to restrict it to 1/1000th the CPU time of the server? Note that other possibilities include strict prioritization of the processes/threads (so the connections are starved until the server becomes idle), and enforcing one-at-a-time processing of messages.

His approach in web-amp was just to limit each serialized argument to 8kb. The objection that this might not be enough is countered by the fact that if you're sending more data than that, you should mark it explicitly (by creating a publish/subscribe model), because there's a good chance that the data is being used on the wrong side of the wire. The attacker is allowed to do whatever evil they can accomplish in 8kb, maybe that means a 2k-deep nested series of lists, but whatever it is won't be too big. I feel that at some point you have to enforce a limit.. in web-amp, you must limit the total number of arguments they can send you, or the number of method calls per second, or something.

The non-DoS-related semantic typechecking (I'm expecting an int, is it really an int?) is just as easily done with assert()s inside the method body. I want this kind of checking to happen as close to the top of the method as possible.. doing it in a RemoteInterface in some separate file feels wrong to me. One approach is a func.guard method attribute (whose constructor takes arguments much like the RemoteInterface methods do), which could be pulled up to the top of the method body with a decorator. The big difference in thought here is the idea of providing objects (which happen to implement a certain set of methods) versus providing methods (which happen to be bound to a particular object).

A lot of the typechecking concerns are eased with finer-grained capabilities. Ideally, the worst they can do by sending you a weird object type is to cause an exception. As long as you haven't registered an Unslicer that gives the resulting object some ambient authority, you aren't going give them any new privileges by invoking a method on something they *can* give you. Tyler says you only do typechecking when you're considering granting them some new privileges. The notion is that it's the bound-method capability that is the basis of power, not what they do with it or what they send to it.

The constraints are useful for method documentation, especially if they can be serialized and passed to an object browser, but can only document the list of methods and the names/types of their arguments. The actual API description still needs to be in epydoc, which can provide (non-machine-parseable) argument name/type docs too.

positional parameters for interoperability with java:

java doesn't have keyword args. To provide interoperability, the python-newpb method call serializer needs to send args in strict order, the java newpb receiver would ignore the argument names (only using the values). In the other direction, the java method call serializer would send None for the argument names, and the python receiver would use the local RemoteInterface to turn the argument list into a kwargs dict.

Finally, I need to study the XML schemas in the web-calculus more closely. In it, the bound method closure URL can be used for two purposes: a GET returns the method schema (a description of what types the positional parameters will accept), while a POST will invoke the closure. However, the object which provided that URL has a class, and the method clause had a name, and the method schema is always the same for any given (class, methodname) pair, so even a fully send-time-checking implementation doesn't have to retrieve any method schema more than once. I had first thought that there was some reduncancy in the XML data being returned, but Tyler's put a lot of thought and time into it to minimize the round-trips and avoid redundancy. newpb would be well-served by studying his approach carefully.

twisted talk

So I think the talk went really well. I spoke for about an hour before the room was needed for another meeting, to about 10 or 15 OSAF developers. I managed to cover the reactor, Protocols, Factories, building higher-level protocols, Failures, Deferreds, reactor.run() vs twistd -y vs mktap/twistd -f, and even a bit of twisted.web (the resource-tree model) and threads (reactor.runInThread/runFromThread). The things that were on my list but which I didn't get to cover were Cred, usage.Options, PB, and Interfaces.

But all in all I think the session helped a lot of people get their heads around the architecture.. I think they're now in a position to understand the existing HOWTOs and other documentation.

After the session, I sat down with Brian and two other OSAF folks: Lisa Dusseault and Grant Baillie. They are working on WebDAV, and have a strong interest in a functional WebDAV client library. As I understand it, this library's top-level API would need to look like an abstract file system, with directory lookups, pathnames, something like file handles, and file attributes. Inside, it would need to have a back end which actually speaks WebDAV to some server, creating new connections when necessary, or re-using persistent connections is possible. There would also need to be some sort of cache-management policy hting, since smart caching can make or break the performance of a WebDAV session.

Given their needs, we agreed that a Twisted WebDAV client library would be a great solution, and they've got the motivation and the knowledge (apparently Lisa was one of the primary WebDAV folks at Microsoft) to pull it off.

I described the recent work that's gone into an abstract file system (by spiv and others, for twisted.ftp), thinking that it would be the best place to start. The next step will probably be to introduce them to spiv, and float a post on twisted-python to see who else has an interest.

Brian also gave me a quick demo of Chandler, giving me a better idea about where they're going and what their plans are. It's funny, about 15 years ago I had a summer job at a research lab who had a similar goal. They were working on OCR and search technology, and wanted to make a box that could digitize and read all the random bits of paper that you produce in the course of a day, then let you index the information contained on them in a useful way. The Chandler folks want to take all the random bits of digital information that you create in the course of a day (email, IMs, calendar entries, todo lists) and organize/share them in a useful way. Kinda neat. I look forward to seeing where it goes.

OSAF Twisted talk

This is a rough outline of the talk I'll be giving at the OSAF tomorrow.

definition of Twisted, resources:
 http://www.twistedmatrix.com
  svn://svn.twistedmatrix.com/svn/Twisted/trunk
  http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
  http://twistedmatrix.com/bugs/
  http://twistedmatrix.com/buildbot/
 #twisted, #twisted.web on freenode

relationship of subprojects, dependencies:
 core, names, mail, web, words, conch, trial
 zope.interface, python2.2
 optional: pyopenssl, db stuff

directory overview:
 twisted.python: usage.Options, Failure, log
 twisted.internet: reactors, base classes for Protocol+Factory, Deferred
 twisted.protocols: simple protocols: finger, socks, telnet
 subproject directories
 doc/*/howto
 doc/core/howto/tutorial/listings/finter/*.py

motivation:
 simple client
 simple server
 not-so-simple server
 client+server
 need for a generalized solution
 threads, processes, event loop
event loop:
 asyncore
 reactor

picture: reactor with select() call, sockets in .readers/.writers
 sockets have .doRead, .doWrite, are scheduled with .addReader/etc
 timers
 different kinds of reactors, using other event loops: gtk, kqueue

picture: Protocol with Transports, reactor
 Protocol: connectionMade, dataReceived, connectionLost, transport.write

how do those Protocols get created?
reactor.listenTCP(port, factory)
picture (server): Protocols, Factory
 listening socket (Port) points to Factory, creates new Protocols
 Factory gets startFactory, stopFactory, buildProtocol
 Protocols generally have .factory

reactor.connectTCP(host, port, factory)
picture (client):
 Factory gets startedConnecting, clientConnectionFailed, clientConnectionLost
  as well as startFactory, stopFactory, buildProtocol
 Connector is responsible for getting a connection to host+port+factory
  possibly multiple times, for ReconnectingClientFactory
 skip over Connector stuff

writing Protocols, using existing ones
picture: t.p.finger.Finger
 overridable methods for getUser, getDomain, forwardQuery
 subclass, override method
 make a Factory which instantiates your new subclass
 attach to listenTCP

Protocols are used for both clients and servers
 state machine
 return one-shot results with Deferreds
 return multi-shot results by overriding methods

larger protocols have more complex setup

names: protocol parses the query, hands to factory
 factory does self.handleQuery, asks self.resolver, calls self.sendReply
 # good example of API, use of deferred: t.n.server.py:120, dns.py:1050

web: basic HTTP protocol creates Requests, then does req.process
 twisted.web.site implements a Resource tree
  picture(web): root, getChild(), isLeaf, render(req)
  specialized subclasses provide CGI processing, static.File, distrib

imap: involves cred, Mailbox objects, Message objects

top-level invocation:
 __main__, reactor.run()
  connectTCP, listenTCP
 or, creating an Application, then using twistd
  motivation: daemonization, logging, setuid/chroot, reactor, profiling
   think /etc/init.d
  picture: trees of Service/MultiService objects
   each gets startService, stopService
   t.a.internet.TCPServer(port, factory), TCPClient
  twistd -y foo.tac, script which creates an Application object
   sidebar: python as a configuration language
  serialize the Application, then launch it again later: twistd -f foo.tap
  shortcuts for common applications: mktap
  mktap plugins: Options, makeService(), register with plugins.tml

threads:
 nothing here needs threads
 where are they useful?
  wrapping blocking APIs: adbapi in particular
  integrating with other code
 threadpool: run a function in a thread, tell me when it is done

t.p.log:
 log.msg(msg, msg) emits a log
 log.err() emits the current exception
 log.err(f) emits a Failure object
 log output goes to an observer
 running from twistd: goes to twistd.log, or syslog
 running from __main__: log messages are discarded
 log.startLogging()

Failure:
 encapsulates a python exception
 can be serialized, printed, queried about what caused it
 Failure() inside an except: block wraps the current exception

Deferred:
 callback management
 use web.client.getPage as an example
 synchronous style:
   a=foo()
   b=bar(a)
   baz(b)
 asynchronous style:
   d=foo();
   d.addCallback(bar)
   d.addCallback(baz)
 callback vs errback, ladder diagram
 fire-before-addCallback is safe
 callbacks can return Deferreds: sub-ladders

usage.Options:
 create subclass, attributes indicate valid options
  optFlags, optParameters, subCommands
  define opt_foo(self,str) to implement --foo=str
 methods can customize processing further
  parseArgs, postOptions
 str() provides usage message
 Options implements the dict interface, opts['foo'], opts['v']
 usually invoked with opts.parseOptions(), which grabs sys.argv
 why? mktap plugins use the 'Options' class from the plugin to parse argv

lore:
 turn .xhtml into .html (or .latex, others)
  inline listings, pretty-print python code
  links to epydoc-generated API docs

pb:
 translucent RPC
 f=pb.PBServerFactory(root); reactor.listenTCP(port, f)
 cf=pb.PBClientFactory(); reactor.connectTCP(host, port, cf)
 d=cf.getRootObject(); d.addCallback(dostuff)
 ref.callRemote("method", args)
 def remote_method(self, args)

cred: howto is really good
 avatar, portal, realm, credentials, checker, mind
 portal has a set of checkers
 checker gets credentials, decides if they're ok, provides an avatarID
 realm gets avatarID and desired interfaces, returns an avatar
 protocol gets back the avatar, does stuff with it

interfaces: PEP245-style
 twisted/python/components.py
 zope.interface, tiny portion of Zope3
 many APIs want "object that can be adapted to IFoo" rather than an instance
  of a specific class
 some systems use it extensively: nevow's 'context': IRequest,ISession,ISite

Brian's weblog

foolscap.lothar.com