Brian's weblog

I just spent several hours getting a reasonable python environment working under Windows, something I had hoped to never have a need for. The Buildbot is having some.. disagreements.. with Windows, and it became clear that being able to reproduce the problem locally was the only sane way to fix it.

Man, was that painful.

For the record, here's what I did. Many thanks to Bear for creating this checklist and walking me through the process.

0. Check to make sure your PATHEXT environment variable has ";.PY" in it -- if not set your global environment to include it.

Control Panels / System / Advanced / Environment Variables / System variables

1. Install python -- 2.4 -- http://python.org * run win32 installer - no special options needed so far

2. install zope interface package -- 3.0.1final -- http://www.zope.org/Products/ZopeInterface * run win32 installer - it should auto-detect your python 2.4 installation

3. python for windows extensions -- build 203 -- http://pywin32.sourceforge.net/ * run win32 installer - it should auto-detect your python 2.4 installation

the installer complains about a missing DLL. Download mfc71.dll from the site mentioned in the warning (http://starship.python.net/crew/mhammond/win32/) and move it into c:\Python24\DLLs

4. at this point, to preserve my own sanity, I grabbed cygwin.com's setup.exe and started it. It behaves a lot like dselect. I installed bash and other tools (but *not* python). I added C:\cygwin\bin to PATH, allowing me to use tar, md5sum, cvs, all the usual stuff. I also installed emacs, going from the notes at http://www.gnu.org/software/emacs/windows/ntemacs.html . Their FAQ at http://www.gnu.org/software/emacs/windows/faq3.html#install has a note on how to swap CapsLock and Control.

I also modified PATH (in the same place as PATHEXT) to include C:\Python24 and C:\Python24\Scripts . This will allow 'python' and (eventually) 'trial' to work in a regular command shell.

5. twisted -- 2.0 -- http://twistedmatrix.com/projects/core/ * unpack tarball and run python setup.py install Note: if you want to test your setup - run: python c:\python24\Scripts\trial.py -o -R twisted (the -o will format the output for console and the "-R twisted" will recursively run all unit tests)

I had to edit Twisted (core)'s setup.py, to make detectExtensions() return an empty list before running builder._compile_helper(). Apparently the test it uses to detect if the (optional) C modules can be compiled causes the install process to simply quit without actually installing anything.

I installed several packages: core, Lore, Mail, Web, and Words. They all got copied to C:\Python24\Lib\site-packages\

At this point

trial --version

works, so 'trial -o -R twisted' will run the Twisted test suite. Note that this is not necessarily setting PYTHONPATH, so it may be running the test suite that was installed, not the one in the current directory.

6. I used CVS to grab a copy of the latest Buildbot sources. To run the tests, you must first add the buildbot directory to PYTHONPATH. Windows does not appear to have a Bourne-shell-style syntax to set a variable just for a single command, so you have to set it once and remember it will affect all commands for the lifetime of that shell session.

set PYTHONPATH=. trial -o -r win32 buildbot.test

To run against both buildbot-CVS and, say, Twisted-SVN, do:

set PYTHONPATH=.;C:\path to\Twisted-SVN

I'm pushing to get a new BuildBot release out on monday, so the last few days have been a flurry of commits (and the weekend will probably be the same). I was very pleased to hear that the Boost crew have implemented a Buildbot to run their (very large) regression test suite, especially because Dave Abrahms and I talked about setting one up two years ago, at PyCon, and I was never able to give them the time to make it happen. I was even more pleased to hear that their goal is to move all their testing over to buildbot. You couldn't ask for better marketing than for the STL heir-apparent to be using your project :).

Both Thomas (at Fluendo) and the Boost folks have patched their buildbots to allow the waterfall display be themed with CSS, and the results look great. I'm looking forward to getting Thomas's code pulled into the mainline sources.. finally a way to make the waterfall display less ugly.

Finally, the metabuildbot is shaping up. This is a buildbot that works to run the buildbot's own unit tests. I need to find a reasonable hostname and link for it, then I'll make it publically visible. Bear has put a lot of time so far into making the win32 slave work correctly, with no success yet (the specific problem is that I'm using Arch to get up-to-date sources out to the buildslaves, and tla is not happy on win32, some kind of 260-character limit on pathnames that tla runs up against when it does a checkout). I've dropped back to CVS for now (with a three-hour timeout in the hopes of getting around sf.net's enormous anoncvs latency), but a separate bug in the buildslave, compounded by a bug in the buildslave's error-handling code, have conspired to get the win32 slave into a state that requires manual intervention to un-jam. Grr, stupid windows.

So I think the talk went really well. I spoke for about an hour before the room was needed for another meeting, to about 10 or 15 OSAF developers. I managed to cover the reactor, Protocols, Factories, building higher-level protocols, Failures, Deferreds, reactor.run() vs twistd -y vs mktap/twistd -f, and even a bit of twisted.web (the resource-tree model) and threads (reactor.runInThread/runFromThread). The things that were on my list but which I didn't get to cover were Cred, usage.Options, PB, and Interfaces.

But all in all I think the session helped a lot of people get their heads around the architecture.. I think they're now in a position to understand the existing HOWTOs and other documentation.

After the session, I sat down with Brian and two other OSAF folks: Lisa Dusseault and Grant Baillie. They are working on WebDAV, and have a strong interest in a functional WebDAV client library. As I understand it, this library's top-level API would need to look like an abstract file system, with directory lookups, pathnames, something like file handles, and file attributes. Inside, it would need to have a back end which actually speaks WebDAV to some server, creating new connections when necessary, or re-using persistent connections is possible. There would also need to be some sort of cache-management policy hting, since smart caching can make or break the performance of a WebDAV session.

Given their needs, we agreed that a Twisted WebDAV client library would be a great solution, and they've got the motivation and the knowledge (apparently Lisa was one of the primary WebDAV folks at Microsoft) to pull it off.

I described the recent work that's gone into an abstract file system (by spiv and others, for twisted.ftp), thinking that it would be the best place to start. The next step will probably be to introduce them to spiv, and float a post on twisted-python to see who else has an interest.

Brian also gave me a quick demo of Chandler, giving me a better idea about where they're going and what their plans are. It's funny, about 15 years ago I had a summer job at a research lab who had a similar goal. They were working on OCR and search technology, and wanted to make a box that could digitize and read all the random bits of paper that you produce in the course of a day, then let you index the information contained on them in a useful way. The Chandler folks want to take all the random bits of digital information that you create in the course of a day (email, IMs, calendar entries, todo lists) and organize/share them in a useful way. Kinda neat. I look forward to seeing where it goes.

This is a rough outline of the talk I'll be giving at the OSAF tomorrow.

definition of Twisted, resources:
 http://www.twistedmatrix.com
  svn://svn.twistedmatrix.com/svn/Twisted/trunk
  http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
  http://twistedmatrix.com/bugs/
  http://twistedmatrix.com/buildbot/
 #twisted, #twisted.web on freenode

relationship of subprojects, dependencies:
 core, names, mail, web, words, conch, trial
 zope.interface, python2.2
 optional: pyopenssl, db stuff

directory overview:
 twisted.python: usage.Options, Failure, log
 twisted.internet: reactors, base classes for Protocol+Factory, Deferred
 twisted.protocols: simple protocols: finger, socks, telnet
 subproject directories
 doc/*/howto
 doc/core/howto/tutorial/listings/finter/*.py

motivation:
 simple client
 simple server
 not-so-simple server
 client+server
 need for a generalized solution
 threads, processes, event loop
event loop:
 asyncore
 reactor

picture: reactor with select() call, sockets in .readers/.writers
 sockets have .doRead, .doWrite, are scheduled with .addReader/etc
 timers
 different kinds of reactors, using other event loops: gtk, kqueue

picture: Protocol with Transports, reactor
 Protocol: connectionMade, dataReceived, connectionLost, transport.write

how do those Protocols get created?
reactor.listenTCP(port, factory)
picture (server): Protocols, Factory
 listening socket (Port) points to Factory, creates new Protocols
 Factory gets startFactory, stopFactory, buildProtocol
 Protocols generally have .factory

reactor.connectTCP(host, port, factory)
picture (client):
 Factory gets startedConnecting, clientConnectionFailed, clientConnectionLost
  as well as startFactory, stopFactory, buildProtocol
 Connector is responsible for getting a connection to host+port+factory
  possibly multiple times, for ReconnectingClientFactory
 skip over Connector stuff

writing Protocols, using existing ones
picture: t.p.finger.Finger
 overridable methods for getUser, getDomain, forwardQuery
 subclass, override method
 make a Factory which instantiates your new subclass
 attach to listenTCP

Protocols are used for both clients and servers
 state machine
 return one-shot results with Deferreds
 return multi-shot results by overriding methods

larger protocols have more complex setup

names: protocol parses the query, hands to factory
 factory does self.handleQuery, asks self.resolver, calls self.sendReply
 # good example of API, use of deferred: t.n.server.py:120, dns.py:1050

web: basic HTTP protocol creates Requests, then does req.process
 twisted.web.site implements a Resource tree
  picture(web): root, getChild(), isLeaf, render(req)
  specialized subclasses provide CGI processing, static.File, distrib

imap: involves cred, Mailbox objects, Message objects

top-level invocation:
 __main__, reactor.run()
  connectTCP, listenTCP
 or, creating an Application, then using twistd
  motivation: daemonization, logging, setuid/chroot, reactor, profiling
   think /etc/init.d
  picture: trees of Service/MultiService objects
   each gets startService, stopService
   t.a.internet.TCPServer(port, factory), TCPClient
  twistd -y foo.tac, script which creates an Application object
   sidebar: python as a configuration language
  serialize the Application, then launch it again later: twistd -f foo.tap
  shortcuts for common applications: mktap
  mktap plugins: Options, makeService(), register with plugins.tml

threads:
 nothing here needs threads
 where are they useful?
  wrapping blocking APIs: adbapi in particular
  integrating with other code
 threadpool: run a function in a thread, tell me when it is done

t.p.log:
 log.msg(msg, msg) emits a log
 log.err() emits the current exception
 log.err(f) emits a Failure object
 log output goes to an observer
 running from twistd: goes to twistd.log, or syslog
 running from __main__: log messages are discarded
 log.startLogging()

Failure:
 encapsulates a python exception
 can be serialized, printed, queried about what caused it
 Failure() inside an except: block wraps the current exception

Deferred:
 callback management
 use web.client.getPage as an example
 synchronous style:
   a=foo()
   b=bar(a)
   baz(b)
 asynchronous style:
   d=foo();
   d.addCallback(bar)
   d.addCallback(baz)
 callback vs errback, ladder diagram
 fire-before-addCallback is safe
 callbacks can return Deferreds: sub-ladders

usage.Options:
 create subclass, attributes indicate valid options
  optFlags, optParameters, subCommands
  define opt_foo(self,str) to implement --foo=str
 methods can customize processing further
  parseArgs, postOptions
 str() provides usage message
 Options implements the dict interface, opts['foo'], opts['v']
 usually invoked with opts.parseOptions(), which grabs sys.argv
 why? mktap plugins use the 'Options' class from the plugin to parse argv

lore:
 turn .xhtml into .html (or .latex, others)
  inline listings, pretty-print python code
  links to epydoc-generated API docs

pb:
 translucent RPC
 f=pb.PBServerFactory(root); reactor.listenTCP(port, f)
 cf=pb.PBClientFactory(); reactor.connectTCP(host, port, cf)
 d=cf.getRootObject(); d.addCallback(dostuff)
 ref.callRemote("method", args)
 def remote_method(self, args)

cred: howto is really good
 avatar, portal, realm, credentials, checker, mind
 portal has a set of checkers
 checker gets credentials, decides if they're ok, provides an avatarID
 realm gets avatarID and desired interfaces, returns an avatar
 protocol gets back the avatar, does stuff with it

interfaces: PEP245-style
 twisted/python/components.py
 zope.interface, tiny portion of Zope3
 many APIs want "object that can be adapted to IFoo" rather than an instance
  of a specific class
 some systems use it extensively: nevow's 'context': IRequest,ISession,ISite

I set up a few tools to post blog entries from emacs. All entries are kept in CVS, and the whole tree is rsync'ed over to the web server. The elisp which actually publishes the entry looks like this:

(defvar pyblosxom-entry-dir "~/stuff/Projects/WebLog/entries")

;; adapted from http://wiki.woozle.org/BlogdorEngine
;; and http://list-archive.xemacs.org/xemacs/200211/msg00022.html

(defun char-isalpha-p (thechar)
  "Check to see if thechar is a letter"
  (and (or (and (>= thechar ?a) (<= thechar ?z))
	   (and (>= thechar ?A) (<= thechar ?Z)))))

(defun char-isnum-p (thechar)
  "Check to see if thechar is a number"
  (and (>= thechar ?0) (<= thechar ?9)))

(defun char-isalnum-p (thechar)
  (or (char-isalpha-p thechar) (char-isnum-p thechar)))


(require 'cl-seq)

(defun blog-publish ()
  "Publish the blog entry in the current buffer"
  (interactive)
  (shell-command (format "cvs commit -m 'blog entry' %s"
                         (file-name-nondirectory buffer-file-name)))
  (shell-command "make -C .. publish")  ; publish
)

(define-minor-mode pyblosxom-post-minor-mode
  "Minor mode for blog posts"
  nil
  " blog-post"                          ; mode-line indicator
  '(
    ("\C-c\C-c" . blog-publish)
    )
  ()                                    ; forms run on mode entry/exit
)

(defun blog-post (title)
  "Create a journal entry"
  (interactive "sTitle: ")
  (let ((filetitle (substitute-if-not ?_
                                      (lambda (c) (char-isalnum-p c))
                                      title)))
    (find-file (concat pyblosxom-entry-dir "/"
                       filetitle
                       (format-time-string "-%Y-%m-%d-%H-%M")
                        ".txt"))
    (goto-char (point-min))
    (insert title "\n\n")
    (save-buffer)
    (vc-register)
    (pyblosxom-post-minor-mode 1)
))

I've been trying to get my project notes online for years now, and I finally realized that I need to start smaller. After a week of intermittent activity, I finally got PyBlosxom set up and behaving fairly well.

In the process, I discovered that the CGI specification doesn't actually require the web server to close the child process' stdin: the child is supposed to read exactly CONTENT_LENGTH bytes and then stop. Pyblosxom violates this (it just reads until EOF), but it usually doesn't matter because most web servers are nice enough to close it. It turns out that Twisted's CGI handling module does not. I fixed it upstream (at least in the old 'web' module.. web2 is another matter), and filed a pyblosxom bug. For now I have to monkeypatch Twisted in my web setup program, at least until Twisted-2.0.1 comes out.

My goal for this web log is twofold. The first is to contain an archive of useful things I come across, cool stuff, new ideas, things I've learned, the usual blog fodder. The second is to publish the diaries that I keep on each of my projects, diaries that I use to help remember what I was thinking when I last put energy into that project, in the hopes of reducing the context-switching penalty that comes about from having a dozen active projects at once. I have something like 1.5MB of diary entries on these projects, going back to mid-2002 when I started keeping them. By getting these projectlogs online (and enabling comments on them), I hope to give the rest of the world a chance to look at my workbench and tell me what they find interesting. Maybe someone else will pick up on an idea that I haven't had time to pursue, maybe they will leave a note with useful directions to go or tools to use. Perhaps by letting others in, and hopefully forming a bit of a community here, I'll be more encouraged to work on the projects that have a chance of forming communities of their own.

I plan to use pyblosxom's categories to put all the projects in places like Projects/BuildBot . Non-project related things like site-setup notes or personal rants can go in other categories next to Projects.

The blog is pretty basic for now, and not necessarily pleasant to look at, but it's a start, and I've learned that waiting for perfection is the best way to never get anything finished at all.

Welcome!

-Brian