The Holy Rant

Jesus, Exposed!


This blog doesn't get used any more.

Please see my website.

Python datetime to RFC2822 Date

Format a Python datetime as a string according to RFC 2822 section 3.3 respecting timezone. Adapted from Python's email.utils.formatdate. As a handy companion, dateutil.parser.parse works quite nicely for parsing them.

def formatrfc2822datetime(value=None, usegmt=False):
    """Returns a date string as specified by RFC 2822 section 3.3., e.g.:
    Fri, 09 Nov 2001 01:08:47 +0900
    Optional value may be a datetime value. Timezone is respected.
    # Note: we cannot use strftime() because that honors the locale and RFC
    # 2822 requires that day and month names be the English abbreviations.
    if value is None:
        value =
    if usegmt:
        # Convert to UTC/GMT
        value = value - value.utcoffset()
        zone = 'GMT'
        offset = value.utcoffset().seconds / 60
        # RFC 2822 3.3.: The form "+0000" SHOULD be used to indicate a
        # time zone at Universal Time.
        zone = '%s%02d%02d' % (offset < 0 and '-' or '+', offset // 60, offset % 60)
    return '%s, %02d %s %04d %02d:%02d:%02d %s' % (
        ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun'][],,
        ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun',
         'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'][value.month - 1],
        value.year, value.hour, value.minute, value.second, zone)

Peach Pie

On Saturday arinellen, tribaldemon, Paige and I went op shopping and I bought this cute pie dish with a a recipe for cherry pie on the bottom. "Mmm, this pie is delicious, can I have the recipe?" "Sure, just eat the rest!"

Naturally Paige and I had to make peach pie!


  • 3 cups self-raising flour
  • 3 tablespoons sugar
  • 250g butter, cold, chopped
  • 2 eggs, lightly beaten

Place the self-raising flour in a bowl. Create a well and fill with eggs and butter. Rub flour into the eggs/butter trying to touch only flour. The pastry should attain a golden colour and flaky texture. Line a pie dish (ours was 9") with about half of the pastry moderately thickly.

Pie filling:

  • 1 large tin peaches in juice
  • 1 cup brown sugar
  • 2 tablespoons butter
  • 2 tablespoons cornflour
  • ½ teaspoon nutmeg
  • ½ teaspoon cinnamon
  • ½ teaspoon vanilla extract

Strain peaches, keeping ½ cup of juice. Combine juice, sugar, butter, cornflour, nutmeg, cinnamon and vanilla extract. Place peaches in pie dish and pour over just enough liquid to submerge. Cover with pastry, sealing edges with a fork. We had a bit of pastry left over (it's so tasty >.>).

Bake for around 20 minutes in a hot (250°C) oven, or until golden brown.

Pics coming soon! (Pie is half-eaten now, though.)

Google Provisioning API 5-day delete restriction hack
class GoogleUsers(object):
    """ Pythonic interface to Google Provisioning API. """
    def __delitem__(self, key):
        delete_key = 'delete%d%d' % (time.time(), os.getpid())
        self.gdata_api.RenameUser(key, delete_key)

OpenLDAP 2.3, Python 2.5 and Python-LDAP on Debian Etch
kill, river, serenity, brain, firefly

In case anyone else has the misfortune of being stuck on a Debian Etch box with Python 2.5 and finds themselves needing Python-LDAP:

Installing the latest version of Python-LDAP 2.3.x doesn't work. It requires OpenLDAP >= 2.3. Installing an old version of Python-LDAP using Python 2.5 against the OpenLDAP ~= 2.1 libraries available in Etch gave me no end of segfaults and weird issues and going back to Python 2.4 wasn't an option.

OpenLDAP 2.3 has been backported by the Debian security team, thankfully. Problem: they didn't create a development package. Solution: I grabbed the source package and fiddled with Python LDAP's setup.cfg. Put the source package somewhere and apply the Debian patch manually. Change the [_ldap] section of Python-LDAP's setup.cfg to something like:

extra_objects = /usr/lib/ /usr/lib/
extra_compile_args = 
libs = sasl2 ssl crypto
library_dirs = /usr/lib
include_dirs = /root/openldap-2.3.30/include /usr/include/sasl

Now easy_install the directory and you're done! Not very Debian, but it works.

Technical musings
Dear Open Source community,

I would like a distributed, fault tolerant, self-optimising, MVCC hash data store with arbitrary metadata and emergent schema. On top of that, I want a nice way of defining distributed indexes. Apart from some academic implementations, I can't see anything robust enough to be used by web applications on an Internet scale.

-- Samuel "fed up with RDBMSes" Cochran

P.S. I'll show you mine if you show me yours.

More Webmail Thoughts

Thinking about webmail has got me thinking about mail at large.

From what I've read/seen most people doing mail open source use something like Dovecot with Large Storage™. What about when that needs scaling? A popular solution seems to be using a combination of a few approaches to horizontal scaling:

  • Partitioning mailboxes using the first letter (or similar) and:
    • telling each user which server to use; or
    • using an IMAP Proxy like Perdition (client – [imap/pop3] → server 1 – [imap/pop3] → {[a-j]* → server 2, [k-z]* → server 3} or similar).
  • Use a shared file storage backend like NFS, GFS, or similar with multiple transport nodes (client – [imap/pop3] → server 1, 2, 3 – [nfs] → server 4 or similar)
  • Use harder/better/stronger/faster hardware.

With the advent of cloud computing how can this be done better? What about a cloud of SMTP/LMTP/IMAP/POP3 servers accessing an S3 like service for mail storage? (Hello Eucalyptus!) One could use MapReduce (using Hadoop? with Hive?) to maintain the mailboxes and create an FTS index. DBMail has an interesting idea but misguided implementation, imho. This would be true open-source Gmail!

I don't think it would be too difficult to create a new mail storage backend for Dovecot and use it for the actual mail transport. (aside: There's currently an experimental backend for SQL mail storage.) It's a pretty good base for any new functionality just by writing new plugins. Putting these atop some lean Ubuntu VMs on Eucalyptus gives massive scalability and HA, and backing onto an S3 w/ MapReduce gives fast, scalable, indexed and HA storage.

Is there any open-source interest in this? Anyone who might get excited with me?

So I've been writing some webmail stuff - braindump time!

After wrestling Horde/IMP, Roundcube, AtMail Open, and so many webmail clients around the place it's not funny the conclusion is that none of them do what I want: modern, fast, efficient webmail. That means gracefully-degrading REST/DRY/AJAX/AcronymOfChoice interface using persistent connections, sensible caching, etc.

The end product should be a nice, clean webmail client which gives you flexible configuration options. It should be just as easy to setup a webmail interface to a single mailbox which requires no authentication as to provide a sophisticated multi-account, multi-mailbox setup. A user should be easily able to log in, view mailboxes, read mail, etc. Someone visiting via Lynx should get all the same functionality as someone using Google Chrome, but the Chrome experience will be slick, AJAXy and and show new mail when it arrives without a reload, etc.

The server should be clever. It should figure out the best caching and pre-fetching strategy for your configuration *at start-up time*. It should be event-driven. It should use IDLE/NOTIFY on IMAP connections and publish back to clients using COMET.

This is a tall order! There are many considerations. PHP is out - it's not persistent, let alone event-driven. Perhaps it could be used for the templating, though. What about if there was an additional IMAP gateway? *It* could be event driven, use persistent IMAP connections, do the IDLE/NOTIFYing, caching and pre-fetching, etc. But how would PHP and the gateway communicate? How about via JSON? This gives us the added benefit of providing an AJAX-compatible data source for all mail operations. Nice! So advanced clients can AJAXify the page and talk directly to the gateway.

But IMAP is just one protocol. The JSON gateway should provide a fairly abstract interface which the webmail application can use, with a backend optimised for each protocol. It may even be better to write direct implementations instead of using too abstracted libraries (like libcamel, libtinymail, etc) as it would let us know that "get folders" means "update the folder navigation, including # unread messages" so we need folder names, hierarchy and unseen count. These can be optimised per backend (IMAP/POP3/Exchange?/filesystem[mbox/Maildir(++)/dbox/..]).

So at the moment I'm hacking something together in PHP using a fairly abstract interface. There are plenty of PHP-hating moments, but it's a great RAD tool. Also, you'll be able to drop it on pretty much any PHP-compatible web server and have it work. When you hit scalability problems, install the PECL mail extension. When you run into problems again, install the gateway. Maybe then offer a complete stack written in something like event-driven C? How about making the JSON gateway part of the mail server accessing mail directly?

It's time for webmail to be done properly in a way that's useful to everyone. Additionally, we need a free, open-source Gmail/Google Calendar/Google XYZ stack - this is a start!

More to come!


In case you didn't know, I'm in Rome. See my Twitter for more details.


Open Source at UWA?
Interesting... A student has recently discovered the University's Open Source Software Policy & Strategy. I found this policy myself recently and started asking why it wasn't being applied more...

The product the student talks about, Zotero, actually looks quite a good alternative to it's proprietary counterpart (EndNote). I imagine managing references from within the browser would be more natural to most academics' online/computer research work-flows. It's also integrated with both Microsoft Word and Open Office, it's open-source rival.

Open source is catching up on the desktop...