This note was published on 29th Apr 2010, and is a mirror of the original post via perscon.net.

The App Engine data collector for Personal Containers is coming on nicely, and is on track for an alpha preview release fairly soon. Working with AppEngine has been interesting; it’s got excellent availability and you can’t beat the price (free), but coding robust Python that doesn’t trip over the tight resource limits for individual requests, asynchronous tasks and queries is tricky. While it is good for small records such as my iPhone or Find My iPhone GPS traces traces, it doesn’t work so well with my gigabytes of photographs or decades of e-mail.

This confirmed our earlier intuition that there is no one perfect solution for personal data handling; instead, we need to embrace diversity and construct an infrastructure that can cope with change over the coming decades. Mobile programming has changed beyond recognition in just a few years, and cloud providers are specialising in different ways (e.g. PiCloud for simple compute, or EC2 for fancy services like elastic load balancing).

So to recognise this, we are building components that all interoperate with your personal data, keep it secure, and ensure it persists for more than a few years. Malte Schwarzkopf came up with the term "digital yurts", and it's stuck. We’ve written a draft paper about it, and would love to hear your comments and feedback on the approach.

There are some interesting recent trends that make doing this particularly important:

If you’re interested, join our group or contact Anil Madhavapeddy directly. At this stage, you need desire and the ability to hack code, but things are settling down over the next few months...