The App Engine data collector for Personal Containers is coming on nicely, and is on track for an alpha preview release fairly soon. Working with AppEngine has been interesting; it’s got excellent availability and you can’t beat the price (free), but coding robust Python that doesn’t trip over the tight resource limits for individual requests, asynchronous tasks and queries is tricky. While it is good for small records such as my iPhone or Find My iPhone GPS traces traces, it doesn’t work so well with my gigabytes of photographs or decades of e-mail.

This confirmed our earlier intuition that there is no one perfect solution for personal data handling; instead, we need to embrace diversity and construct an infrastructure that can cope with change over the coming decades. Mobile programming has changed beyond recognition in just a few years, and cloud providers are specialising in different ways (e.g. PiCloud for simple compute, or EC2 for fancy services like elastic load balancing).

So to recognise this, we are building components that all interoperate with your personal data, keep it secure, and ensure it persists for more than a few years. Malte Schwarzkopf came up with the term "digital yurts", and it's stuck. We’ve written a draft paper about it, and would love to hear your comments and feedback on the approach.

There are some interesting recent trends that make doing this particularly important:

• The New York Times wrote about the data-driven life increasingly influencing our decision making. Current sensor data such as GPS traces are just harbringers for the privacy disaster that would be information such as heart rates or your consumption habits getting into the public domain. (link via Derek Murray).
• Facebook has announced a brand new API platform to get access to your information. The EFF has a fantastic timeline of Facebook’s Eroding Privacy over the last five years, to demonstrate how unsafe it is to trust your data to any third-party. We’ve started developing an information dump plugin for Facebook, but the API just changed mid-way and so it has to be started again (volunteers welcome!).
• In the UK, the Digital Economy Act is an extremely controversial act that makes anonymity and privacy all the more important. We’re assembling an open-source dust cloud that integrates Tor into personal containers to automatically grant you anonymity as you communicate with your friends.

If you’re interested, join our group or contact Anil Madhavapeddy directly. At this stage, you need desire and the ability to hack code, but things are settling down over the next few months...