As cloud computing empowered the creation of vast data silos, I investigated how decentralised technologies might be deployed to allow individuals more vertical control over their own data. Personal containers was the prototype we built to learn how to stem the flow of our information out to the ad-driven social tarpits.
I've had a passion for self-hosted, decentralised computing for many years
since Nick Ludlam and I set up the recoil.org collective in the late 90s. In
late 2008, I'd been working on early cloud computing as part of the
Xen Virtualisation
project and already seeing the rapid rise of centralised data gathering in
the early cloud providers. When I left Citrix in 2009, I joined Derek McCauley
and
Jon Crowcroft
in their new Horizon Digital Economy
centre to lead a charge into building more privacy-centred digital infrastructure.
I had the huge privilege of receiving a strings-free 5-year postdoctoral fellowship in
Cambridge. It's rare to see such long term postdoc opportunities these days, but
something I am hugely supportive of for new projects.
My hacking first began with Nick Ludlam in 2008 on a prototype of a
lifedb server and app, which we
envisioned as a place to aggregate all the messages from disparate sources (for
example, to mirror the then-new Twitter service into my IMAP email). I worked
on 2010-smarte-privacybutler to add a policy engine to this prototype.
While the prototype worked well enough for me, it was largely a negative result
since it was just too risky to put all that private data in one location
(especially aggregated).
Now back at Cambridge in 2010, I began working with Thomas Gazagnaire on a more robust
implementation of data aggregation that would have stronger end-to-end security
and privacy. We started coding up an implementation in OCaml to followup
my
Functional Internet work, and built out infrastructure like an OCaml ORM in
2010-dyntype-wgt to make it easier to work with databases. It became
obvious pretty quickly that having this much data in one place required
end users to become sysadmins, and so I started to lay out a new architecture
for this sort of end-user managed data in 2010-bcs-visions.
Our first prototype of a personal container running as a unikernel was published in 2010-hotcloud-lamp, and would form the basis of the MirageOS project. To this day, the MirageOS community remains passionate about decentralised systems from these origins! We explored a number of directions in the early days:
One of the main drivers for personal containers was to drive applications that would otherwise be too invasive from a privacy perspective. Ian M. Leslie and I worked on the "c-aware" project in 2012-mpm-caware to figure out if personal containers could help influence user behaviour to reduce carbon usage. Overall, this project taught us just how much effort it would be to deploy real-world infrastructure in corporate environments like the University of Cambridge. We also struggled to get any users to deploy our prototype servers, something explored more in user studies with colleagues in Horizon Nottingham in de13-dataware.
My work on personal data processing petered out from a research perspective in around 2013 since the underlying infrastructure I had built really started gathering steam with Unikernels and
OCaml Labs. We hadn't quite cracked the problem of how to break the cloud hegemony, but (as with XenoServers and Xen), the pieces that succeeded emerged from the research questions we asked.
However, I don't consider this project permanently closed by any means -- after all, I've been self hosting my email since 1997! We've been working steadily over the past decade of MirageOS (as of 2021) to build out a really solid, self-hosted protocol stack that will work as a unikernel. I am revisiting the question of decentralisation in the form of physical infrastructure in the
Interspatial OS project, and you can read my early thoughts in 2018-hotpost-osmose.