Unikernels

While working on Personal Containers in late 2008, I had a need to run lots of distributed edge nodes holding personal data. The state of computer security is generally a disaster when it comes to leaving software unupgraded for even a few months, so building robust infrastructure that normal people could use was proving quite difficult. Meanwhile, my PhD research in building Functional Internet had constructed really viable prototypes of network protocols written in pure OCaml, and I'd previously used OCaml industrially in the Xen Virtualisation hypervisor to write lots of system management code.

The Early Days

All of these ideas came crashing together in late 2009 and I decided to have a go at putting together a complete OCaml-based operating system. The adventure began with grabbing the Xen mini-os and the C lwIP stack to provide networking and sqlite for persistent storage, and hacking for a few months until everything booted and was reasonably stable. I then convinced Thomas Gazagnaire (then at Inria) to help me with storage integration with OCaml in “Statically-typed value persistence for ML” and we had a remarkably good prototype that we presented in “Turning down the LAMP: Software Specialisation for the Cloud”.

I wrote up my early thoughts on “Multiscale not Multicore: Efficient Heterogeneous Cloud Computing” to describe this emerging idea of heterogenous cloud and edge computing combined into a single programming model. After realising that the prototype worked well, I started steadily removing C bindings (like lwIP) and replacing them with pure OCaml code all the way down to the VM Xen interface (e.g. like mirage-tcpip). These early heady days saw lots of prototypes and experimentation:

• I experimented with various models for edge computing for personal data handling, such as “Unclouded Vision” and “Using Dust Clouds to Enhance Anonymous Communication”. These mechanisms are still surprisingly unrealised in the wild, with some aspects becoming popular (e.g. serverless functions), but not the aggregation networks.
• In the office next door, Derek Murray and friends were doing their PhDs and building distributed execution engines. I helped with building out “CIEL: a universal execution engine for distributed data-flow computing” and experimenting with what a functional interface would look like in DataCaml - a first look at distributed dataflow programming in OCaml. As of 2021, I'm revisiting this approach in the context of algebraic effects in our multicore OCaml project.
• I looked into closer integration with hypervisors as well, via investigating “Reconfigurable Data Processing for Clouds” (TL;DR -- too early, but happened a few years later in commercial clouds) and “Programming the Xen Cloud using OCaml”.

Building MirageOS and figuring out unikernels

One of the earliest decisions I made in MirageOS was to self-host as soon as possible. I registered openmirage.org in late 2009, and (joined by Richard Mortier and David Scott) we had a Xen-based website running in short order in 2010 (now mirage-www). A big boost to the project was winning a grant from the Verisign Infrastructure Awards, which was the first external validation that this thing might be of interest to other people. As my OCaml Labs group grew in the University, more intrepid hackers joined the group and started making MirageOS work properly.

A year of intense work in 2012 turned the prototype into a fully-fleshed out paper which got soundly rejected by the OSDI review committee as we hadn't identified what the core systems research contribution was (as opposed to the impressive programming work, which they acknowledged in the rejection). I'd just gone to visit Timothy Roscoe's group in ETH where they had been working on the Barrelfish multikernel OS, and the answer came right to me while in the pub with Jon Crowcroft. What MirageOS represented was a revival of the concept of library operating systems, but with the additional twist that it specialised the compilation into single-user mode. Thus, I settled on the term "unikernels" to describe this idea and rewrote the paper and duly published it in “Unikernels: Library Operating Systems for the Cloud”.

Publishing a major research paper in ASPLOS led to further momentum and interest:

• David Scott and I published a note in the Communications of the ACM dubbed “Unikernels: The Rise of the Virtual Library Operating System” which was pretty widely read at the time.
• Thomas Gazagnaire moved to Cambridge and started building the storage stack that we'd wanted for years. It was initially called “Irminsule: a branch-consistent distributed library database” (later shortened to irmin) and kicked off our interest in moving beyond CRDTs to “Mergeable Persistent Data Structures”. Irmin picked up a life of its own and was later used by Arthur Breitman as the storage stack in the Tezos proof-of-stack blockchain in 2017.
• Magnus Skjegstad also returned to the group and we began hacking on real-time edge infrastructure using unikernels, such as “Kadupul: Livin' on the Edge with Virtual Currencies and Time-Locked Puzzles”. Although this work got put on ice in 2015, I'm revisiting it in 2022 in the context of Interspatial OS.
• Thomas Leonard, David Sheets and Balraj Singh joined our burgeoning group and we all prototyped the idea of real-time booting of edge unikernels in “Jitsu: Just-In-Time Summoning of Unikernels”. This represented the first time we'd booted VMs on ARM, as it was very much a niche architecture for virtualisation back then.
• Meanwhile, in the beach in Mirleft in Morrocco, David Kaloper-Meršinjak and Hannes Menhert built an entire TLS stack in OCaml which we published in “Not-quite-so-broken TLS: lessons in re-engineering a security protocol specification and implementation”. This was a real turning point in the project as it represented an external open source contribution (with both of them joining the University subsequently) and also grew our belief that it wasn't a completely dumb idea to rebuild every Internet protocol in a functional language.

MirageOS also gave us ideas for other top systems research, such as the filesystem verification idas in “SibylFS: formal specification and oracle-based testing for POSIX and real-world file systems” (which I still intend to use for a proper POSIX compatibility layer on top of Irmin at some point), and “FLICK: Developing and Running Application-Specific Network Services” (to build domain-specific data processing platforms, something that I'm now working on in 2021 in uctrees).

To Unikernel Systems and Docker

By this point, MirageOS was also a thriving open source community with regular IRC meetings and the beginning of hack retreats. There were several organisations using it, and the overall OCaml community started using some of our protocol implementations independently of the unikernel ideas. For example, the cohttp was something I rapidly hacked together for the ASPLOS deadline, but the Unix/Lwt/Async backends are now used in quite a few major systems (including within Jane Street, no less).

We had to deal with all this growth, as a university isn't the easiest place to have a very large group. In 2015, Balraj Singh (who had made huge contributions to the Mirage TCP/IP stack) Thomas Gazagnaire and myself founded Unikernel Systems along with Jeremy Yallop, Thomas Leonard, Magnus Skjegstad, Mindy Preston, Justin Cormack, David Sheets, Amir Chaudhry, and David Scott. After a fun few months pitching to west coast VCs in California (including fun chats with the likes of Jerry Yang), Peter Fenton from Benchmark convinced us to meet Solomon Hykes over at Docker. This conversation changed the course of our careers, as he shared his vision for the future of containerisation and how unikernels could fit in there.

A short set of negotiations later, and Unikernel Systems was acquired by Docker in 2016. We spent a very fun couple of years commercialising the technology and incorporating it into Docker for Desktop. Our work ended up shipping as Docker for Desktop which remains one of the most popular developer tools in the world, and I describe its architecture in this talk.

Unikernels in 2021

Our startup aside, the core development of MirageOS continued to be nicely distributed in several spinouts:

• KC Sivaramakrishnan and Gemma Gordon founded OCLC in 2016 as a commercial spinout from the university group to drive OCaml tooling and core compiler development.
• Hannes Menhert setup the <robur.io> cooperative in late 2017 with a large set of Mirage projects.
• Thomas Gazagnaire founded Tarides in 2018 after leaving Docker, where they maintainM irageOS and drive development of the Irmin storage stack in particular.

The wider industry also saw a number of interesting spinouts, as many other communities also latched on to the ideas of unikernels and began their own language-specific and domain-specific versions. I joined the advisory boards of IncludeOS (now sadly defunct) and Zededa (now thankfully going from strength to strength in edge computing) to help guide strategy and adoption outside of just MirageOS. Dr Pierre Oliver maintains a great list of unikernel papers where you can see the diversity and interest in unikernels. One of the most exciting implementations of a C-based unikernel can be found in Unikraft.

As for my interest in unikernels moving forward? My heart always remains in finding the intersection of safety and performance, which means I mostly pay attention to language-based approaches. MirageOS continues to thrive (particularly with the effect system being integrated into OCaml in 2022, which will really change the way we develop OCaml code for embedded systems). Since 2020, I've been investigating the application of DIFC to embedded infrastructure, for example via “Snape: The Dark Art of Handling Heterogeneous Enclaves”.