< back to projects

Unikernels

(2010 - 2019)

I proposed the concept of unikernels -- specialised machine images constructed by using library operating system architectures and safer programming languages such as OCaml. I also co-founded the MirageOS project which is a complete unikernel framework written in pure OCaml.

While working on project iconPersonal Containers in late 2008, I had a need to run lots of distributed edge nodes holding personal data. The state of computer security is generally a disaster when it comes to leaving software unupgraded for even a few months, so building robust infrastructure that normal people could use was proving quite difficult. Meanwhile, my PhD research in building project iconFunctional Internet had constructed really viable prototypes of network protocols written in pure OCaml, and I'd previously used OCaml industrially in the project iconXen Virtualisation hypervisor to write lots of system management code.

The Early Days

All of these ideas came crashing together in late 2009 and I decided to have a go at putting together a complete OCaml-based operating system. The adventure began with grabbing the Xen mini-os and the C lwIP stack to provide networking and sqlite for persistent storage, and hacking for a few months until everything booted and was reasonably stable. I then convinced person iconThomas Gazagnaire (then at Inria) to help me with storage integration with OCaml in 2010-dyntype-wgt and we had a remarkably good prototype that we presented in 2010-hotcloud-lamp.

I wrote up my early thoughts on 2010-bcs-visions to describe this emerging idea of heterogenous cloud and edge computing combined into a single programming model. After realising that the prototype worked well, I started steadily removing C bindings (like lwIP) and replacing them with pure OCaml code all the way down to the VM Xen interface (e.g. like mirage-tcpip). These early heady days saw lots of prototypes and experimentation:

  • I experimented with various models for edge computing for personal data handling, such as 2011-icdcn-droplets and 2010-iswp-dustclouds. These mechanisms are still surprisingly unrealised in the wild, with some aspects becoming popular (e.g. serverless functions), but not the aggregation networks.
  • In the office next door, Derek Murrayperson icon and friends were doing their PhDs and building distributed execution engines. I helped with building out 2011-nsdi-ciel and experimenting with what a functional interface would look like in project iconDataCaml - a first look at distributed dataflow programming in OCaml. As of 2021, I'm revisiting this approach in the context of algebraic effects in our multicore OCaml project.
  • I looked into closer integration with hypervisors as well, via investigating 2011-fccm-cloudfpga (TL;DR -- too early, but happened a few years later in commercial clouds) and 2012-oud-xen.

Building MirageOS and figuring out unikernels

One of the earliest decisions I made in MirageOS was to self-host as soon as possible. I registered openmirage.org in late 2009, and (joined by person iconRichard Mortier and headshotDavid Scott) we had a Xen-based website running in short order in 2010 (now mirage-www). A big boost to the project was winning a grant from the Verisign Infrastructure Awards, which was the first external validation that this thing might be of interest to other people. As my project iconOCaml Labs group grew in the University, more intrepid hackers joined the group and started making MirageOS work properly.

A year of intense work in 2012 turned the prototype into a fully-fleshed out paper which got soundly rejected by the OSDI review committee as we hadn't identified what the core systems research contribution was (as opposed to the impressive programming work, which they acknowledged in the rejection). I'd just gone to visit Timothy Roscoe's group in ETH where they had been working on the Barrelfish multikernel OS, and the answer came right to me while in the pub with Jon Crowcroftperson icon. What MirageOS represented was a revival of the concept of library operating systems, but with the additional twist that it specialised the compilation into single-user mode. Thus, I settled on the term "unikernels" to describe this idea and rewrote the paper and duly published it in 2013-asplos-mirage.

Publishing a major research paper in ASPLOS led to further momentum and interest:

  • headshotDavid Scott and I published a note in the Communications of the ACM dubbed rise-of-libos which was pretty widely read at the time.
  • person iconThomas Gazagnaire moved to Cambridge and started building the storage stack that we'd wanted for years. It was initially called 2014-oud-irminsule (later shortened to irmin) and kicked off our interest in moving beyond CRDTs to 2015-jfla-irmin. Irmin picked up a life of its own and was later used by Arthur Breitman as the storage stack in the Tezos proof-of-stack blockchain in 2017.
  • Magnus Skjegstadperson icon also returned to the group and we began hacking on real-time edge infrastructure using unikernels, such as 2015-diynet-kadupul. Although this work got put on ice in 2015, I'm revisiting it in 2022 in the context of project iconInterspatial OS.
  • Thomas Leonardperson icon, David Sheetsperson icon and Balraj Singhperson icon joined our burgeoning group and we all prototyped the idea of real-time booting of edge unikernels in 2015-nsdi-jitsu. This represented the first time we'd booted VMs on ARM, as it was very much a niche architecture for virtualisation back then.
  • Meanwhile, in the beach in Mirleft in Morrocco, David Kaloper-MerÅ¡injakperson icon and Hannes Menhertperson icon built an entire TLS stack in OCaml which we published in 2015-usenixsec-nqsb. This was a real turning point in the project as it represented an external open source contribution (with both of them joining the University subsequently) and also grew our belief that it wasn't a completely dumb idea to rebuild every Internet protocol in a functional language.

MirageOS also gave us ideas for other top systems research, such as the filesystem verification idas in 2015-sosp-sibylfs (which I still intend to use for a proper POSIX compatibility layer on top of Irmin at some point), and 2016-usenix-flick (to build domain-specific data processing platforms, something that I'm now working on in 2021 in uctrees).

To Unikernel Systems and Docker

By this point, MirageOS was also a thriving open source community with regular IRC meetings and the beginning of hack retreats. There were several organisations using it, and the overall OCaml community started using some of our protocol implementations independently of the unikernel ideas. For example, the cohttp was something I rapidly hacked together for the ASPLOS deadline, but the Unix/Lwt/Async backends are now used in quite a few major systems (including within Jane Street, no less).

We had to deal with all this growth, as a university isn't the easiest place to have a very large group. In 2015, Balraj Singhperson icon (who had made huge contributions to the Mirage TCP/IP stack) person iconThomas Gazagnaire and myself founded Unikernel Systems along with Jeremy Yallopperson icon, Thomas Leonardperson icon, Magnus Skjegstadperson icon, Mindy Prestonperson icon, Justin Cormackperson icon, David Sheetsperson icon, Amir Chaudhryperson icon, and headshotDavid Scott. After a fun few months pitching to west coast VCs in California (including fun chats with the likes of Jerry Yang), Peter Fenton from Benchmark convinced us to meet Solomon Hykes over at Docker. This conversation changed the course of our careers, as he shared his vision for the future of containerisation and how unikernels could fit in there.

A short set of negotiations later, and Unikernel Systems was acquired by Docker in 2016. We spent a very fun couple of years commercialising the technology and incorporating it into Docker for Desktop. Our work ended up shipping as Docker for Desktop which remains one of the most popular developer tools in the world, and I describe its architecture in this talk.

Unikernels in 2021

Our startup aside, the core development of MirageOS continued to be nicely distributed in several spinouts:

  • KC Sivaramakrishnanperson icon and Gemma Gordonperson icon founded OCLC in 2016 as a commercial spinout from the university group to drive OCaml tooling and core compiler development.
  • Hannes Menhertperson icon setup the <robur.io> cooperative in late 2017 with a large set of Mirage projects.
  • person iconThomas Gazagnaire founded Tarides in 2018 after leaving Docker, where they maintainM irageOS and drive development of the Irmin storage stack in particular.

The wider industry also saw a number of interesting spinouts, as many other communities also latched on to the ideas of unikernels and began their own language-specific and domain-specific versions. I joined the advisory boards of IncludeOS (now sadly defunct) and Zededa (now thankfully going from strength to strength in edge computing) to help guide strategy and adoption outside of just MirageOS. Dr Pierre Oliver maintains a great list of unikernel papers where you can see the diversity and interest in unikernels. One of the most exciting implementations of a C-based unikernel can be found in Unikraft.

As for my interest in unikernels moving forward? My heart always remains in finding the intersection of safety and performance, which means I mostly pay attention to language-based approaches. MirageOS continues to thrive (particularly with the effect system being integrated into OCaml in 2022, which will really change the way we develop OCaml code for embedded systems). Since 2020, I've been investigating the application of DIFC to embedded infrastructure, for example via 2019-edgesys-snape.

Related publications

Related projects

2009 - 2015 project iconPersonal Containers
2003 - 2008 project iconFunctional Internet
2002 - 2009 project iconXen Virtualisation
2012 - 2021 project iconOCaml Labs