< back to projects

Unikernels

(2010 - 2019)

I proposed the concept of unikernels -- specialised machine images constructed by using library operating system architectures and safer programming languages such as OCaml. I also co-founded the MirageOS project which is a complete unikernel framework written in pure OCaml.

While working on Personal Containers in late 2008, I had a need to run lots of distributed edge nodes holding personal data. The state of computer security is generally a disaster when it comes to leaving software unupgraded for even a few months, so building robust infrastructure that normal people could use was proving quite difficult. Meanwhile, my PhD research in building Functional Internet had constructed really viable prototypes of network protocols written in pure OCaml, and I'd previously used OCaml industrially in the Xen Virtualisation hypervisor to write lots of system management code.

The Early Days

All of these ideas came crashing together in late 2009 and I decided to have a go at putting together a complete OCaml-based operating system. The adventure began with grabbing the Xen mini-os and the C lwIP stack to provide networking and sqlite for persistent storage, and hacking for a few months until everything booted and was reasonably stable. I then convinced Thomas Gazagnaire (then at Inria) to help me with storage integration with OCaml in “Statically-typed value persistence for ML” and we had a remarkably good prototype that we presented in “Turning down the LAMP: Software Specialisation for the Cloud” .

I wrote up my early thoughts on “Multiscale not Multicore: Efficient Heterogeneous Cloud Computing” to describe this emerging idea of heterogenous cloud and edge computing combined into a single programming model. After realising that the prototype worked well, I started steadily removing C bindings (like lwIP) and replacing them with pure OCaml code all the way down to the VM Xen interface (e.g. like mirage-tcpip). These early heady days saw lots of prototypes and experimentation:

Building MirageOS and figuring out unikernels

One of the earliest decisions I made in MirageOS was to self-host as soon as possible. I registered openmirage.org in late 2009, and (joined by Richard Mortier and David Scott) we had a Xen-based website running in short order in 2010 (now mirage-www). A big boost to the project was winning a grant from the Verisign Infrastructure Awards, which was the first external validation that this thing might be of interest to other people. As my OCaml Labs group grew in the University, more intrepid hackers joined the group and started making MirageOS work properly.

A year of intense work in 2012 turned the prototype into a fully-fleshed out paper which got soundly rejected by the OSDI review committee as we hadn't identified what the core systems research contribution was (as opposed to the impressive programming work, which they acknowledged in the rejection). I'd just gone to visit Timothy Roscoe's group in ETH where they had been working on the Barrelfish multikernel OS, and the answer came right to me while in the pub with Jon Crowcroft . What MirageOS represented was a revival of the concept of library operating systems, but with the additional twist that it specialised the compilation into single-user mode. Thus, I settled on the term "unikernels" to describe this idea and rewrote the paper and duly published it in “Unikernels: Library Operating Systems for the Cloud” .

Publishing a major research paper in ASPLOS led to further momentum and interest:

MirageOS also gave us ideas for other top systems research, such as the filesystem verification idas in “SibylFS: formal specification and oracle-based testing for POSIX and real-world file systems” (which I still intend to use for a proper POSIX compatibility layer on top of Irmin at some point), and “FLICK: Developing and Running Application-Specific Network Services” (to build domain-specific data processing platforms, something that I'm now working on in 2021 in uctrees).

To Unikernel Systems and Docker

By this point, MirageOS was also a thriving open source community with regular IRC meetings and the beginning of hack retreats. There were several organisations using it, and the overall OCaml community started using some of our protocol implementations independently of the unikernel ideas. For example, the cohttp was something I rapidly hacked together for the ASPLOS deadline, but the Unix/Lwt/Async backends are now used in quite a few major systems (including within Jane Street, no less).

We had to deal with all this growth, as a university isn't the easiest place to have a very large group. In 2015, Balraj Singh (who had made huge contributions to the Mirage TCP/IP stack) Thomas Gazagnaire and myself founded Unikernel Systems along with Jeremy Yallop , Thomas Leonard , Magnus Skjegstad , Mindy Preston , Justin Cormack , David Sheets , Amir Chaudhry , and David Scott. After a fun few months pitching to west coast VCs in California (including fun chats with the likes of Jerry Yang), Peter Fenton from Benchmark convinced us to meet Solomon Hykes over at Docker. This conversation changed the course of our careers, as he shared his vision for the future of containerisation and how unikernels could fit in there.

A short set of negotiations later, and Unikernel Systems was acquired by Docker in 2016. We spent a very fun couple of years commercialising the technology and incorporating it into Docker for Desktop. Our work ended up shipping as Docker for Desktop which remains one of the most popular developer tools in the world, and I describe its architecture in this talk.

Unikernels in 2021

Our startup aside, the core development of MirageOS continued to be nicely distributed in several spinouts:

  • KC Sivaramakrishnan and Gemma Gordon founded OCLC in 2016 as a commercial spinout from the university group to drive OCaml tooling and core compiler development.
  • Hannes Menhert setup the <robur.io> cooperative in late 2017 with a large set of Mirage projects.
  • Thomas Gazagnaire founded Tarides in 2018 after leaving Docker, where they maintainM irageOS and drive development of the Irmin storage stack in particular.

The wider industry also saw a number of interesting spinouts, as many other communities also latched on to the ideas of unikernels and began their own language-specific and domain-specific versions. I joined the advisory boards of IncludeOS (now sadly defunct) and Zededa (now thankfully going from strength to strength in edge computing) to help guide strategy and adoption outside of just MirageOS. Dr Pierre Oliver maintains a great list of unikernel papers where you can see the diversity and interest in unikernels. One of the most exciting implementations of a C-based unikernel can be found in Unikraft.

As for my interest in unikernels moving forward? My heart always remains in finding the intersection of safety and performance, which means I mostly pay attention to language-based approaches. MirageOS continues to thrive (particularly with the effect system being integrated into OCaml in 2022, which will really change the way we develop OCaml code for embedded systems). Since 2020, I've been investigating the application of DIFC to embedded infrastructure, for example via “Snape: The Dark Art of Handling Heterogeneous Enclaves” .

Related publications

Statically-typed value persistence for ML
Thomas Gazagnaire and Anil Madhavapeddy.
Workshop paper in the Workshop on Generative Technologies (WGT 2010) on Apr 2010 at Paphos, Cyprus.
Turning down the LAMP: Software Specialisation for the Cloud
Anil Madhavapeddy, Richard Mortier, Ripduman Sohan, Thomas Gazagnaire, Steven Hand, Tim Deegan, Derek McCauley and Jon Crowcroft.
Multiscale not Multicore: Efficient Heterogeneous Cloud Computing
Anil Madhavapeddy, Richard Mortier, Jon Crowcroft and Steven Hand.
Conference paper in the ACM/BCS Visions of Computer Science on Apr 2010 at Edinburgh, United Kingdom.
Unclouded Vision
Jon Crowcroft, Anil Madhavapeddy, Malte Schwarzkopf, Theodore Hong and Richard Mortier.
Using Dust Clouds to Enhance Anonymous Communication
Richard Mortier, Anil Madhavapeddy, Theodore Hong, Derek Murray and Malte Schwarzkopf.
Workshop paper in the Eighteenth International Workshop on Security Protocols (IWSP 2010) on Apr 2010 at Cambridge, United Kingdom.
CIEL: a universal execution engine for distributed data-flow computing
Derek Murray, Malte Schwarzkopf, Christopher Smowton, Steven Smith, Anil Madhavapeddy and Steven Hand.
Reconfigurable Data Processing for Clouds
Anil Madhavapeddy and Satnam Singh.
Programming the Xen Cloud using OCaml
David Scott, Anil Madhavapeddy and Richard Mortier.
Workshop paper in the OCaml Users and Developers Workshop (OUD 2012) on Sep 2012 at Copenhagen, Denmark.
Unikernels: Library Operating Systems for the Cloud
Anil Madhavapeddy, Richard Mortier, Charalampos Rotsos, David Scott, Balraj Singh, Thomas Gazagnaire, Steven Smith, Steven Hand and Jon Crowcroft.
Unikernels: The Rise of the Virtual Library Operating System
Anil Madhavapeddy and David Scott.
Journal paper in the Communications of the ACM on Jan 2014.
Irminsule: a branch-consistent distributed library database
Thomas Gazagnaire, Amir Chaudhry, Anil Madhavapeddy, Richard Mortier, David Scott, David Sheets, Gregory Tsipenyuk and Jon Crowcroft.
Workshop paper in the the 4th ACM OCaml Users and Developers Workshop on Sep 2014 at Gothenberg, Sweden.
Mergeable Persistent Data Structures
Benjamin Farinier, Thomas Gazagnaire and Anil Madhavapeddy.
Conference paper in the Vingt-sixièmes Journées Francophones des Langages Applicatifs (JFLA15) on Jan 2015 at Le Val d’Ajol, France.
Kadupul: Livin' on the Edge with Virtual Currencies and Time-Locked Puzzles
Magnus Skjegstad, Anil Madhavapeddy and Jon Crowcroft.
Conference paper in the DIY Networking Workshop at MobiSys 2015 on May 2015 at Florence, Italy.
Jitsu: Just-In-Time Summoning of Unikernels
Anil Madhavapeddy, Thomas Leonard, Magnus Skjegstad, Thomas Gazagnaire, David Sheets, David Scott, Richard Mortier, Amir Chaudhry, Balraj Singh, Jonathan Ludlam, Jon Crowcroft and Ian M. Leslie.
Conference paper in the 12th USENIX Symposium on Networked System Design and Implementation (NSDI 2015) on May 2015 at Oakland, California, USA.
Not-quite-so-broken TLS: lessons in re-engineering a security protocol specification and implementation
David Kaloper-Meršinjak, Hannes Menhert, Anil Madhavapeddy and Peter Sewell.
Conference paper in the 24th USENIX Security Symposium (UseSec15) on Aug 2015 at Washington DC, USA.
SibylFS: formal specification and oracle-based testing for POSIX and real-world file systems
Tom Ridge, David Sheets, Thomas Tuerk, Anil Madhavapeddy, Andrea Giugliano and Peter Sewell.
Conference paper in the 25th ACM Symposium on Operating Systems Principles (SOSP) on Aug 2015 at Monterey, California, USA.
FLICK: Developing and Running Application-Specific Network Services
Abdul Alim, Richard Clegg, Luo Mai, Lukas Rupprecht, Eric Seckler, Paolo Costa, Peter Pietzuch, Alexander L Wolf, Nik Sultana, Jon Crowcroft, Anil Madhavapeddy, Andrew W. Moore, Richard Mortier, Masoud Koleni, Luis Oviedo, Matteo Miliavacca and Derek McCauley.
Conference paper in the 2016 USENIX Annual Technical Conference on Jun 2016 at Denver, USA.
Snape: The Dark Art of Handling Heterogeneous Enclaves
Zahra Tarkhani, Anil Madhavapeddy and Richard Mortier.

Related projects

2009 - 2015 Personal Containers
2003 - 2008 Functional Internet
2002 - 2009 Xen Virtualisation
2012 - 2021 OCaml Labs