Summary. I proposed the concept of "unikernels" -- single-purpose appliances that are compile-time specialised into standalone bootable kernels, and sealed against modification when deployed to a cloud platform. In return they offer significant reduction in image sizes, improved efficiency and security, and reduce operational costs. I also co-founded the MirageOS project which is one of the first complete unikernel frameworks, and also integrated them to create the Docker for Desktop apps that are used by hundreds of millions of users daily.
While working on Personal Containers in late 2008, I had a need to run lots of distributed edge nodes holding personal data. The state of computer security is generally a disaster when it comes to leaving software unupgraded for even a few months, so building robust infrastructure that normal people could use was proving quite difficult. Meanwhile, my PhD research in building Functional Internet Services had constructed really viable prototypes of network protocols written in pure OCaml, and I'd previously used OCaml industrially in the Xen Hypervisor hypervisor to write lots of system management code.
The Early Days
All of these ideas came crashing together in late 2009 and I decided to have a go at putting together a complete OCaml-based operating system. The adventure began with grabbing the Xen mini-os and the C lwIP stack to provide networking and sqlite for persistent storage, and hacking for a few months until everything booted and was reasonably stable. I then convinced Thomas Gazagnaire (then at Inria) to help me with storage integration with OCaml in Dynamics for ML using Meta-Programming and we had a remarkably good prototype that we presented in Turning Down the LAMP: Software Specialisation for the Cloud.
I wrote up my early thoughts on Multiscale not multicore: efficient heterogeneous cloud computing to describe this emerging idea of heterogenous cloud and edge computing combined into a single programming model. After realising that the prototype worked well, I started steadily removing C bindings (like lwIP) and replacing them with pure OCaml code all the way down to the VM Xen interface (e.g. like mirage-tcpip). These early heady days saw lots of prototypes and experimentation:
- I experimented with various models for edge computing for personal data handling, such as Unclouded vision and Using Dust Clouds to Enhance Anonymous Communication. These mechanisms are still surprisingly unrealised in the wild, with some aspects becoming popular (e.g. serverless functions), but not the aggregation networks.
- In the office next door, @mrry and friends were doing their PhDs and building distributed execution engines. I helped with building out CIEL: A universal execution engine for distributed data-flow computing and experimenting with what a functional interface would look like in DataCaml: distributed dataflow programming in OCaml. As of 2021, I'm revisiting this approach in the context of algebraic effects in our multicore OCaml project.
- I looked into closer integration with hypervisors as well, via investigating Reconfigurable Data Processing for Clouds (TL;DR -- too early, but happened a few years later in commercial clouds) and Programming the Xen cloud using OCaml.
Building MirageOS and figuring out unikernels
One of the earliest decisions I made in MirageOS was to self-host as soon as possible. I registered openmirage.org in late 2009, and (joined by @mort and @djs55) we had a Xen-based website running in short order in 2010 (now mirage-www). A big boost to the project was winning a grant from the Verisign Infrastructure Awards, which was the first external validation that this thing might be of interest to other people. As my OCaml Labs group grew in the University, more intrepid hackers joined the group and started making MirageOS work properly.
A year of intense work in 2012 turned the prototype into a fully-fleshed out paper which got soundly rejected by the OSDI review committee as we hadn't identified what the core systems research contribution was (as opposed to the impressive programming work, which they acknowledged in the rejection). I'd just gone to visit Timothy Roscoe's group in ETH where they had been working on the Barrelfish multikernel OS, and the answer came right to me while in the pub with Jon Crowcroft. What MirageOS represented was a revival of the concept of library operating systems, but with the additional twist that it specialised the compilation into single-user mode. Thus, I settled on the term "unikernels" to describe this idea and rewrote the paper and duly published it in Unikernels: library operating systems for the cloud.
Publishing a major research paper in ASPLOS led to further momentum and interest:
- Dave Scott and I published a note in the Communications of the ACM dubbed Unikernels: Rise of the Virtual Library Operating System which was pretty widely read at the time.
- Thomas Gazagnaire moved to Cambridge and started building the storage stack that we'd wanted for years. It was initially called Irminsule: a branch-consistent distributed library database (later shortened to irmin) and kicked off our interest in moving beyond CRDTs to Mergeable persistent data structures. Irmin picked up a life of its own and was later used by Arthur Breitman as the storage stack in the Tezos proof-of-stack blockchain in 2017.
- Magnus Skjegstad also returned to the group and we began hacking on real-time edge infrastructure using unikernels, such as Kadupul: Livin' on the Edge with Virtual Currencies and Time-Locked Puzzles. Although this work got put on ice in 2015, I'm revisiting it in 2022 in the context of Interspatial OS.
- Thomas Leonard, David Sheets and Balraj Singh joined our burgeoning group and we all prototyped the idea of real-time booting of edge unikernels in Jitsu: Just-In-Time Summoning of Unikernels. This represented the first time we'd booted VMs on ARM, as it was very much a niche architecture for virtualisation back then.
- Meanwhile, in the beach in Mirleft in Morrocco, David Kaloper-Meršinjak and Hannes Mehnert built an entire TLS stack in OCaml which we published in Not-Quite-So-Broken TLS. This was a real turning point in the project as it represented an external open source contribution (with both of them joining the University subsequently) and also grew our belief that it wasn't a completely dumb idea to rebuild every Internet protocol in a functional language.
MirageOS also gave us ideas for other top systems research, such as the filesystem verification idas in SibylFS: formal specification and oracle-based testing for POSIX and real-world file systems (which I still intend to use for a proper POSIX compatibility layer on top of Irmin at some point), and FLICK: Developing and Running Application-Specific Network Services (to build domain-specific data processing platforms, something that I'm now working on in 2021 in Trusted Carbon Credits).
To Unikernel Systems and Docker
By this point, MirageOS was also a thriving open source community with regular IRC meetings and the beginning of hack retreats. There were several organisations using it, and the overall OCaml community started using some of our protocol implementations independently of the unikernel ideas. For example, the cohttp was something I rapidly hacked together for the ASPLOS deadline, but the Unix/Lwt/Async backends are now used in quite a few major systems (including within Jane Street, no less).
We had to deal with all this growth, as a university isn't the easiest place to have a very large group. In 2015, Balraj Singh (who had made huge contributions to the Mirage TCP/IP stack) Thomas Gazagnaire and myself founded Unikernel Systems along with Jeremy Yallop, Thomas Leonard, Magnus Skjegstad, Mindy Preston, Justin Cormack, David Sheets, Amir Chaudhry, and Dave Scott. After a fun few months pitching to west coast VCs in California (including fun chats with the likes of Jerry Yang), Peter Fenton from Benchmark convinced us to meet Solomon Hykes over at Docker. This conversation changed the course of our careers, as he shared his vision for the future of containerisation and how unikernels could fit in there.
A short set of negotiations later, and Unikernel Systems was acquired by Docker in 2016. We spent a very fun couple of years commercialising the technology and incorporating it into Docker for Desktop. Our work ended up shipping as Docker for Desktop which remains one of the most popular developer tools in the world, and I describe its architecture in this talk.
Unikernels in 2021
Our startup aside, the core development of MirageOS continued to be nicely distributed in several spinouts:
- KC Sivaramakrishnan and Gemma Gordon founded OCLC in 2016 as a commercial spinout from the university group to drive OCaml tooling and core compiler development.
- Hannes Mehnert setup the <robur.io> cooperative in late 2017 with a large set of Mirage projects.
- Thomas Gazagnaire founded Tarides in 2018 after leaving Docker, where they maintain MirageOS and drive development of the Irmin storage stack in particular.
The wider industry also saw a number of interesting spinouts, as many other communities also latched on to the ideas of unikernels and began their own language-specific and domain-specific versions. I joined the advisory boards of IncludeOS (now sadly defunct) and Zededa (now thankfully going from strength to strength in edge computing) to help guide strategy and adoption outside of just MirageOS. Dr Pierre Oliver maintains a great list of unikernel papers where you can see the diversity and interest in unikernels. One of the most exciting implementations of a C-based unikernel can be found in Unikraft.
As for my interest in unikernels moving forward? My heart always remains in finding the intersection of safety and performance, which means I mostly pay attention to language-based approaches. MirageOS continues to thrive (particularly with the effect system being integrated into OCaml in 2022, which will really change the way we develop OCaml code for embedded systems). Since 2020, I've been investigating the application of DIFC to embedded infrastructure, for example via Snape: The Dark Art of Handling Heterogeneous Enclaves.
Relevant Ideas
- Distributed Task Scheduling Framework over Irmin
by Mohammed Daudali in 2019 (Completed, Part II). - A strongly consistent index for email using git and MirageOS
by Oliver Hope cosupervised with David Allsopp in 2019 (Completed, Part II). - CausalRPC: a traceable distributed computation framework
by Craig Ferguson in 2018 (Completed, Part II). - Consolidating Trust for Client Groups that use TLS to Secure Connections
by Johann Beleites cosupervised with David Sheets in 2014 (Completed, Part II). - Control flow analysis for privilege separation
by Chris Harding and Ross McIlroy cosupervised with Robert M Watson in 2011 (Completed). - Extending 64-bit MIPS support for LLVM
by William Morland cosupervised with Robert M Watson in 2011 (Completed).
Relevant Papers
|
[»] Real World OCaml: Functional Programming for the Masses |  |
|
[»] Banyan: Coordination-Free Distributed Transactions over Mergeable Types |  |
|
[»] MirageOS 4: the dawn of practical build systems for exotic targets |  |
|
[»] Programming Unikernels in the Large via Functor Driven Development |
|
[»] DaLi: Database as a Library |  |
|
[»] FLICK: Developing and Running Application-Specific Network Services |  |
|
[»] Declarative Foreign Function Binding Through Generic Programming |  |
|
[»] Personal Data: Thinking Inside the Box |  |
|
[»] SibylFS: formal specification and oracle-based testing for POSIX and real-world file systems |  |
|
[»] Not-Quite-So-Broken TLS |  |
|
[»] Jitsu: Just-In-Time Summoning of Unikernels |  |
|
[»] Mergeable persistent data structures |  |
|
[»] Raft Refloated: Do We Have Consensus? |  |
|
[»] Irminsule: a branch-consistent distributed library database |  |
|
[»] Using Dust Clouds to Enhance Anonymous Communication |  |
|
[»] Trevi: watering down storage hotspots with cool fountain codes |  |
|
[»] Unikernels: Rise of the Virtual Library Operating System |
|
[»] Lost in the Edge: Finding Your Way with DNSSEC Signposts |  |
|
[»] Unikernels: library operating systems for the cloud |  |
|
[»] Evolving TCP: how hard can it be? |  |
|
[»] Exploring Compartmentalisation Hypotheses with SOAAP |  |
|
[»] Programming the Xen cloud using OCaml |  |
|
[»] Cost, Performance & Flexibility in OpenFlow: Pick three |  |
|
[»] Confidential carbon commuting: exploring a privacy-sensitive architecture for incentivising 'greener' commuting |  |
|
[»] The case for reconfigurable I/O channels |  |
|
[»] Dynamics for ML using Meta-Programming |  |
|
[»] Reconfigurable Data Processing for Clouds |  |
|
[»] CIEL: A universal execution engine for distributed data-flow computing |  |
|
[»] Unclouded vision |  |
|
[»] Turning Down the LAMP: Software Specialisation for the Cloud |  |
|
[»] Multiscale not multicore: efficient heterogeneous cloud computing |  |
Relevant Talks
Rebuilding Operating Systems with Functional Principles on 26th Feb 2020. Part 1 of my distinguished lecture series at St Andrews. Part of the Unikernels project.
The First Billion Real Deployments of Unikernels on 26th Feb 2020. Part 2 of my distinguished lecture series at St Andrews. Part of the Unikernels project.
Programming the Next Trillion Embedded Devices on 26th Feb 2020. Part 3 of my distinguished lecture series at St Andrews. Part of the Unikernels project.
Unikernels: the rise of the library hypervisor in MirageOS on 14th Oct 2016. At DockerCon. Part of the Unikernels project.
Ian Eyberg, Joshua Bernstein, Anil Madhavapeddy at OSCON in Austin on 6th Jun 2016. Speaking to Alex Williams of The New Stack. Part of the Unikernels project.
Unikernel Systems is now part of Docker on 21st Jan 2016. Announcing the acquisition of Unikernel Systems by Docker. Part of the Unikernels project.
Immutable Distributed Infrastructure with Unikernels on 29th Sep 2015. Invited talk at NetPL 2015 on immutable infrastructure. Part of the Unikernels project.
Unikernels: Functional Infrastructure with Mirage OS on 12th May 2015. At Esper's OCaml Meetup in California. Part of the Unikernels project.
BOB 2015 Keynote: Towards Functional Operating Systems on 23rd Jan 2015. Opening speaker at BOB 2015 on functional operating systems. Part of the Unikernels project.
Jitsu: Just-in-Time Summoning of Unikernels (new directions in operating systems) on 25th Nov 2014. Talking in London at New Directions in Operating Systems. Read more in Jitsu: Just-In-Time Summoning of Unikernels. Part of the Unikernels project.
MirageOS 2.0: branch consistency for Xen Stub Domains on 17th Oct 2014. At the Xen Summit explaining how Irmin+MirageOS can build stub domains for Xen hosts. Part of the Unikernels project.
Haskell Symposium 2014 Keynote on functional OS design on 5th Sep 2014. Me slightly nervously talking to a bunch of Haskellers about OCaml modules. Read more in Unikernels: library operating systems for the cloud. Part of the Unikernels project.
FLOSS Weekly 302: Open Mirage on 23rd Jul 2014. Randal Schwartz and Simon Phipps interview Anil Madhavapeddy about MirageOS on the FLOSS Weekly podcast (Episode 302). Part of the Unikernels project.
SE Radio Episode 204: Anil Madhavapeddy on the Mirage Cloud Operating System and the OCaml Language on 1st May 2014. I talk to Robert Blumen about OCaml and MirageOS on the Software Engineering Radio podcast. Part of the Unikernels project.
MirageOS and XAPI project update at XenSummit on 13th Nov 2013. Announcing the MirageOS 1.0 release at the Xen Summit 2013. Part of the Unikernels project.
Mirage Developer Preview 1 screencast on 26th Jul 2013. The MirageOS Developer Preview 1 screencast, showing how to initialise and build Unix and Xen kernels. Part of the Unikernels project.
OCaml Meeting 2011 - MirageOS on 19th Oct 2011. The OCaml Meeting 2011 talk on MirageOS in France, with full notes in Camel Spotting in Paris. Part of the Unikernels project.
Mirage: A New Multi-Scale Operating System for Clouds and Crowds (2014) on 25th Oct 2010. An early tech talk at LinkedIn on MirageOS. Part of the Unikernels project.
Turning Down the LAMP: Software Specialisation for the Cloud on 22nd Jun 2010. The first outing for MirageOS at USENIX HotCloud 2010. Read more in Turning Down the LAMP: Software Specialisation for the Cloud. Part of the Unikernels project.
News Updates
» The second edition of Real World OCaml available from Cambridge University Press!» Paper on Banyan for coordination-free distributed transactions in ASPLAS 2020.
» Part 3 / «» Part 2 / «» Delivered the distinguished seminar series at St Andrews on rebuilding Operating Systems with functional principles.» Paper on the MirageOS 4 build system at OCaml Workshop.» Preprint on programming unikernels with ML modules.» Position paper on building databases-as-a-library at SNAPL 2017.» DockerCon talk on unikernels and MirageOS.» Paper on application-specific network services at USENIX ATC 2016 / «» Interviewed by The New Stack at OSCON in Austin, Texas / «
» Updated the MirageOS community about the 2016 hack retreat.» Announced HyperKit, VPNKit and DataKit for the Docker ecosystem.» Paper on declarative approaches to foreign function bindings at FLOPS 2016.» Announced that Unikernel Systems is now part of Docker.» Paper on formal specificaiton and testing of filesystems at SOSP 2015 / «» Paper on personal databoxes at the one-in-a-decade Aarhus conference.» Invited talk at NetPL on Immutable Distributed Infrastructure with Unikernels.» Paper on rebuilding TLS securely but practically at USENIX Security 2015.» Paper on spinning up low-latency unikernels per-connection at NSDI 2015 / «» Talk at Esper on functional programming with unikernels.» Updated the OCaml community on annual OCaml Labs activities for 2014.» Paper on reproducing the raft consensus protocol / «» Paper on mergeable data structures using Irmin (nee Irminsule) at JFLA 2015 / «» Gave BOB 2015 keynote on functional Operating Systems.» New Directions in Operating Systems talk on Jitsu.» At the Xen Summit speaking about branch consistency for Xen Stub Domains.» Paper at the OCaml Workshop on the Irmin database library / «» Gave Haskell Symposium 2014 Keynote on functional OS design.» Announced the release of MirageOS 1.2, and a roadmap towards MirageOS 2.0 / «» Announced the long-awaited release of MirageOS 2.0 / «» Appeared on FLOSS Weekly 302 about Open Mirage.» Appeared on SE Radio Episode 204 about Mirage and OCaml.» Paper on building dust clouds for anonymous communication.» Note on the discussions around my recent CACM article on unikernels.» Article on the Communications of the ACM on unikernels is published / «» The 1st Edition of Real World OCaml by O'Reilly associates is released! / «» Paper on fountain coding for datacentre networking at HotNets 2013 / «» MirageOS and XAPI project update at XenSummit / «» Note on integrating Docker and opam more effectively.» Paper on DNSSEC-based Signpost servers for better p2p communications at USENIX FOCI.» Mirage Developer Preview 1 screencast.» The first paper on unikernels is pubished at ASPLOS 2013.» Paper on extending TCP in a backwards compatible way at CoNeXT 2013.» Discussing how we disaggregated MirageOS into opam packages.» Paper on programming the Xen cloud using OCaml at the OCaml Workshop / «» Paper on control flow analysis to break up applications into compartments / «» Note on how to build XenStore stub domains using MirageOS.» Paper on using MirageOS for better SDN infrastructure with OpenFlow.» Paper on our use of data lockers within Cambridge to incentivise more green commuting patterns.» Paper on a new design for reconfigurable IO that copes with heterogenous software/hardware.» Note on using ARM Dreamplugs with OCaml published.» At the OCaml Meeting 2011 speaking about MirageOS.» Liveblog on the talks at CUFP 2011.» Published dyntype at the Workshop on Generative Technologies.» Note discussing an OCaml interface to our new CIEL dataflow engine / «» Published a note on delimited continuations vs Lwt in OCaml for MirageOS.» Paper on what a Xen+FPGA cloud would look like at FCCM.» Paper on CIEL, a distributed dataflow engine, at USENIX NSDI 2011.» Paper on a vision for a semi-federated cloud for personal data at ICDCN.» Announced that the MirageOS website now self-hosted on MirageOS! / «» At LinkedIn giving tech talk about Mirage.» Workshop paper on the early MirageOS architecture and evaluation at HotCloud 2010 / «» At HotCloud for the first talk about MirageOS.» Paper on our vision for multiscale programming at the BCS Visions 2010 conference.