Towards a frugal userspace for Linux / Dec 2024
All the work we've been doing on biodiversity (such as LIFE) comes at a fairly large computation and storage cost due to the amount of data that we churn through. This gets worse when you consider the exploratory nature of science -- we sometimes just need to mess around with the large dataset to test hypotheses which are often shown to be wrong. So then, when the LOCO conference came around, we wrote up our thoughts on what a frugal Linux userspace might look like.
The key insight is that the Linux kernel already exposes a number of namespace mechanisms (that we use in Docker, for example), and so we explore a new OS architecture which defaults to deterministic, reusable computation with the careful recording of side-effects. This in turn allows Linux to guide complex computations towards previously acquired intermediate results, but still allowing for recomputation when required by the user. We're putting this together into a new shell known as "Shark", and this first abstract describes our early results.
Void Processes: Minimising privilege by default / Jan 2021
This is an idea proposed as a Cambrige Computer Science Part III or MPhil project, and has been completed by Jake Hillion. It was supervised by Anil Madhavapeddy.
Void processes intend to make it easier for all developers to produce effectively privilege separated applications. The project has two primary goals: show the merits of starting from zero privilege, and provide the utilities to make this feasible for the average developer.
Building void processes involves first reliably removing all privilege from a process then systematically adding back in what is required, and no more. This project utilises Linux namespaces to revoke privilege from an application, showing how this can be done and why its easier in some domains than others. It then shows how to inject sufficient privilege for applications to perform useful work, developing new APIs that are friendly for privilege separation. These elements compose a shim called the "void orchestrator", a framework for restricting Linux processes. […158 words]
Unikernels: the rise of the library hypervisor in MirageOS / Oct 2016
DockerCon talk on unikernels and MirageOS
The functional innards of Docker for Mac and Windows / Jun 2016
I gave a talk at the Functional Works meetup, held in Jane Street London about how Docker for Mac and Windows use OCaml and unikernels under the hood.
Improving Docker with Unikernels (via Docker) / May 2016
I announce the release of three big components that form the basis for Docker for Desktop: a hypervisor framework called HyperKit, a networking framework for host translation called VPNKit, and a versioned data management store called DataKit.
Unikernel Systems is now part of Docker / Jan 2016
Announced that Unikernel Systems is now part of Docker
Unikernel Systems acquired by Docker (via The New Stack) / Jan 2016
My startup Unikernel Systems was acquired by Docker, and I'll be joining and setting up a UK branch of Docker along with the rest of my team.
'Just like we did with containers, we are interested is democratizing that technology, making it available and useful to the millions of developers and IT pros out there, said Solomon Hykes, founder and chief technology officer for Docker. 'Unikernels allow you to basically get rid of the operating system, and instead compile into the application the small bits of the operating system it really needs.' -- The New Stack […111 words]
Unikernels: Functional Infrastructure with Mirage OS / May 2015
Talk at Esper on functional programming with unikernels
Jitsu: Just-In-Time Summoning of Unikernels / May 2015
NSDI 2015 talk on Jitsu
Using Docker to bulk-build OPAM packages on Linux / Nov 2013
Now that OCaml 4.01 has been released, there is a frenzy of commit activity in the development trunk of OCaml as the new features for 4.02 are all integrated. These include some enhancements to the type system such as injectivity, module aliases and extension points as a simpler alternative to syntax extensions.
The best way to ensure that these all play well together is to test against the ever-growing OPAM package database as early as possible. While we’re working on more elaborate continuous building solutions, it’s far easier if a developer can quickly run a bulk build on their own system. The difficulty with doing this is that you also need to install all the external dependencies (e.g. libraries and header files for bindings) needed by the thousands of packages in OPAM.
Enter a hip new lightweight container system called Docker. While containers aren’t quite as secure as type-1 hypervisors such as Xen, they are brilliant for spawning lots of lightweight tasks such as installing (and reverting) package installations. Docker is still under heavy development, but it didn’t take me long to follow the documentation and put together a configuration file for creating an OCaml+OPAM image to let OCaml developers do these bulk builds. […748 words]