Under the hood with Apple's new Containerization framework / Jun 2025
Apple made a notable announcement in WWDC 2025 that they've got a new containerisation framework in the new Tahoe beta. This took me right back to the early Docker for Mac days in 2016 when we announced the first mainstream use of the hypervisor framework, so I couldn't resist taking a quick peek under the hood.
There were two separate things announced: a Containerization framework and also a container CLI tool that aims to be an OCI compliant tool to manipulate and execute container images. The former is a general-purpose framework that could be used by Docker, but it wasn't clear to me where the new CLI tool fits in among the existing layers of runc, containerd and of course Docker itself. The only way to find out is to take the new release for a spin, since Apple open-sourced everything (well done!).
[…1934 words]
notesUsing Komodo to manage Docker compose on a small cluster / May 2025
With the sunsetting of Equinix Metal I've also been migrating the Recoil machines over to new hosts in Mythic Beasts. This time around, rather than manually setting up services, I've turned to a nice new tool called Komodo which helps with deploying Docker containers across multiple servers. Unlike many other container management solutions, Komodo is refreshingly simple. It has a mode where it can take existing Docker compose files on a given host, and run them, and provide a web-based monitor to keep an eye on a few machines.
[…629 words]Socially self-hosting source code with Tangled on Bluesky / Mar 2025
I've been an avid user of GitHub since its launch, and it really has revolutionised how communities come together to work on open source. In recent years though, I find myself utterly overwhelmed by its notifications and want to experiment with alternative workflows. This experimentation also has a more serious undertone due to the increasing need for data sovereignty and so I'm starting to move my source code to self-hosted solutions that are less reliant on centralised services.
This has also come up persistently over the years in the OCaml community, with questions over why participation in packaging requires a GitHub account ever since the early days of opam. I've never found a good answer... until now, with the launch of an exciting new service that's built over the same protocol that Bluesky uses. As I noted a few weeks ago, the ATProto can be used for more than just microblogging. It can also be an identity layer, across which other applications can be built which reuse the social fabric from Bluesky accounts.
"Tangled" is a new service launched (just yesterday!) by opilli and icyphox to manage Git repositories. I'm having a lot of fun trying it out, even in its early alpha stages! The coolest thing about Tangled is that you can self-host your own knots, which control where the source code repositories are actually stored.
[…1104 words]Towards a frugal userspace for Linux / Dec 2024
All the work we've been doing on biodiversity (such as LIFE) comes at a fairly large computation and storage cost due to the amount of data that we churn through. This gets worse when you consider the exploratory nature of science -- we sometimes just need to mess around with the large dataset to test hypotheses which are often shown to be wrong. So then, when the LOCO conference came around, we wrote up our thoughts on what a frugal Linux userspace might look like.
The key insight is that the Linux kernel already exposes a number of namespace mechanisms (that we use in Docker, for example), and so we explore a new OS architecture which defaults to deterministic, reusable computation with the careful recording of side-effects. This in turn allows Linux to guide complex computations towards previously acquired intermediate results, but still allowing for recomputation when required by the user. We're putting this together into a new shell known as "Shark", and this first abstract describes our early results.
papersVoid Processes: Minimising privilege by default / Jan 2021
This is an idea proposed as a Cambridge Computer Science Part III or MPhil project, and has been completed by Jake Hillion.
Void processes intend to make it easier for all developers to produce effectively privilege separated applications. The project has two primary goals: show the merits of starting from zero privilege, and provide the utilities to make this feasible for the average developer.
Building void processes involves first reliably removing all privilege from a process then systematically adding back in what is required, and no more. This project utilises Linux namespaces to revoke privilege from an application, showing how this can be done and why its easier in some domains than others. It then shows how to inject sufficient privilege for applications to perform useful work, developing new APIs that are friendly for privilege separation. These elements compose a shim called the "void orchestrator", a framework for restricting Linux processes.
[…158 words]
ideasUnikernels: the rise of the library hypervisor in MirageOS / Oct 2016
DockerCon talk on unikernels and MirageOS
talksThe functional innards of Docker for Mac and Windows / Jun 2016
I gave a talk at the Functional Works meetup, held in Jane Street London about how Docker for Mac and Windows use OCaml and unikernels under the hood.
Improving Docker with Unikernels (via Docker)/ May 2016
I announce the release of three big components that form the basis for Docker for Desktop: a hypervisor framework called HyperKit, a networking framework for host translation called VPNKit, and a versioned data management store called DataKit.
Unikernel Systems is now part of Docker / Jan 2016
Announced that Unikernel Systems is now part of Docker
Unikernel Systems acquired by Docker (via The New Stack)/ Jan 2016
My startup Unikernel Systems was acquired by Docker, and I'll be joining and setting up a UK branch of Docker along with the rest of my team.
[…111 words]'Just like we did with containers, we are interested is democratizing that technology, making it available and useful to the millions of developers and IT pros out there, said Solomon Hykes, founder and chief technology officer for Docker. 'Unikernels allow you to basically get rid of the operating system, and instead compile into the application the small bits of the operating system it really needs.' -- The New Stack
Unikernels: Functional Infrastructure with Mirage OS / May 2015
Talk at Esper on functional programming with unikernels
Jitsu: Just-In-Time Summoning of Unikernels / May 2015
NSDI 2015 talk on Jitsu
Using Docker to bulk-build OPAM packages on Linux / Nov 2013
Now that OCaml 4.01 has been released, there is a frenzy of commit activity in the development trunk of OCaml as the new features for 4.02 are all integrated. These include some enhancements to the type system such as injectivity, module aliases and extension points as a simpler alternative to syntax extensions.
The best way to ensure that these all play well together is to test against the ever-growing OPAM package database as early as possible. While we’re working on more elaborate continuous building solutions, it’s far easier if a developer can quickly run a bulk build on their own system. The difficulty with doing this is that you also need to install all the external dependencies (e.g. libraries and header files for bindings) needed by the thousands of packages in OPAM.
Enter a hip new lightweight container system called Docker. While containers aren’t quite as secure as type-1 hypervisors such as Xen, they are brilliant for spawning lots of lightweight tasks such as installing (and reverting) package installations. Docker is still under heavy development, but it didn’t take me long to follow the documentation and put together a configuration file for creating an OCaml+OPAM image to let OCaml developers do these bulk builds.
[…748 words]