Under the hood with Apple's new Containerization framework / Jun 2025

Apple made a notable announcement in WWDC 2025 that they've got a new containerisation framework in the new Tahoe beta. This took me right back to the early Docker for Mac days in 2016 when we announced the first mainstream use of the hypervisor framework, so I couldn't resist taking a quick peek under the hood.

There were two separate things announced: a Containerization framework and also a container CLI tool that aims to be an OCI compliant tool to manipulate and execute container images. The former is a general-purpose framework that could be used by Docker, but it wasn't clear to me where the new CLI tool fits in among the existing layers of runc, containerd and of course Docker itself. The only way to find out is to take the new release for a spin, since Apple open-sourced everything (well done!).

Getting up and running

To get the full experience, I chose to install the macOS Tahoe beta, as there have been improvements to the networking frameworks^[1] that are only present in the new beta. It's essential you only use the Xcode 26 beta as otherwise you'll get Swift link errors against vmnet. I had to force my installation to use the right toolchain via:

sudo xcode-select --switch /Applications/Xcode-beta.app/Contents/Developer

Once that was done, it was simple to clone and install the container repo with a make install. The first thing I noticed is that everything is written in Swift with no Go in sight. They still use Protobuf for communication among the daemons, as most of the wider Docker ecosystem does.

Starting our first Apple container

Let's start our daemon up and take the container CLI for a spin.

$ container system start
Verifying apiserver is running...
Installing base container filesystem...
No default kernel configured.
Install the recommended default kernel from [https://github.com/kata-containers/kata-containers/releases/download/3.17.0/kata-static-3.17.0-arm64.tar.xz]? [Y/n]: y
Installing kernel... 
⠙ [1/2] Downloading kernel 33% (93.4/277.1 MB, 14.2 MB/s) [5s]

The first thing we notice is it downloading a full Linux kernel from the Kata Containers project. This system spins up a VM per container in order to provide more isolation. Although I haven't tracked Kata closely since its launch in 2017, I did notice it being used to containerise confidential computing enclaves while Zahra Tarkhani and I were working on TEE programming models a few years ago.

The use of Kata tells us that container spins up a new kernel using the macOS Virtualization framework every time a new container is started. This is ok for production use (where extra isolation may be appropriate in a multitenant cloud environment) but very memory inefficient for development (where it's usual to spin up 4-5 VMs for a development environment with a database etc). In contrast, Docker for Mac uses a single Linux kernel and runs the containers within that instead.

It's not quite clear to me why Apple chose the extra overheads of a VM-per-container, but I suspect this might be something to do with running code securely inside the many hardware enclaves present in modern Apple hardware, a usecase that is on the rise with Apple Intelligence.

Peeking under the hood of the Swift code

Once the container daemon is running, we can spin up our first container using Alpine, which uses the familiar Docker-style run:

$ time container run alpine uname -a 
Linux 3c555c19-b235-4956-bed8-27bcede642a6 6.12.28 #1 SMP
Tue May 20 15:19:05 UTC 2025 aarch64 Linux
0.04s user 0.01s system 6% cpu 0.733 total

The container spinup time is noticable, but still less than a second and pretty acceptable for day to day use. This is possible thanks to a custom userspace they implement via a Swift init process that's run by the Linux kernel as the sole binary in the filesystem, and that provides an RPC interface to manage other services. The vminitd is built using the Swift static Linux SDK, which links musl libc under the hood (the same one used by Alpine Linux).

We can see the processes running by using pstree:

|- 29203 avsm /System/Library/Frameworks/Virtualization.framework/
   Versions/A/XPCServices/com.apple.Virtualization.VirtualMachine.xpc/
   Contents/MacOS/com.apple.Virtualization.VirtualMachine
|- 29202 avsm <..>/plugins/container-runtime-linux/
   bin/container-runtime-linux
   --root <..>/f82d3a52-c89b-4ff0-9e71-c7127cb5eee1
   --uuid f82d3a52-c89b-4ff0-9e71-c7127cb5eee1 --debug
|- 28896 avsm <..>/bin/container-network-vmnet
   start --id default
   --service-identifier <..>network.container-network-vmnet.default
|- 28899 avsm <..>/bin/container-core-images start
|- 29202 avsm <..>/bin/container-runtime-linux
   --root <..>/f82d3a52-c89b-4ff0-9e71-c7127cb5eee1
   --uuid f82d3a52-c89b-4ff0-9e71-c7127cb5eee1 --debug
|- 28896 avsm <..>/container-network-vmnet start --id default
   --service-identifier <..>network.container-network-vmnet.default

You can start to see the overheads of a VM-per-container now, as each container needs the host process infrastructure to not only run the computation, but also to feed it with networking and storage IO (which have to be translated from the host). Still, its a drop in the ocean for macOS these days, as I'm running 850 processes in the background on my Macbook Air from an otherwise fresh installation! This isn't the lean, fast MacOS X Cheetah I used on my G4 Powerbook anymore, sadly.

Finding the userspace ext4 in Swift

I then tried to run a more interesting container for my local dev environment: the ocaml/opam Docker images that we use in OCaml development. This showed up an interesting new twist in the Apple rewrite: they have an entire ext4 filesystem implementation written in Swift! This is used to extract the OCI images from the Docker registry and then construct a new filesystem.

$ container run ocaml/opam opam list
⠦ [2/6] Unpacking image for platform linux/arm64 (112,924 entries, 415.9 MB, Zero KB/s) [9m 22s] 
⠹ [2/6] Unpacking image for platform linux/arm64 (112,972 entries, 415.9 MB, Zero KB/s) [9m 23s] 
⠇ [2/6] Unpacking image for platform linux/arm64 (113,012 entries, 415.9 MB, Zero KB/s) [9m 23s] 
⠼ [2/6] Unpacking image for platform linux/arm64 (113,059 entries, 415.9 MB, Zero KB/s) [9m 23s] 
⠋ [2/6] Unpacking image for platform linux/arm64 (113,104 entries, 415.9 MB, Zero KB/s) [9m 24s] 
# Packages matching: installed                                                                      
# Name                # Installed # Synopsis
base-bigarray         base
base-domains          base
base-effects          base
base-threads          base
base-unix             base
ocaml                 5.3.0       The OCaml compiler (virtual package)
ocaml-base-compiler   5.3.0       pinned to version 5.3.0
ocaml-compiler        5.3.0       Official release of OCaml 5.3.0
ocaml-config          3           OCaml Switch Configuration
opam-depext           1.2.3       Install OS distribution packages

The only hitch here is how slow this process is. The OCaml images do have a lot of individual files within the layers (not unusual for a package manager), but I was surprised that this took 10 minutes on my modern M4 Macbook Air, versus a few seconds on Docker for Mac. I filed a bug upstream to investigate further since (as with any new implementation) there are many edge cases when handling filesystems in userspace, and the Apple code seems to have other limitations as well. I'm sure this will all shake out as the framework gets more users, but it's worth bearing in mind if you're thinking of using it in the near term in a product.

What's conspicuously missing?

I was super excited when this announcement first happened, since I thought it might be the beginning of a few features I've needed for years and years. But they're missing...

Running macOS containers: nope

In OCaml-land, we have gone to ridiculous lengths to be able to run macOS CI on our own infrastructure. Patrick Ferris first wrote a custom snapshotting builder using undocumented interfaces like userlevel sandboxing, subsequently taken over and maintained by Mark Elvers. This is a tremendous amount of work to maintain, but the alternative is to depend on very expensive hosted services to spin up individual macOS VMs which are slow and energy hungry.

What we really need are macOS containers! We have dozens of mechanisms to run Linux ones already, and only a few heavyweight alternatives to run macOS itself within macOS. However, the VM-per-container mechanism chosen by Apple might be the gateway to supporting macOS itself in the future. I will be first in line to test this if it happens!

Running iOS containers: nope

Waaaay back when we were first writing Docker for Mac, there were no mainstream users of the Apple Hypervisor framework at all (that's why we built and released Hyperkit. The main benefit we hoped to derive from using Apple-blessed frameworks is that they would make our app App-Store friendly for distribution via those channels.

But while there do exist entitlements to support virtualisation on macOS, there is no support for iOS or iPadOS to this day! All of the trouble to sign binaries and deal with entitlements and opaque Apple tooling only gets it onto the Mac App store, which is a little bit of a graveyard compared to the iOS ecosystem. This thus remains on my wishlist for Apple: the hardware on modern iPad adevices easily supports virtualisation, but Apple is choosing to cripple these devices from having a decent development experience by not unlocking the software capability by allowing the hypervisor, virtualisation and container frameworks to run on there.

Running Linux containers: yeah but no GPU

One reason to run Linux containers on macOS is to handle machine learning workloads. Actually getting this to be performant is tricky, since macOS has its own custom MLX-based approach to handling tensor computations. Meanwhile, the rest of the world mostly uses nVidia or AMD interfaces for those GPUs, which is reflected in container images that are distributed.

There is some chatter on the apple/container GitHub about getting GPU passthrough working, but I'm still unclear on how to get a more portable GPU ABI. The reason Linux containers work so well is that the Linux kernel provides a very stable ABI, but this breaks down with GPUs badly.

Does this threaten Docker's dominance?

I have mixed feelings about the Containerization framework release. On one hand, it's always fun to see more systems code in a new language like Swift, and this is an elegant and clean reimplementation of classic containerisation techniques in macOS. But the release fails to unlock any real new end-user capabilities, such as running a decent development environment on my iPad without using cloud services. Come on Apple, you can make that happen; you're getting ever closer every release!

I don't believe that Docker or Orbstack are too threatened by this release at this stage either, despite some reports that they're being Sherlocked. The Apple container CLI is quite low-level, and there's a ton of quality-of-life features in the full Docker for Mac app that'll keep me using it, and there seems to be no real blocker from Docker adopting the Containerization framework as one of its optional backends. I prefer having a single VM for my devcontainers to keep my laptop battery life going, so I think Docker's current approach is better for that usecase.

Apple has been a very good egg here by open sourcing all their code, so I believe this will overall help the Linux container ecosystem by adding choice to how we deploy software containers. Well done Michael Crosby, Madhu Venugopal and many of my other former colleagues who are all merrily hackily away on this for doing so! As an aside, I'm also just revising a couple of papers about the history of using OCaml in several Docker components, and a retrospective look back at the hypervisor architecture backing Docker for Desktop, which will appear in print in the next couple of months (I'll update this post when they appear). But for now, back to my day job of marking undergraduate exam scripts...

1. vmnet is a networking framework for VMs/containers that I had to reverse engineer back in 2014 to use with OCaml/MirageOS.

   ↩︎︎