I founded a research group called OCaml Labs at the University of Cambridge, with the goal of pushing OCaml and functional programming forward as a platform, making it a more effective tool for all users (including large-scale industrial deployments), while at the same time growing the appeal of the language, broadening its applicability and popularity.
In my PhD work on Functional Internet in around 2003-2007, I developed high performance and reliable protocol implementations in OCaml. Subsequently from 2010, I worked on Personal Containers
to build high assurance private data processing platforms. This research lead me to really appreciate functional programming as a powerful approach to building robust software, and I got involved in the Commercial Users of Functional Programming workshop, first as a speaker and then an organiser in “Commercial Users of Functional Programming 2011 Workshop Report”
.
It was around this time in 2011 that my work on Unikernels and MirageOS was starting to materialise into a real project, but the OCaml language that we wrote everything in didn't have a unified open source community. Instead, there were islands of developers all over the world: the core maintainers concentrated in Inria in France, and academics teaching it in various universities, and some industrial shops like Jane Street or my own experiences from “Using Functional Programming within an Industrial Product Group: Perspectives and Perceptions”
. I put my head together with Yaron Minsky
in Tokyo at IFCP 2011 to see if we could try something a little unique for the time – establishing a centre for excellence in functional programming that would focus on the open-source and community building aspects of functional programming as well as traditional academic research.
In 2012, we launched the centre from the Cambridge Computer Lab in Announcing OCaml Labs. Things moved very quickly indeed as the group quickly grew to around 6 full time postdocs and engineers, with lots of interns coming through our doors. Our general strategy at this point was to understand the basic problems we were going to tackle, and so started with a few concrete projects to bootstrap the ecosystem:
We also began organising community events, both online and offline:
ocaml
organisation there, such as via Easily OPAM switching to any OCaml feature requestThere was enough activity in the early days that I managed to capture it in annual blog posts:
After 2014 though, things had grown to the point where it was just too difficult for me to keep up with the flurry of movement. We then aggregated into a "middle age" research project around 2015 with the following projects that would take the next few years.
One of the main thrusts in OCaml Labs was to construct the tools to enable effective development workflows for OCaml usage at an industrial scale, while remaining maintainable with a small community that needed to migrate from existing workflows. This effort was dubbed the "OCaml Platform" and really picked up stream after our release of the opam package manager, since it began the process of unifying the OCaml community around a common package collection.
While much of the work was lead from OCaml Labs, it's also been highly collaborative with other organisations and individuals in the community. And of course, 100% of the work was released as open source software under a liberal license. I've been giving annual talks since 2013 or so about the steady progress we've been making towards building, testing, documentation and package management for OCaml.
As you can see, it's quite a journey to build community-driven development tools. A key to our approach was to "leave no OCaml project behind", and we spent considerable effort ensuring that every step of the tooling evolution had a migration path for older OCaml projects. As a result, it's often still possible to compile 20 year old OCaml code using the modern tooling.
The other big research project we drove from OCaml Labs was the effort to bring multicore parallelism to OCaml. While this might seem straightforward, we quickly realised that the challenge was in preserving existing sequential performance while also allowing new code to take advantage of multicore CPUs.
The first talk we gave was in 2014 on “Multicore OCaml”. Little did we know how much work it would take to get this production worthy!
After several years of hacking, we finally had several breakthroughs:
While working on parallelism in OCaml with Leo White and Stephen Dolan
, KC Sivaramakrishnan
joined our group after completing his PhD and started us down the path of using algebraic effects to express concurrency in OCaml code.
In around 2020, I started publishing multicore monthlies on the OCaml discussion forum. This was because we had begin the journey to upstream our feature into the mainline OCaml compiler. As of 2021, this is going well and we are planning to incorporate domains-parallelism and runtime fibres into OCaml 5.0 in early 2022. The amount of work that we put into multicore has been way more than I expected at the outset of the project, but the results are deeply satisfying. I'm finding that coding using effects in a mainstream PL like OCaml to be really fun, and anticipate this having a big boost for Unikernels in MirageOS that are struggling somewhat under the weight of over-functorisation for portability.
The OCaml Labs research project at the University of Cambridge finally came to
a happy end in 2021, after almost ten years. After the first decade of fundamental
research and early engineering, the maintainership and stewarding of the resulting code has only
picked up pace as the OCaml userbase grows. There are now three commercial
companies who have taken over the work from the University, all run by research
staff originally in the Computer Lab group (Gemma Gordon, KC Sivaramakrishnan
and Thomas Gazagnaire
).
There's really exciting work happening there -- finishing up the upstreaming of the multicore OCaml feature into mainline OCaml (due in 2022), making unikernels and MirageOS ever more practical and robust to deploy, and shipping end-to-end Windows support in the OCaml toolchain.