OCaml Labs

Background

In my PhD work on Functional Internet in around 2003-2007, I developed high performance and reliable protocol implementations in OCaml. Subsequently from 2010, I worked on Personal Containers to build high assurance private data processing platforms. This research lead me to really appreciate functional programming as a powerful approach to building robust software, and I got involved in the Commercial Users of Functional Programming workshop, first as a speaker and then an organiser in “Commercial Users of Functional Programming 2011 Workshop Report”.

It was around this time in 2011 that my work on Unikernels and MirageOS was starting to materialise into a real project, but the OCaml language that we wrote everything in didn't have a unified open source community. Instead, there were islands of developers all over the world: the core maintainers concentrated in Inria in France, and academics teaching it in various universities, and some industrial shops like Jane Street or my own experiences from “Using Functional Programming within an Industrial Product Group: Perspectives and Perceptions”. I put my head together with Yaron Minsky in Tokyo at IFCP 2011 to see if we could try something a little unique for the time – establishing a centre for excellence in functional programming that would focus on the open-source and community building aspects of functional programming as well as traditional academic research.

Early Days (2012-2014)

In 2012, we launched the centre from the Cambridge Computer Lab in Announcing OCaml Labs. Things moved very quickly indeed as the group quickly grew to around 6 full time postdocs and engineers, with lots of interns coming through our doors. Our general strategy at this point was to understand the basic problems we were going to tackle, and so started with a few concrete projects to bootstrap the ecosystem:

• publishing “Real World OCaml: functional programming for the masses (1st Ed)” with O'Reilly, which sold lots of copies in the early days and created plenty of buzz for OCaml. It was quite fun attending author signings around the world and having lines of people queuing up for a signature!
• I worked closely with Thomas Gazagnaire (then CTO at OCamlPro) who lead the development of the first version of the opam package manager. Both of us were also establishing the MirageOS project at the time, and so we ended up bootstrapping a big chunk of the opam-repository for use by it, and we also took a (in hindsight excellent) decision to use the nascent GitHub platform as the primary mechanism for managing packages instead of hosting a database. After a few releases in 2012 and then OPAM 1.1 beta available, with pretty colours, the package manager rapidly established itself as the defacto standard for the OCaml ecosystem. I've been the chief maintainer of the opam-repository ever since then (with many wonderful co-maintainers who do much of the heavy lifting, of course!). As of 2021, there are over 20000 packages in the repository.

We also began organising community events, both online and offline:

• Didier Remy and I organised the inaugral OCaml Users and Developer's workshop in 2012, which morphed in subsequent years into the OCaml Workshop. See Camel Spotting in Paris for an earlier user group meeting as well.
• Cambridge Compiler Hacking sessions ran from 2013 to 2017 and served as introductions to new developers with experienced mentors on hand.
• the conference highlight of the year were undoubtedly the CUFP workshops at ICFP as they combined a really active academic and industrial crowd. The writeups are in “Commercial Users of Functional Programming 2011 Workshop Report” and “Commercial Users of Functional Programming 2013 Scribe's Report” to give you a sense of what went on.
• we worked with Ashish Agarwal and Christophe Troestler to develop a brand new website to replace the original https://caml.inria.fr one, and this eventually became ocaml.org in around 2012. Almost a decade later, I announced the replacement of this one with a v3 version as well.
• helping to open up OCaml compiler development by improving the GitHub infrastructure and starting the ocaml organisation there, such as via Easily OPAM switching to any OCaml feature request. Eventually, compiler development moved over entirely to GitHub thanks to a big push from the core developor team.

There was enough activity in the early days that I managed to capture it in annual blog posts:

• Reviewing the first year of OCaml Labs in 2013
• Talks from OCaml Labs during ICFP 2014
• Reviewing the second year of OCaml Labs in 2014

After 2014 though, things had grown to the point where it was just too difficult for me to keep up with the flurry of movement. We then aggregated into a "middle age" research project around 2015 with the following projects that would take the next few years.

The OCaml Platform

One of the main thrusts in OCaml Labs was to construct the tools to enable effective development workflows for OCaml usage at an industrial scale, while remaining maintainable with a small community that needed to migrate from existing workflows. This effort was dubbed the "OCaml Platform" and really picked up stream after our release of the opam package manager, since it began the process of unifying the OCaml community around a common package collection.

While much of the work was lead from OCaml Labs, it's also been highly collaborative with other organisations and individuals in the community. And of course, 100% of the work was released as open source software under a liberal license. I've been giving annual talks since 2013 or so about the steady progress we've been making towards building, testing, documentation and package management for OCaml.

• “Real World OCaml: functional programming for the masses (1st Ed)” was the book published by O'Reilly that explained how to use OCaml with the Core library.
• My 2013 talk on “The OCaml Platform v0.1” first introduced the OCaml Platform just after opam was first released.
• My 2014 talk on “The OCaml Platform v1.0” continued the steady adoption of opam within the OCaml community, to start bringing a standard package database across the different users.
• My 2015 Platform talk then introduced continous integration for opam, as well the start of the central documentation efforts (which were finally completed in 2021 after some herculean efforts!).
• By my 2017 Platform talk in Oxford, we had most of the OCaml community using opam and released opam 2.0, started contributing to the new jbuilder build tool from Jane Street, and began the shift from camlp4 to ppx and the development of the new odoc tool.
• In my 2018 Platform talk in Missouri, we had helped evolve jbuilder into the Dune build system (now the build tool of choice in OCaml), and started to combine packaging and build into a cohesive platform. The key challenge so far had been to fill in gaps in functionality, and now we could begin to weave together the components we'd built.
• My 2019 Platform talk in Berlin focussed on how workflows using all these tools would work, such as for package managers or application developers or end users.
• My 2020 Platform talk saw the unveiling of the VSCode OCaml Platform plugin, which provided a seamless integration with the IDE to let all the workflows and tools from earlier years "just work" out of the box.
• In 2021, we embarked on a huge mission to rebuild the ocaml.org online presence with a central documentation site that built 20000 packages with cross-referenced HTML documentation.

As you can see, it's quite a journey to build community-driven development tools. A key to our approach was to "leave no OCaml project behind", and we spent considerable effort ensuring that every step of the tooling evolution had a migration path for older OCaml projects. As a result, it's often still possible to compile 20 year old OCaml code using the modern tooling.

Multicore OCaml

The other big research project we drove from OCaml Labs was the effort to bring multicore parallelism to OCaml. While this might seem straightforward, we quickly realised that the challenge was in preserving existing sequential performance while also allowing new code to take advantage of multicore CPUs.

The first talk we gave was in 2014 on “Multicore OCaml”. Little did we know how much work it would take to get this production worthy! After several years of hacking, we finally had several breakthroughs:

• Any multicore-capable language needs a well-defined memory model, and we realised that none of the existing ones (e.g. in C++ or Java) were particularly satisfactory. Our PLDI paper on “Bounding Data Races in Space and Time” defined a sensible and novel memory model for OCaml that was predictable for developers.
• Our garbage collector and runtime design won the best paper award at ICFP for its systematic approach to the design and evaluation of several minor heap collectors, in “Retrofitting parallelism onto OCaml”.

Algebraic Effects

While working on parallelism in OCaml with Leo White and Stephen Dolan, KC Sivaramakrishnan joined our group after completing his PhD and started us down the path of using algebraic effects to express concurrency in OCaml code.

• The “Effectively Tackling the Awkward Squad” and “Concurrent System Programming with Effect Handlers” papers were our first forays into using the effect system for realistic usecases such as Unix systems programming.
• We then spent a few years engineering a full production-quality version of runtime fibres in “Retrofitting effect handlers onto OCaml”, again with a focus on maintaining tooling compatibility (e.g. with debuggers) and also having a minimal impact on sequential performance for existing code.

In around 2020, I started publishing multicore monthlies on the OCaml discussion forum. This was because we had begin the journey to upstream our feature into the mainline OCaml compiler. As of 2021, this is going well and we are planning to incorporate domains-parallelism and runtime fibres into OCaml 5.0 in early 2022. The amount of work that we put into multicore has been way more than I expected at the outset of the project, but the results are deeply satisfying. I'm finding that coding using effects in a mainstream PL like OCaml to be really fun, and anticipate this having a big boost for Unikernels in MirageOS that are struggling somewhat under the weight of over-functorisation for portability.

OCaml Labs (2021-present)

The OCaml Labs research project at the University of Cambridge finally came to a happy end in 2021, after almost ten years. After the first decade of fundamental research and early engineering, the maintainership and stewarding of the resulting code has only picked up pace as the OCaml userbase grows. There are now three commercial companies who have taken over the work from the University, all run by research staff originally in the Computer Lab group (Gemma Gordon, KC Sivaramakrishnan and Thomas Gazagnaire).

• OCaml Labs Consultancy is based in Cambridge in the UK.
• Tarides is based in Paris, France.
• Segfault Systems is based in Chennai, India.

There's really exciting work happening there -- finishing up the upstreaming of the multicore OCaml feature into mainline OCaml (due in 2022), making unikernels and MirageOS ever more practical and robust to deploy, and shipping end-to-end Windows support in the OCaml toolchain.