Summary. I founded a research group called OCaml Labs at the University of Cambridge, with the goal of pushing OCaml and functional programming forward as a platform, making it a more effective tool for all users (including large-scale industrial deployments), while at the same time growing the appeal of the language, broadening its applicability and popularity. Over a decade, we retrofitted multicore parallelism into the mainline OCaml manager, wrote a popular book on the language, and helped start and grow an OCaml package and tooling ecosystem that is thriving today.

Background

In my PhD work on Functional Internet Services in around 2003-2007, I developed high performance and reliable protocol implementations in OCaml. Subsequently from 2010, I worked on Personal Containers to build high assurance private data processing platforms. This research lead me to really appreciate functional programming as a powerful approach to building robust software, and I got involved in the Commercial Users of Functional Programming workshop, first as a speaker and then an organiser in CUFP 2011 Workshop Report.

It was around this time in 2011 that my work on Unikernels and MirageOS was starting to materialise into a real project, but the OCaml language that we wrote everything in didn't have a unified open source community. Instead, there were islands of developers all over the world: the core maintainers concentrated in Inria in France, and academics teaching it in various universities, and some industrial shops like Jane Street or my own experiences from Using functional programming within an industrial product group: perspectives and perceptions. I put my head together with Yaron Minsky in Tokyo at IFCP 2011 to see if we could try something a little unique for the time – establishing a centre for excellence in functional programming that would focus on the open-source and community building aspects of functional programming as well as traditional academic research.

Early Days (2012-2014)

In 2012, we launched the centre from the Cambridge Computer Lab in Announcing OCaml Labs. Things moved very quickly indeed as the group quickly grew to around 6 full time postdocs and engineers, with lots of interns coming through our doors. Our general strategy at this point was to understand the basic problems we were going to tackle, and so started with a few concrete projects to bootstrap the ecosystem:

• publishing Real World OCaml: Functional programming for the masses with O'Reilly, which sold lots of copies in the early days and created plenty of buzz for OCaml. It was quite fun attending author signings around the world and having lines of people queuing up for a signature!
• I worked closely with Thomas Gazagnaire (then CTO at OCamlPro) who lead the development of the first version of the opam package manager. Both of us were also establishing the MirageOS project at the time, and so we ended up bootstrapping a big chunk of the opam-repository for use by it, and we also took a (in hindsight excellent) decision to use the nascent GitHub platform as the primary mechanism for managing packages instead of hosting a database. After a few releases in 2012 and then OPAM 1.1 beta available, with pretty colours, the package manager rapidly established itself as the defacto standard for the OCaml ecosystem. I've been the chief maintainer of the opam-repository ever since then (with many wonderful co-maintainers who do much of the heavy lifting, of course!). As of 2021, there are over 20000 packages in the repository. I've been less active since about 2023, but still the repository administrator.

We also began organising community events, both online and offline:

• Didier Remy and I organised the inaugral OCaml Users and Developer's workshop in 2012, which morphed in subsequent years into the OCaml Workshop. See Camel Spotting in Paris for an earlier user group meeting as well.
• Cambridge Compiler Hacking sessions ran from 2013 to 2017 and served as introductions to new developers with experienced mentors on hand.
• the conference highlight of the year were undoubtedly the CUFP workshops at ICFP as they combined a really active academic and industrial crowd. The writeups are in CUFP 2011 Workshop Report, Commercial users of functional programming workshop report and CUFP'13 scribe's report to give you a sense of what went on.
• we worked with Ashish Agarwal and Christophe Troestler to develop a brand new website to replace the original https://caml.inria.fr one, and this eventually became ocaml.org in around 2012. Almost a decade later, I announced the replacement of this one with a v3 version as well.
• helping to open up OCaml compiler development by improving the GitHub infrastructure and starting the ocaml organisation there, such as via OCaml/GitHub integration. Eventually, compiler development moved over entirely to GitHub thanks to a big push from the core developer team.

There was enough activity in the early days that I managed to capture it in annual blog posts:

• Reviewing the first year of OCaml Labs in 2013
• Talks from OCaml Labs during ICFP 2014
• Reviewing the second year of OCaml Labs in 2014

After 2014 though, things had grown to the point where it was just too difficult for me to keep up with the flurry of movement. We then aggregated into a "middle age" research project around 2015 with the following projects that would take the next few years.

The OCaml Platform

One of the main thrusts in OCaml Labs was to construct the tools to enable effective development workflows for OCaml usage at an industrial scale, while remaining maintainable with a small community that needed to migrate from existing workflows. This effort was dubbed the "OCaml Platform" and really picked up stream after our release of the opam package manager, since it began the process of unifying the OCaml community around a common package collection.

While much of the work was lead from OCaml Labs, it's also been highly collaborative with other organisations and individuals in the community. And of course, 100% of the work was released as open source software under a liberal license. I've been giving annual talks since 2013 or so about the steady progress we've been making towards building, testing, documentation and package management for OCaml.

• Real World OCaml: Functional programming for the masses was the book published by O'Reilly that explained how to use OCaml with the Core library.
• My 2013 talk on The OCaml Platform v0.1 first introduced the OCaml Platform just after opam was first released.
• My 2014 talk on The OCaml Platform v1.0 continued the steady adoption of opam within the OCaml community, to start bringing a standard package database across the different users.
• My 2015 Platform talk then introduced continous integration for opam, as well the start of the central documentation efforts (which were finally completed in 2021 after some herculean efforts!).
• By my 2017 Platform talk in Oxford, we had most of the OCaml community using opam and released opam 2.0, started contributing to the new jbuilder build tool from Jane Street, and began the shift from camlp4 to ppx and the development of the new odoc tool.
• In my 2018 Platform talk in Missouri, we had helped evolve jbuilder into the Dune build system (now the build tool of choice in OCaml), and started to combine packaging and build into a cohesive platform. The key challenge so far had been to fill in gaps in functionality, and now we could begin to weave together the components we'd built.
• My 2019 Platform talk in Berlin focussed on how workflows using all these tools would work, such as for package managers or application developers or end users.
• My 2020 Platform talk saw the unveiling of the VSCode OCaml Platform plugin, which provided a seamless integration with the IDE to let all the workflows and tools from earlier years "just work" out of the box.
• In 2021, we embarked on a huge mission to rebuild the ocaml.org online presence with a central documentation site that built 20000 packages with cross-referenced HTML documentation.

As you can see, it's quite a journey to build community-driven development tools. A key to our approach was to "leave no OCaml project behind", and we spent considerable effort ensuring that every step of the tooling evolution had a migration path for older OCaml projects. As a result, it's often still possible to compile 20 year old OCaml code using the modern tooling.

Multicore OCaml

The other big research project we drove from OCaml Labs was the effort to bring multicore parallelism to OCaml. While this might seem straightforward, we quickly realised that the challenge was in preserving existing sequential performance while also allowing new code to take advantage of multicore CPUs.

The first talk we gave was in 2014 on Multicore OCaml. Little did we know how much work it would take to get this production worthy! After several years of hacking, we finally had several breakthroughs:

• Any multicore-capable language needs a well-defined memory model, and we realised that none of the existing ones (e.g. in C++ or Java) were particularly satisfactory. Our PLDI paper on Bounding data races in space and time defined a sensible and novel memory model for OCaml that was predictable for developers.
• Our garbage collector and runtime design won the best paper award at ICFP for its systematic approach to the design and evaluation of several minor heap collectors, in Retrofitting parallelism onto OCaml.

Algebraic Effects

While working on parallelism in OCaml with Leo White and Stephen Dolan, KC Sivaramakrishnan joined our group after completing his PhD at Purdue, and started us down the path of using algebraic effects to express concurrency in OCaml code.

• The Effectively tackling the awkward squad and Concurrent System Programming with Effect Handlers papers were our first forays into using the effect system for realistic usecases such as Unix systems programming.
• We then spent a few years engineering a full production-quality version of runtime fibres in Retrofitting effect handlers onto OCaml, again with a focus on maintaining tooling compatibility (e.g. with debuggers) and also having a minimal impact on sequential performance for existing code.

In around 2020, I started publishing multicore monthlies on the OCaml discussion forum. This was because we had begin the journey to upstream our feature into the mainline OCaml compiler. At the end of 2020, KC Sivaramakrishnan opened up a pull request to the mainline OCaml repository (#10831) and it got merged in early 2022, adding domains-parallelism and runtime fibres into OCaml 5.0! The amount of work that we put into multicore has been way more than I expected at the outset of the project, but the results are deeply satisfying. I'm finding that coding using effects in a mainstream PL like OCaml to be really fun, and anticipate this having a big boost for Unikernels in MirageOS that are struggling somewhat under the weight of over-functorisation for portability. It was also really fun seeing how much online attention we got as we went through the upstreaming journey.

OCaml Labs to Tarides (2021-present)

The OCaml Labs research project at the University of Cambridge finally came to a happy end in 2021, after almost ten years. After the first decade of fundamental research and early engineering, the maintainership and stewarding of the resulting code has only picked up pace as the OCaml userbase grows. There are now three commercial companies who have taken over the work from the University, all run by research staff originally in the Computer Lab group (Gemma Gordon, KC Sivaramakrishnan and Thomas Gazagnaire).

• OCaml Labs Consultancy is based in Cambridge in the UK.
• Tarides is based in Paris, France.
• Segfault Systems is based in Chennai, India.

All of those groups merged into one unified Tarides in 2022 (OCLC and Segfault), making it easier to manage a growing community of maintainers. There's really exciting work happening there to continue the upstreaming of the multicore OCaml features into mainline OCaml, making unikernels and MirageOS ever more practical and robust to deploy, and shipping end-to-end Windows support in the OCaml toolchain. You can read about all this and more on the Tarides blog, which is regularly updated with news on their projects.

Relevant Ideas

• Parallel traversal effect handlers for OCaml
  by Sky Batchelor cosupervised with Patrick Ferris in 2024 (Ongoing, Part II).
• Implementing a higher-order choreographic language
  by Rokas Urbonas cosupervised with Dmirtij Szamozvancev in 2024 (Completed, Part II).
• Using effect handlers for efficient parallel scheduling
  by Bartosz Modelski in 2022 (Completed, MPhil).
• Probabilistic Programming in OCaml
  by Hari Chandrasekaran cosupervised with Tom Kelly and Liang Wang in 2018 (Completed, Part II).
• Concurrent revisions for OCaml
  by Dimitar Popov in 2013 (Completed, Part II).
• Macro- and Micro-benchmarking in OCaml
  by Sebastian Funk in 2012 (Completed, Part II).
• Analysis of the Raft Consensus Protocol
  by Heidi Howard in 2012 (Completed, Part II).
• Functional Reactive Web Applications
  by Henry Hughes in 2010 (Completed, Part II).

Relevant Papers

[»] Eio 1.0 – Effects-based IO for OCaml 5 Thomas Leonard, Patrick Ferris, Christiano Haesbaert, Lucas Pluvinage, Vesa Karvonen, Sudha Parimala, KC Sivaramakrishnan, Vincent Balat and Anil Madhavapeddy In proceedings of the 2023 OCaml Users and Developers Workshop, Sep 2023 PDF   URL   BibTeX

[»] State of the OCaml Platform 2023 Thibaut Mattio, Anil Madhavapeddy, Thomas Gazagnaire and David Allsopp In proceedings of the 2023 OCaml Users and Developers Workshop, Sep 2023 PDF   URL   BibTeX

[»] Real World OCaml: Functional Programming for the Masses Anil Madhavapeddy and Yaron Minsky Book published by Cambridge University Press, Oct 2022 URL   BibTeX   DOI

[»] Experiences with Effects Thomas Leonard, Craig Ferguson, Patrick Ferris, Sadiq Jaffer, Tom Kelly, KC Sivaramakrishnan and Anil Madhavapeddy In proceedings of the 2021 OCaml Users and Developers Workshop, Aug 2021 PDF   URL   BibTeX

[»] Retrofitting effect handlers onto OCaml KC Sivaramakrishnan, Stephen Dolan, Leo White, Tom Kelly, Sadiq Jaffer and Anil Madhavapeddy In proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, Jun 2021 PDF   URL   BibTeX   DOI

[»] Parallelising your OCaml Code with Multicore OCaml Sadiq Jaffer, Sudha Parimala, KC Sivaramakrishnan, Tom Kelly and Anil Madhavapeddy In proceedings of the 2020 OCaml Users and Developers Workshop, Aug 2020 PDF   URL   BibTeX

[»] Retrofitting parallelism onto OCaml KC Sivaramakrishnan, Stephen Dolan, Leo White, Sadiq Jaffer, Tom Kelly, Anmol Sahoo, Sudha Parimala, Atul Dhiman and Anil Madhavapeddy Journal paper in Proceedings of the ACM on Programming Languages (vol 4 issue ICFP), Aug 2020 PDF   URL   BibTeX   DOI

[»] MirageOS 4: the dawn of practical build systems for exotic targets Lucas Pluvinage, Romain Calascibetta, Rudi Grinberg and Anil Madhavapeddy In proceedings of the OCaml Workshop 2019, Aug 2019 PDF   URL   BibTeX

[»] A modular foreign function interface Jeremy Yallop, David Sheets and Anil Madhavapeddy Journal paper in Science of Computer Programming (vol 164), Oct 2018 PDF   URL   BibTeX   DOI

[»] Bounding data races in space and time Stephen Dolan, KC Sivaramakrishnan and Anil Madhavapeddy In proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, Jun 2018 PDF   URL   BibTeX   DOI

[»] Concurrent System Programming with Effect Handlers Stephen Dolan, Spiros Eliopoulos, Daniel Hillerstrom, Anil Madhavapeddy, KC Sivaramakrishnan and Leo White In trends in Functional Programming, Apr 2018 PDF   URL   BibTeX   DOI

[»] Effectively tackling the awkward squad Stephen Dolan, Spiros Eliopoulos, Daniel Hillerstrom, Anil Madhavapeddy, KC Sivaramakrishnan and Leo White In the ACM ML Workshop, Sep 2017 PDF   BibTeX

[»] DaLi: Database as a Library Gowtham Kaki, KC Sivaramakrishnan, Thomas Gazagnaire, Anil Madhavapeddy and Suresh Jagannathan In 2nd Summit on Advances in Programming Languages (SNAPL), May 2017 PDF   BibTeX

[»] Declarative Foreign Function Binding Through Generic Programming Jeremy Yallop, David Sheets and Anil Madhavapeddy In the proceedings of Functional and Logic Programming (FLOPS), Feb 2016 PDF   URL   BibTeX   DOI

[»] Mergeable persistent data structures Benjamin Farinier, Thomas Gazagnaire and Anil Madhavapeddy In vingt-sixiemes Journees Francophones des Langages Applicatifs (JFLA 2015), Jan 2015 PDF   BibTeX

[»] Multicore OCaml Stephen Dolan, Leo White and Anil Madhavapeddy In the 4th ACM OCaml Users and Developers Workshop, Sep 2014 PDF   BibTeX

[»] Irminsule: a branch-consistent distributed library database Thomas Gazagnaire, Amir Chaudhry, Jon Crowcroft, Anil Madhavapeddy, Richard Mortier, Dave Scott, David Sheets and Gregory Tsipenyuk In the 4th ACM OCaml Users and Developers Workshop, Sep 2014 PDF   BibTeX

[»] The OCaml Platform v1.0 Anil Madhavapeddy, Amir Chaudhry, Jeremie Dimino, Thomas Gazagnaire, Louis Gesbert, Thomas Leonard, David Sheets, Mark Shinwell, Leo White and Jeremy Yallop In the 4th ACM OCaml Users and Developers Workshop, Sep 2014 PDF   BibTeX   Video

[»] The OCaml Platform v0.1 Anil Madhavapeddy, Amir Chaudhry, Thomas Gazagnaire, David Sheets, Phillipe Wang, Leo White and Jeremy Yallop In the 2nd ACM OCaml Users and Developers Workshop, Sep 2013 PDF   BibTeX

[»] Programming the Xen cloud using OCaml Dave Scott, Richard Sharp and Anil Madhavapeddy In the 1st ACM OCaml Users and Developers Workshop, Sep 2012 PDF   BibTeX   Video

[»] Dynamics for ML using Meta-Programming Thomas Gazagnaire and Anil Madhavapeddy Journal paper in Electronic Notes in Theoretical Computer Science (vol 264 issue 5), Jul 2011 PDF   URL   BibTeX   DOI

[»] Dynamics for ML using Meta-Programming Thomas Gazagnaire and Anil Madhavapeddy Journal paper in Electronic Notes in Theoretical Computer Science (vol 264 issue 5), Jul 2011 PDF   URL   BibTeX   DOI

[»] Using functional programming within an industrial product group: perspectives and perceptions Dave Scott, Richard Sharp, Thomas Gazagnaire and Anil Madhavapeddy In proceedings of the 15th ACM SIGPLAN international conference on Functional programming, Sep 2010 PDF   URL   BibTeX   DOI   Video

Relevant Talks

News Updates

Sep 2023. «» Presentation on the OCaml Platform in 2023 / «» An update on the OCaml EIO library at the OCaml Workshop 2023.

Jan 2023. «» Recap of the OCaml community in 2022 posted along with plans to join the 'fediverse'.

Oct 2022. «» The second edition of Real World OCaml available from Cambridge University Press!

Sep 2022. «» Paper on our incremental computation DSL ocurrent presented in OCaml Workshop 2022.

Apr 2022. «» Update posted to the OCaml community on progress towards OCaml 5.0.

Feb 2022. «» Update posted to the OCaml community on post-multicore-merge activities in OCaml trunk.

Dec 2021. «» Update posted to the OCaml community on major multicore code review with Inria.

Oct 2021. «» Update posted to the OCaml community on confirmation that the effect system will appear in OCaml 5.0.

Aug 2021. «» Paper on programming with effects in OCaml / «» Published a roadmap for OCaml.org's revamp.

Jun 2021. «» Paper on retrofitting effects in OCaml presented in PLDI 2024.

Aug 2020. «» Won best paper award at ICFP 2020 for our paper on retrofitting parallelism onto OCaml! / «» Presented the new OCaml DSL for continuous integration at the OCaml Workshop / «» Paper on how to parallelise OCaml code at the OCaml Workshop / «» Update on multicore OCaml being ready for testing / «» Talk on the state of the OCaml Platform in 2020.

Apr 2020. «» Update the OCaml community on multicore progress and linked preprint of ICFP submission on the runtime.

Jan 2020. «» Began updating the OCaml community regularly with multicore monthly reports.

Aug 2019. «» Paper on the MirageOS 4 build system at OCaml Workshop / «» Annual update on the OCaml Platform in 2019.

May 2019. «» Preprint on programming unikernels with ML modules.

Nov 2018. «» Rearranged the OCaml opam repository layout.

Oct 2018. «» Journal paper on building modular foreign function interfaces.

Jun 2018. «» Paper on the OCaml memory model and underlying theory at PLDI 2018.

Apr 2018. «» Paper on concurrent systems programming with effect handlers at TFP 2017.

Sep 2017. «» Paper on how to tackle awkward IO patterns with effect handlers / «» Talk on the state of the OCaml Platform.

May 2017. «» Position paper on building databases-as-a-library at SNAPL 2017.

May 2016. «» Announced HyperKit, VPNKit and DataKit for the Docker ecosystem.

Feb 2016. «» Paper on declarative approaches to foreign function bindings at FLOPS 2016.

Sep 2015. «» Update on the state of the OCaml Platform.

Apr 2015. «» Updated the OCaml community on annual OCaml Labs activities for 2014.

Feb 2015. «» Announced a call for sponsorships for ICFP 2015 in my role as industrial relations chair.

Jan 2015. «» Paper on mergeable data structures using Irmin (nee Irminsule) at JFLA 2015 / «» Published the scribe's report for CUFP 2013 in JFP.

Sep 2014. «» Paper on the OCaml Platform status / «» Paper at the OCaml Workshop on the Irmin database library / «» First paper on multicore OCaml's design at the OCaml Workshop / «» Talk on the OCaml Platform reaching v1.0.

Aug 2014. «» Note on talks from OCaml Labs at ICFP 2014.

Mar 2014. «» Call for sponsorships for ICFP 2014 in my role as industrial relations chair / «» Note on an experiment to improve the coupling of opam with GitHub / «» Note on the Codio support for the opam package manager.

Dec 2013. «» Updated the OCaml community on the activities of OCaml Labs in 2013.

Nov 2013. «» Published the scribe's report for CUFP 2012 / «» The 1st Edition of Real World OCaml by O'Reilly associates is released! / «» Note on integrating Docker and opam more effectively.

Sep 2013. «» Paper on the OCaml Platform at the OCaml Workshop 2013 / «» Note on the release of the OPAM 1.1 beta / «» Liveblog from the CUFP 2013 workshop at ICFP / «» Liveblog from the OCaml 2013 workshop.

Apr 2013. «» Note on how to grep the source code for all packages on opam.

Oct 2012. «» Announced the opening of OCaml Labs at Cambridge, along with Jane Street support.

Sep 2012. «» Paper on programming the Xen cloud using OCaml at the OCaml Workshop / «» Recording of the OCaml Labs announcement.

Feb 2012. «» Note on using ARM Dreamplugs with OCaml published.

Jan 2012. «» Published the scribe's report for CUFP 2011.

Jul 2011. «» Published dyntype at the Workshop on Generative Technologies / «» Paper on statically typed value persistence for OCaml in ENTCS 2011.

Apr 2011. «» Trip report about the new OCaml Users Group in Paris / «» Blogged about the growth of functional programming in the Systems Research Group.

Sep 2010. «» Paper on our experiences with writing the Xen control stack in OCaml at ICFP 2010.