Now that OCaml 4.01 has been released, there is a frenzy of commit activity in the development trunk of OCaml as the new features for 4.02 are all integrated. These include some enhancements to the type system such as injectivity, module aliases and extension points as a simpler alternative to syntax extensions.

The best way to ensure that these all play well together is to test against the ever-growing OPAM package database as early as possible. While we’re working on more elaborate continuous building solutions, it’s far easier if a developer can quickly run a bulk build on their own system. The difficulty with doing this is that you also need to install all the external dependencies (e.g. libraries and header files for bindings) needed by the thousands of packages in OPAM.

Enter a hip new lightweight container system called Docker. While containers aren’t quite as secure as type-1 hypervisors such as Xen, they are brilliant for spawning lots of lightweight tasks such as installing (and reverting) package installations. Docker is still under heavy development, but it didn’t take me long to follow the documentation and put together a configuration file for creating an OCaml+OPAM image to let OCaml developers do these bulk builds.

A basic Docker and OPAM setup

I started by spinning up a fresh Ubuntu Saucy VM on the Rackspace Cloud, which has a recent enough kernel version to work out-of-the-box with Docker. The installation instructions worked without any problems.

Next, I created a Dockerfile to represent the set of commands needed to prepare the base Ubuntu image with an OPAM and OCaml environment. You can find the complete repository online at https://github.com/avsm/docker-opam. Let’s walk through the Dockerfile in chunks.

FROM ubuntu:latest
MAINTAINER Anil Madhavapeddy <anil@recoil.org>
RUN apt-get -y install sudo pkg-config git build-essential m4 software-properties-common
RUN git config --global user.email "docker@example.com"
RUN git config --global user.name "Docker CI"
RUN apt-get -y install python-software-properties
RUN echo "yes" | add-apt-repository ppa:avsm/ocaml41+opam11
RUN apt-get -y update -qq
RUN apt-get -y install -qq ocaml ocaml-native-compilers camlp4-extra opam
ADD opam-installext /usr/bin/opam-installext

This sets up a basic OCaml and OPAM environment using the same Ubuntu PPAs as the Travis instructions I posted a few months ago. The final command adds a helper script which uses the new depexts feature in OPAM 1.1 to also install operating system packages that are required by some libraries. I’ll explain in more detail in a later post, but for now all you need to know is that opam installext ctypes will not only install the ctypes OCaml library, but also invoke apt-get install libffi-dev to install the relevant development library first.

RUN adduser --disabled-password --gecos "" opam
RUN passwd -l opam
ADD opamsudo /etc/sudoers.d/opam
USER opam
ENV HOME /home/opam
ENV OPAMVERBOSE 1
ENV OPAMYES 1

The next chunk of the Dockerfile configures the OPAM environment by installing a non-root user (several OPAM packages fail with an error if configured as root). We also set the OPAMVERBOSE and OPAMYES variables to ensure we get the full build logs and non-interactive use, respectively.

Running the bulk tests

We’re now set to build a Docker environment for the exact test that we want to run.

RUN opam init git://github.com/mirage/opam-repository#add-depexts-11
RUN opam install ocamlfind
ENTRYPOINT ["usr/bin/opam-installext"]

This last addition to the Dockerfile initializes our OPAM package set. This is using my development branch which adds a massive diff to populate the OPAM metadata with external dependency information for Ubuntu and Debian.

Building an image from this is a single command:

$ docker build -t avsm/opam github.com/avsm/docker-opam

The ENTRYPOINT tells Docker that our wrapper script is the “root command” to run for this container, so we can install a package in a container by doing this:

$ docker run avsm/opam ctypes

The complete output is logged to stdout and stderr, so we can capture that as easily as a normal shell command. With all these pieces in place, my local bulk build shell script is trivial:

pkg=`opam list -s -a`
RUN=5
mkdir -p /log/$RUN/raw /log/$RUN/err /log/$RUN/ok
for p in $pkg; do
  docker run avsm/opam $p > /log/$RUN/raw/$p 2>&1
  if [ $? != 0 ]; then
    ln -s /log/$RUN/raw/$p /log/$RUN/err/$p
  else
    ln -s /log/$RUN/raw/$p /log/$RUN/ok/$p
  fi
done  

This iterates through a local package set and serially builds everything. Future enhancements I’m working on: parallelising these on a multicore box, and having a linked container that hosts a local package repository so that we don’t require a lot of external bandwidth. Stay tuned!