Now that OCaml 4.01 has been released, there is a frenzy of commit activity in the development trunk of OCaml as the new features for 4.02 are all integrated. These include some enhancements to the type system such as injectivity, module aliases and extension points as a simpler alternative to syntax extensions.
The best way to ensure that these all play well together is to test against the ever-growing OPAM package database as early as possible. While we’re working on more elaborate continuous building solutions, it’s far easier if a developer can quickly run a bulk build on their own system. The difficulty with doing this is that you also need to install all the external dependencies (e.g. libraries and header files for bindings) needed by the thousands of packages in OPAM.
Enter a hip new lightweight container system called Docker. While containers aren’t quite as secure as type-1 hypervisors such as Xen, they are brilliant for spawning lots of lightweight tasks such as installing (and reverting) package installations. Docker is still under heavy development, but it didn’t take me long to follow the documentation and put together a configuration file for creating an OCaml+OPAM image to let OCaml developers do these bulk builds.
A basic Docker and OPAM setup
I started by spinning up a fresh Ubuntu Saucy VM on the Rackspace Cloud, which has a recent enough kernel version to work out-of-the-box with Docker. The installation instructions worked without any problems.
Next, I created a Dockerfile to represent the set of commands needed to prepare the base Ubuntu image with an OPAM and OCaml environment. You can find the complete repository online at https://github.com/avsm/docker-opam. Let’s walk through the
Dockerfile in chunks.
FROM ubuntu:latest MAINTAINER Anil Madhavapeddy <email@example.com> RUN apt-get -y install sudo pkg-config git build-essential m4 software-properties-common RUN git config --global user.email "firstname.lastname@example.org" RUN git config --global user.name "Docker CI" RUN apt-get -y install python-software-properties RUN echo "yes" | add-apt-repository ppa:avsm/ocaml41+opam11 RUN apt-get -y update -qq RUN apt-get -y install -qq ocaml ocaml-native-compilers camlp4-extra opam ADD opam-installext /usr/bin/opam-installext
This sets up a basic OCaml and OPAM environment using the same Ubuntu PPAs as the Travis instructions I posted a few months ago. The final command adds a helper script which uses the new
depexts feature in OPAM 1.1 to also install operating system packages that are required by some libraries. I’ll explain in more detail in a later post, but for now all you need to know is that
opam installext ctypes will not only install the
ctypes OCaml library, but also invoke
apt-get install libffi-dev to install the relevant development library first.
RUN adduser --disabled-password --gecos "" opam RUN passwd -l opam ADD opamsudo /etc/sudoers.d/opam USER opam ENV HOME /home/opam ENV OPAMVERBOSE 1 ENV OPAMYES 1
The next chunk of the Dockerfile configures the OPAM environment by installing a non-root user (several OPAM packages fail with an error if configured as root). We also set the
OPAMYES variables to ensure we get the full build logs and non-interactive use, respectively.
Running the bulk tests
We’re now set to build a Docker environment for the exact test that we want to run.
RUN opam init git://github.com/mirage/opam-repository#add-depexts-11 RUN opam install ocamlfind ENTRYPOINT ["usr/bin/opam-installext"]
This last addition to the
Dockerfile initializes our OPAM package set. This is using my
development branch which adds a massive diff to populate the OPAM
metadata with external dependency information for Ubuntu and Debian.
Building an image from this is a single command:
docker build -t avsm/opam github.com/avsm/docker-opam
ENTRYPOINT tells Docker that our wrapper script is the “root command” to
run for this container, so we can install a package in a container by doing
docker run avsm/opam ctypes
The complete output is logged to stdout and stderr, so we can capture that as easily as a normal shell command. With all these pieces in place, my local bulk build shell script is trivial:
pkg=`opam list -s -a` RUN=5 mkdir -p /log/$RUN/raw /log/$RUN/err /log/$RUN/ok for p in $pkg; do docker run avsm/opam $p > /log/$RUN/raw/$p 2>&1 if [ $? != 0 ]; then ln -s /log/$RUN/raw/$p /log/$RUN/err/$p else ln -s /log/$RUN/raw/$p /log/$RUN/ok/$p fi done
This iterates through a local package set and serially builds everything. Future enhancements I’m working on: parallelising these on a multicore box, and having a linked container that hosts a local package repository so that we don’t require a lot of external bandwidth. Stay tuned!