anil.recoil.org

Announcing the OCaml Labs project

2012-10-19T00:00:00+01:00

I’m very excited to announce OCaml Labs, the latest project to hit the Cambridge Computer Lab. As anyone that hangs out near me probably realises, I very much enjoy functional programming. My weapon of choice tends to be OCaml, as it condenses decades of research into a pragmatic blend of functional, imperative and object-oriented programming styles. What’s perhaps less well known are the steady inroads that OCaml has been making into mission-critical areas of industry. At Jane Street, billions of dollars of transactions are routed through a huge ML code-base that is designed to catch bugs at compile-time. At Citrix, the Xen management toolstack that powers millions of hosts in the cloud is largely written in OCaml. Facebook does sophisticated static analysis using OCaml over their vast PHP codebase to close security holes.

The OCaml community is small but dedicated, but there is always more to do to improve the language and ecosystem. So, thanks to a generous platform grant from Jane Street, we are launching a program to help with the open-source development of OCaml from Cambridge.

The OCaml Labs are based in the Cambridge Computer Lab and led my myself, Alan Mycroft and Ian Leslie. We’re closely affiliated with other groups, and will be:

developing the OCaml Platform, which will bundle the official OCaml compiler from INRIA with a tested set of community libraries that refreshed every six months.
working with the core OCaml team at INRIA’s Gallium group on the compiler, and with commercial partners like OCamlPro on tool development. OCamlPro are making some very impressive progress already with the OPAM packge manager and TypeRex IDE helper.
supporting the online presence with more teaching material and content. Yaron, Jason and I are working hard on a new book that will be published next year, and the OCaml Web team (led by Ashish and Christophe) have made great progress on a brand new website that we will move to the ocaml.org domain soon.

Research efforts

Of course, it is difficult to hack on a language in a void, and we also use OCaml heavily in our own research. The other half of OCaml Lab’s goals are more disruptive (and riskier!):

The upcoming first beta release of Mirage, which is an operating system designed for cloud and embedded environments, and is written almost entirely from the ground up in OCaml. The outputs of Mirage include a large number of libraries which are usable separately, such as pure implementations of TCP/IP, DNS, SSH, DHCP and HTTP. The Xen hackers, led by David Scott, are out in force to integrate Mirage into their next-generation platform. Meanwhile, Raphael Proust is busy eliminating the garbage collector with his cut-down “LinearML” variant.
Working with our collaborators at the Horizon Institute on privacy-preserving technologies such as Signposts which let you build and maintain your own personal clouds that operate autonomously from the central cloud. You can read more about our privacy-by-design philosophy too.
Extending OCaml to run on secure hardware platforms that doesn’t compromise on performance, using the MIPS64-based capability processor that is being developed at at the Lab.
The Trilogy was a hugely successful EU-funded effort on future evolution of the Internet, and resulted in numerous RFCs on subjects such as multipath-TCP. We’re partipating in the follow-up (imaginatively dubbed “Trilogy2”), and look forward to working on more structured abstractions for programming large-scale networks.

Getting involved

So, how can you get involved? We are initially advertising three positions for full-time developers and researchers (junior and senior) to help us get started with the OCaml Platform and compiler development. These aren’t conventional pure research jobs, and a successful candidate should enjoy the open-source development cycle (you retain your own copyright for your own projects). The Computer Lab offers a pretty unique environment: a friendly, non-hierarchical group in a beautiful city, and some of the best faculty and students you could hope to hang out with.

And finally, there is a longer lead time on applying for PhDs, but this is a great time to get involved. When I started at the Lab in 2002, a little project called Xen was just kicking off, and many of us had a wild (and oft great) time riding that wave. Get in touch with myself, Alan, Ian or Jon soon if you are interested in applying! There’s some more information available on the OCaml Labs pages about options.

Why you should go to ICFP 2012

2012-07-31T00:00:00+01:00

Functional programming has been gaining popularity pretty rapidly recently. We’ve got serious projects from big Internet shops (Microsoft, Twitter and Facebook), to varied domains such as radio, knowledge bases, the energy grid, cryptography, NoSQL databases and storage, intricate PS3 games, and even making safer cars. Throughout all of this, one conference has been around since the very beginning: the ACM SIGPLAN International Conference on Functional Programming. The early days consisted of academics (some with fine beards and sandals) developing the tools and theories behind FP, and spawned the famous ICFP Programming Contest. The modern ICFP, however, also caters to the practical industry practitioner at all levels of knowledge, and this blog post is meant to introduce you to what to expect this year.

ICFP 2012 is a week-long conference on September 9-15th, held in beautiful Copenhagen. The theory-oriented workshops are held just before the main event on Sunday, on topics such as cross-model, generic and higher-order programming. The main conference lasts three days, with a combination of academic papers and experience reports that are formally published by the ACM (see my experience report or a Xenstored full paper for two Xen-related publications).

The main conference finishes mid-week, and the focus switches to more informal, interactive workshops that are relevant to the FP practitioner. The registration system lets you buy a day pass and attend any combination, so here’s a list that I’ve come up, along with “who” it is intended for. Needless to say, these are my personal views (although I’m co-chairing CUFP and the OCaml Workshop, and any mistakes there would be bad!).

Commercial Users of Functional Programming (CUFP)

CUFP has been going since 2004, acting as a voice for commercial users of functional programming languages and technology. CUFP is the biggest workshop at ICFP and spread over the last three days of the conference week.

Never done any Functional Programming before: For the first time, CUFP offers day-long tutorials for complete beginners to functional programming. You can spend a full day learning the basics of Scala or two days on Haskell (including concurrency and multicore). These tutorials are separate events with their own registrations so you can attend just them if you wish, and still meet the other attendees during the regular breaks and evening events.
Done some FP, and want to improve or learn a new language: There are half-day tutorials that are more focussed on specific topics: real-world OCaml (by me and Yaron!), Clojure and F#, and popular web frameworks in Erlang and Haskell.
There’s safety in crowds: The final day of CUFP (on Saturday 15th) is a full day of experience reports from industry speakers who have applied FP techniques to their day-to-day work. These talks are not intended to be deeply technical, but instead reflect on the social and engineering challenges around integrating a new (and often risky) technology into an existing business. The sessions this year are diverse, with talks from Citrix on client-side cloud computing, on building NoSQL databases, and a number of talks on biotech (bigdata, visualisation and medical software), developing real-time and offline analytics for national power grids, and even using FP to write software that runs directly on FPGAs (including directly on a new 10GBs switch from Arista networks). We really try to make this talk session a balanced view of the upsides and downsides of using FP, and encourage speakers to relate what went wrong as well as right in their experiences.
CTO/executive, aka “schmoozing”: A huge part of CUFP is the great networking between engineers and academics (with job-hunting grad students thrown into the mix). There are regular breaks which are full of animated conversations, and we are planning an industrial reception where vendors can show off their wares (either on Thursday or Friday) where you find commercial support, or recruit bright graduate students from top schools across the world. The breaks are also a chance to corner your favourite language hacker-professor and make them aware of your particular needs!

I’m co-chairing CUFP this year along with Michael Sperber, so feel free to direct any questions you have about it to us, and see below for registration information.

Language Workshops

Some of the bigger communities have day-long workshops which are a combination of short presentations and regular breaks for discussions. If you are using (or just considering the use of) these languages, then these are a good way to meet all the right people who could help you with your efforts.

OCaml and ML

I’m co-chairing the first OCaml Users and Developers workshop on Friday (Sep 14th). This event has traditionally been run in Paris in past years, but we decided it was getting big enought to run alongside ICFP. The talk schedule is a mix of experience reports and pratical new toolchain developments. There will be talks on all the latest development tools (memory profiling, concurrent programming), standard library efforts (Core, OPAM packaging), as well as experience reports (the Arakoon k/v store, Xen Cloud, and building 3D WebGL engines for NASA, among others). The hot topic at this workshop is the emerging consensus on building a more integrated OCaml Platform that acts as a stable base for larger applications, and the state-of-the-world talks from Xavier Leroy, Yaron Minsky and Fabrice le Fessant will no doubt kick off heated discussions that will go on late into dinner (last year, we all ended up in a very fine Tokyo restaurant drinking sake and arguing about GADTs and the global financial crisis).

OCaml is the most widely used variant of the ML family, and the ML workshop on the day before (Sep 13) takes a broader view on the theory and implementations of ML. Talks here include the Coq theorem prover, efficient implementations of OCaml running on the JVM, multicore-ML, and experimental extensions to ML-style programming (effect tracking, model checking, applicative functors, and GADTs).

Both of these workshops are convenient to attend over a two-day period (Thu and Fri), and will give you a huge amount of information about the latest developments in the land of ML.

Haskell

The Haskell community has been growing rather fast over the last few years, and so there are two separate days of talks.

The Haskell Symposium has formal published proceedings, and consists of a diverse array of research work that has direct implications on the future of Haskell. My personal highlights include the parallel array fusion and vectorisation for data processing on modern hardware, the intriguing “Wormhole” effect-tracking FRP, and Safe Haskell for practical information-flow tracking.

The second day has the Haskell Implementors Workshop, which is a more informal affair aimed at the day-to-day infrastructure and toolchain needs of Haskell. The schedule includes overview talks from the two Simons, and ongoing work on distributed programming and Javascript compilation.

So if you are after a broad overview of Haskell then attend the Symposium, and if you wish to join the Haskell Hogwarts and start compiler hacking, then the implementors workshop will have all the right people to help you out.

Erlang

The Erlang workshop at ICFP has been running for eleven years now, and tends to be a combination of big projects and work-in-progress reports. This year has the latest on porting Erlang to use LLVM for high-performance computing, as well as experience reports on scalability testing and the widely-used Riak distributed database. Despite my prelediction towards static typing, Erlang is a language that I keep intending to make bigger use of, so I plan to drop into this session for sure.

Last year we had to learn Japanese, the hardest functional language

We also made a lot of new local friends

Luckily, the weather made us feel at home though

Registration

This has been a whirlwind tour of the ICFP week (or rather, me writing this down while planning my own schedule). The early registration deadline is next week (Aug 9th), and the online registration has all the details of the full schedule and day prices. There is also local information about where to stay and visit in Copenhagen, if you want to take a late-summer break while making the trip.

Something worth noting is that all of this is run under the auspices of the ACM by volunteers, which accounts for the somewhat chaotic spread of information across all the websites. We’ve tried to demystify the CUFP process here, but please get in touch with any of the organisers if you are confused and want some help. We always welcome volunteer offers too, and the first round of drinks is on me for anyone who helps out during the week!

Dreaming of an ARM OCaml

2012-02-25T00:00:00+00:00

I’ve been meaning to play with Plug Computers for some time now, as I need a low-power embedded system around the house. I recently bought a Soekris Net6501 (a pretty powerful Intel CPU, that even has VT support), but had annoying issues getting it working reliably. I ordered an ARM-based Dreamplug as an alternative (and as a bonus, the Dreamplug is 6x cheaper than the Soekris!). Here are my notes on getting it to work.

Requirements:

Aside from the Dreamplug itself, make sure you order the optional JTAG module. This provides a serial console that is essential to getting any development done with it.
I also grabbed the extra 16GB Class 10 SLC SD Card, to act as my home directory.
You will also need another functional system running Debian (or a VM on your Mac; whatever is easiest). The JTAG drivers for the USB serial are easiest to get running on Linux.

The Dreamplug arrived with a working installation, but running the absolutely ancient Debian Lenny. A dist-upgrade through to Wheezy led to bricking it almost immediately, and so I did a fresh installation from scratch.

For a fresh installation, place a USB stick of suitable size (greater than 2GB is best) into your functional Debian installation. Then:

The Marvell bootloader boots from a VFAT partition, so you will need two partitions. The first should be a small fat16 (I picked 150MB) and the remainder an ext3 partition for Linux itself. There are good instructions available on the Tor/Dreamplug wiki which show you how to do this.
I grabbed the latest kernel (at this time, 3.2.7) from with-linux, and installed it with the following commands (assuming your USB stick is /dev/sdb).

	$ sudo mount /dev/sdb1 /mnt
	$ sudo cp uImage /mnt
	$ sudo umount /mnt

You now need to use debootstrap to install a fresh root image. Because it is ARM and your main PC is probably an x86, you will need to setup the QEMU CPU emulator. An extremely cool feature of QEMU is that it can do transparent emulation of foreign binaries, so you can chroot directly into the ARM filesystem and run commands as if they were x86. The qemu-deboostrap command will take care of this for you, if you perform the steps below (again, assuming your USB stick is /dev/sdb).

	$ sudo apt-get install qemu-user-static debootstrap
	$ sudo mount /dev/sdb2 /mnt
	$ sudo mkdir -p /mnt/usr/bin
	$ sudo cp /usr/bin/qemu-arm-static /mnt/usr/bin/
	$ sudo qemu-debootstrap --arch=armel wheezy http://ftp.uk.debian.org/debian/

Now grab the kernel modules from the same place as your uImage (for 3.2.7, from here). Then, chroot into your fresh installation and untar them.

	$ cd /mnt
	$ sudo tar -zxvf ~/sheeva-3.2.7-Modules.tar.gz
	$ sudo chroot /mnt
	$ depmod -a
	# edit /etc/network/interfaces
	# edit /etc/resolv.conf

The wireless setup involves some extremely crap firmware which relentlessly kernel panicked for me, so I just disabled it by adding the following to /etc/modprobe.d/dpwifiap.conf, as I only want wired access:

	blacklist libertas
	blacklist libertas_sdio

From there on, put the USB stick into the Dreamplug, and follow the rest of the boot instructions from the Tor wiki to interact with the Marvell BIOS and boot from the USB stick. I copied the contents of the USB stick onto the internal MicroSD, and it all boots standalone now.

OCaml on ARM

One of the reasons I wanted an ARM-based setup is to experiment with the OCaml native code generation. Benedikt Meurer has been doing some excellent work on improving code generation for embedded systems, including support for 16-bit Thumb code, exception backtraces, and dynamic linking and profiling.

Once Linux was up and running, compiling up the latest ocaml-trunk was straightforward.

	$ sudo apt-get install build-essential git
	$ git clone http://github.com/OCamlPro/ocp-ocaml svn-trunk
	$ cd ocp-ocaml
	$ ./configure && make world opt opt.opt install

This compiles the bytecode and native code compilers, and then compiles them again using the native code generator. This takes a while to do on the poor little ARM CPU. Once that finished, I compiled up a few simple modules and they worked great! Since the trunk of OCaml is a development branch, you may run into a few packaging issues (use the very latest OASIS to regenerate any setup.ml, and you will need a small patch until PR 5503 is applied).

Incidentally, if anyone is interested in working on a Mirage port to ARM as an internship in the Cambridge Computer Lab, do get in touch with me…

Commercial Users of Functional Programming 2011 Schedule

2011-08-09T00:00:00+01:00

The CUFP 2011 schedule is now available and the conference registration is open! The cufp.org website is having a few DNS propagation issues at the moment and is unreachable, so I have replicated the schedule below for your perusal. Hope to see lots of new faces in Tokyo!

September 22nd, 2011

Time	Speakers
09:00 AM - 12:30 PM	T1: Building Reliable Client-Server Applications in Erlang	Francesco Cesarini Erlang Solutions Ltd.
09:00 AM - 12:30 PM	T2: JaneStreet's OCaml Core Library	Yaron Minsky Jane Street
02:00 PM - 05:30 PM	T3: Building a Functional OS	Anil Madhavapeddy University of Cambridge David Scott Citrix Richard Mortier university of nottingham
02:00 PM - 05:30 PM	T4: Collaborative Scientific Software	Ashish Agarwal New York University

September 23rd, 2011

Time	Speakers
09:00 AM - 12:30 PM	T5: Parallel Programming in Haskell (Simon Peyton Jones, Simon Marlow, Manuel Chakravarty)	Simon Peyton-Jones Microsoft
09:00 AM - 12:30 PM	T6: Systems Programming in Scala (Steven Jenson, Marius Eriksen)	Marius Eriksen Twitter
02:00 PM - 05:30 PM	T7: The Snap Framework for web applications in Haskell (Gregory Collins)
02:00 PM - 05:30 PM	T8: F# for the Working Functional Programmer	Michael Sperber Active Group

September 24th, 2011

Time	Speakers
09:00 AM - 10:00 AM	Keynote: Pragmatic Haskell	Lennart Augustsson Standard Chartered Bank
10:00 AM - 10:30 AM	Tea Break
10:30 AM - 11:00 AM	Theorem-based derivation of an AES Implementation	John Launchbury Galois
11:00 AM - 11:30 AM	Discrete Event Simulation using Erlang	Mr. Olivier Boudeville EDF R&D, energy utility
11:30 AM - 12:00 PM	Model based testing of AUTOSAR automotive components	Dr. Thomas Arts QuviQ
12:00 PM - 01:30 PM	Lunch Break
01:30 PM - 02:00 PM	HTML5 web application development in OCaml	Mr. Keigo Imai IT Planning, Inc.
02:00 PM - 02:30 PM	Large-scale Internet Services in Scala at Twitter	Wilhelm Bierbaum Twitter Steve Jenson Twitter
02:30 PM - 03:00 PM	Applying Functional Programming to Build Platform-Independent Mobile Applications	Adam Granicz IntelliFactory
03:00 PM - 03:30 PM	Tea Break
03:30 PM - 04:00 PM	Fourteen Days of Haskell: A Real Time Programming Project in Real Time	Gregory Wright Alcatel-Lucent
04:00 PM - 04:30 PM	Disco: using Erlang to implement Mapreduce, Nokia	Jared Flatow Erlang, Nokia Prashanth Mundkur Erlang, Nokia Ville Tuulos Nokia Research
04:30 PM - 05:00 PM	Functional mzScheme DSLs in Game Development	Dan Liebgold Naughty Dog Inc.
05:00 PM - 05:30 PM	OCaml and Acunu Experience Report	Andrew Byde Acunu Tom Wilkie Acunu

DataCaml - a first look at distributed dataflow programming in OCaml

2011-06-18T00:00:00+01:00

Distributed programming frameworks like Hadoop and Dryad are popular for performing computation over large amounts of data. The reason is programmer convenience: they accept a query expressed in a simple form such as MapReduce, and automatically take care of distributing computation to multiple hosts, ensuring the data is available at all nodes that need it, and dealing with host failures and stragglers.

A major limitation of Hadoop and Dryad is that they are not well-suited to expressing iterative algorithms or dynamic programming problems. These are very commonly found patterns in many algorithms, such as k-means clustering, binomial options pricing or Smith Waterman for sequence alignment.

Over in the SRG in Cambridge, we developed a Turing-powerful distributed execution engine called CIEL that addresses this. The NSDI 2011 paper describes the system in detail, but here’s a shorter introduction.

The CIEL Execution Engine

CIEL consists of a master coordination server and workers installed on every host. The engine is job-oriented: a job consists of a graph of tasks which results in a deterministic output. CIEL tasks can run in any language and are started by the worker processes as needed. Data flows around the cluster in the form of references that are fed to tasks as dependencies. Tasks can publish their outputs either as concrete references if they can finish the work immediately or as a future reference. Additionally, tasks can dynamically spawn more tasks and delegate references to them, which makes the system Turing-powerful and suitable for iterative and dynamic programming problems where the task graph cannot be computed statically.

The first iteration of CIEL used a domain-specific language called Skywriting to coordinate how tasks should run across a cluster. Skywriting is an interpreted language that is “native” to CIEL, and when it needs to block it stores its entire execution state inside CIEL as a continuation. Derek Murray has written a blog post explaining this in more detail.

More recently, we have been working on eliminating the need for Skywriting entirely, by adding direct support for CIEL into languages such as Python, Java, Scala, and the main subject of this post – OCaml. It works via libraries that communicate with CIEL to spawn tasks, publish references, or suspend itself into the cluster to be woken up when a future reference is completed.

DataCaml API

Rather than go into too much detail about the innards of CIEL, this post describes the OCaml API and gives some examples of how to use it. The simplest interface to start with is:

	  type 'a ref
	  val deref : 'a ref -> 'a

The type 'a ref represents a CIEL reference. This data might not be immediately present on the current node, and so must be dereferenced using the deref function.

If the reference has been completed, then the OCaml value is unmarshalled and returned. If it is not present, then the program needs to wait until the computation involving the reference has completed elsewhere. The future reference might contain a large data structure and be on another host entirely, and so we should serialise the program state and spawn a task that is dependent on the future’s completion. This way, CIEL can resume execution on whatever node finished that computation, avoiding the need to move data across the network.

Luckily, we do not need to serialise the entire heap to suspend the program. DataCaml uses the delimcc delimited continuations library to walk the stack and save only the subset required to restart this particular task. Delimcc abstracts this in the form a “restartable exception” that supplies a closure which can be called later to resume the execution, as if the exception had never happened. Delimcc supports serialising this closure to an output channel, which you can read about in Oleg’s paper.

So how do we construct references? Lets fill in more of the interface:

	module Ciel = struct
	  type 'a ref
	  val deref : 'a ref -> 'a
	  val spawn : ('a -> 'b) -> 'a -> 'b ref
	  val run : (string list -> 'a) -> ('a -> string) -> unit
        end

The spawn function accepts a closure and an argument, and returns a future of the result as a reference. The run function begins the execution of a job, with the first parameter taking some string arguments and returning an 'a value. We also supply a pretty-printer second argument to convert the 'a into a string for returning as the result of the job (this can actually be any JSON value in CIEL, and just simplified here).

	let r1 = spawn (fun x -> x + 5) arg1 in
  	let r2 = spawn (fun x -> deref r1 + 5) arg1 in
	deref r2

We first spawn a function r1 which simply adds 5 to the job argument. A job in CIEL is lazily scheduled, so this marshals the function to CIEL, creates a future, and returns immediately. Next, the r2 function spawns a task which also adds 5, but to the dereferenced value of r1. Again, it is not scheduled yet as the return reference has not been dereferenced.

Finally, we attempt to dereference r2, which causes it be scheduled on a worker. While executing, it will try to dereference r1 that will schedule it, and all the tasks will run to completion.

Programming language boffins will recognise that this interface is very similar to AliceML’s concept of lazy futures. The main difference is that it is implemented as a pure OCaml library, and uses a general-purpose distributed engine that can also work with other languages.

Streaming References

The references described so far only have two states: they are either concrete or futures. However, there are times when a task can progressively accept input and make forward progress. For these situations, references can also be typed as opaque references that are accessed via in_channel and out_channel, as networks are:

	type opaque_ref

	val spawn_ref : (unit -> opaque_ref) -> opaque_ref
	val output : ?stream:bool -> ?pipe:bool -> (out_channel -> unit) -> opaque_ref
	val input : (in_channel -> 'a) -> opaque_ref -> 'a

This interface is a lower-level version of the previous one:

spawn_ref creates a lazy future as before, but the type of references here is completely opaque to the program.
Inside a spawned function, output is called with a closure that accepts an out_channel. The stream argument informs CIEL that a dependent task can consume the output before it is completed, and pipe forms an even more closely coupled shared-memory connection (requiring the tasks to be scheduled on the same host). Piping is more efficient, but will require more work to recover from a fault, and so using it is left to the programmer to decide.
The input function is used by the receiving task to parse the input as a standard in_channel.

The CIEL engine actually supports multiple concurrent input and output streams to a task, but I’ve just bound it as a single version for now while the bindings find their feet. Here’s an example of how streaming references can be used:

	let x_ref = spawn_ref (fun () ->
	  output ~stream:true (fun oc ->
	    for i = 0 to 5 do
	      Unix.sleep 1;
	      fprintf oc "%d\n%!" i;
	    done
	  )
	) in
	let y_ref = spawn_ref (fun () ->
	  input (fun ic ->
	    output ~stream:true (fun oc ->
	      for i = 0 to 5 do
	        let line = input_line ic in
	        fprintf oc "LINE=%s\n%!" line
	      done
	    )
	  ) x_ref
	) in

We first spawn an x_ref which pretends to do 5 seconds of work by sleeping and outputing a number. This would of course be heavy number crunching in a real program. The y_ref then inputs this stream, and outputs its own result by prepending a string to each line.

Try it out

If you are interested in a more real example, then read through the binomial options calculator that uses streaming references to parallelise a dynamic programming problem (this would be difficult to express in MapReduce). On my Mac, I can run this by:

check out CIEL from from Derek’s Git repository.
install all the Python libraries required (see the INSTALL file) and OCaml libraries (delimcc and Yojson).
add <repo>/src/python to your PYTHONPATH
in one terminal: ./scripts/run_master.sh
in another terminal: ./scripts/run_worker.sh -n 5 (this allocates 5 execution slots)
build the OCaml libraries: cd src/ocaml && make
start the binomial options job: ./scripts/sw-start-job -m http://localhost:8000 ./src/package/ocaml_binopt.pack
there will be a URL printed which shows the execution progress in real-time
you should see log activity on the worker(s), and a result reference with the answer (10.x)
let us know the happy news if it worked or sad news if something broke

Discussion

The DataCaml bindings outlined here provide an easy way to write distributed, fault-tolerant and cluster-scheduled jobs in OCaml. The current implementation of the engine is aimed at cluster computation, but Malte has been working on condensing CIEL onto multicore hardware. Thus, this could be one approach to ‘solving the OCaml multicore problem’ for problems that fit nicely into the dataflow paradigm.

The biggest limitation for using these bindings is that delimited continuation serialisation only works in bytecode. Native code delimcc supports shift/reduce in the same program, but serialising is problematic since native code continuations contain a C stack, which may have unwrapped integers. One way to work around this is by switching to a monadic approach to dereferencing, but I find delimcc programming more natural (also see this discussion).

Another important point is that tasks are lazy and purely functional (remind you of Haskell?). This is essential for reliable fault-tolerance and reproducibility, while allowing individual tasks to run fast, strict and mutable OCaml code. The tasks must remain referentially transparent and idempotent, as CIEL may choose to schedule them multiple times (in the case of faults or straggler correction). Derek has been working on integrating non-determinism into CIEL, so this restriction may be relaxed soon.

Finally, these ideas are not limited to OCaml at all, but also apply to Scala, Java, and Python. We have submitted a draft paper dubbed ’A Polyglot Approach to Cloud Programming’ with more details and the ubiquitous evaluation versus Hadoop. There is a really interesting line to explore between low-level MPI coding and high-level MapReduce, and we think CIEL is a useful spot in that design space.

Incidentally, I was recently hosted by Nokia Research in Palo Alto by my friend Prashanth Mundkur, where they work on the Python/Erlang/OCaml Disco MapReduce engine. I’m looking forward to seeing more critical comparisons and discussions of alternatives to Hadoop, from them and others.

Thanks are due to Derek, Chris and Malte for answering my incessant CIEL questions while writing this post! Remember that DataCaml is a work in progress and a research prototype, and feedback is most welcome.

Recoil gets IPv6 at last

2011-06-08T00:00:00+01:00

Since it is World IPv6 Day today, I decided to update the recoil.org machines to support it.

The changes required turned out to be very straightforward. Our amazing hosting provider Bytemark has been IPv6-ready for some time, and so I just had to reconfigure OpenBSD to add the IPv6 equivalent address as an alias to the network device. This is done by adding a single line to the /etc/hostname.re0 file:

inet6 alias 2001:41c8:0010:02ad::1 64

…and an additional default route in the /etc/mygate file:

fe80::1%re0

The %re0 at the end indicates which network interface to route across, as fe80::1 is a link-local address.

Then, I had to add an IPv6 AAAA record in the DNS. We use a hosting provider called EasyDNS who have support for this in their new user interface. One thing to be aware of is that you must upgrade your domain to their new system if you haven’t touched the DNS setup for a whole, which took me a while to find out (but they responded very quickly on Twitter!).

Once that was done, I can now check the host record in DNS, and connect to Google over IPv6!

$ host -t aaaa www.recoil.org
www.recoil.org is an alias for static.recoil.org.
static.recoil.org is an alias for dark.recoil.org.
dark.recoil.org has IPv6 address 2001:41c8:10:2ad::1

$ ping6 ipv6.google.com
PING6(56=40+8+8 bytes) 2001:41c8:10:2ad::1 --> 2a00:1450:8002::69
16 bytes from 2a00:1450:8002::69, icmp_seq=0 hlim=56 time=13.313 ms
16 bytes from 2a00:1450:8002::69, icmp_seq=1 hlim=56 time=13.165 ms
16 bytes from 2a00:1450:8002::69, icmp_seq=2 hlim=56 time=13.332 ms

Awesome! One final change is to modify system daemons to also listen on IPv6 in addition to IPv4. I’ve only changed our nginx web server so far, by following these IPv6 instructions. The OpenBSD ports version of nginx already has IPv6 support compiled in, so I just added a single line to the configuration:

listen [::]:80;

Since I don’t have access to any other IPv6 machines, I tested that the website is available by going to the IPv6 website reachability tester, and typing in www.recoil.org. And it all worked, and I got this cool little badge to display here…

Camel Spotting in Paris

2011-04-15T00:00:00+01:00

I’m at the 2011 OCaml Users Group in Paris, reporting on some splendid talks this year. It looked like around 60-70 people in the room, and I had the pleasure of meeting users all the way from Russia to New York as well as all the Europeans!

Js_of_ocaml

First up was Pierre Chambart talking about the js_of_ocaml compiler. It compiles OCaml bytecode directly to Javascript, with few external dependencies. Since the bytecode format changes very rarely, it is simpler to maintain than alternatives (such as Jake Donham’s ocamljs) that require patching the compiler tool-chain. Javascript objects are mapped to dynamic OCaml objects via a light-weight ## operator, so you can simply write code like:

	class type window = object
	  method alert : js_string t -> unit meth
	  method name : js_string t prop
	end
	let window : window t =
	  JS.Unsafe.variable "window"
	
	let () = 
	  window##alert ( window##name)
	  name <- Js.string "name"

Overloading is handled similarly to PyObjC, with each parameter combination being mapped into a uniquely named function. Raphael Proust then demonstrated a cool game he wrote using via bindings to the Raphael Javascript vector graphics library. Performance of js_of_ocaml is good compared to writing it by hand, and they have have quite a few benchmarks on their website.

Overall the project looks very usable: the main omissions are Bigarray, no dynlink, no Str (replaced by native regexps), no recursive modules or weak references. None of these missing features seem very critical for the sorts of applications that js_of_ocaml is intended for.

OCaml on a PIC (OCAPIC)

Next up Phillipe Wang presented something completely different: running OCaml on tiny 8-bit PIC microcontrollers! These PICs have 4-128Kb of flash (to store the code), and from 256 bytes to 4 kilobytes. Not a lot of room to waste there. He demonstrated an example with a game with 24 physical push buttons that beat humans at a conference (JFLA).

It works by translating OCaml bytecode through several stages: ocamlclean to eliminate dead code in the bytecode (which would be very useful for native code too!), a compression step that does run-length encoding, and then translation to PIC assembly. They have a replacement stop-and-copy GC (150 lines of assembly) and a full collection cycle runs in less than 1.5ms. Integers are 15-bits (with 1 bit reserved) and the block representation is the same as native OCaml. Very cool project!

Frama-C

We went onto static analysis and Julien Signoles presented Frama-C, a powerful static analysis tool for real-world C. It forks the CIL project from Berkeley and adds ocamlgraph and GUI support. He demonstrated a simple loop counter plugin to count them in C code, and the homepage has many interesting plugins maintained by the community.

I hadn’t realised that CIL was still maintained in the face of clang, so it’s nice to see it live on as part of Frama-C.

Ocsigen

The ever-cheerful Vincent Balat updated us about the Ocsigen web framework, including unveiling their exciting new logo! This was written using an amazing collaborative editor that lets users edit in real time.

Ocsigen is based around services of type service: parameters -> page. Services are first-class values, and can be registered dynamically and associated with sessions. The code for the collaborative editor was about 100 lines of code.

There is a syntax extension to distinguish between client and server side code, and both can be written in the same service (invoking js_of_ocaml to compile the client code to Javascript). They have bindings to Google Closure in order to provide UI support. There is a really nice “bus” service to pass messages between the server and the client, with seamless integration of Lwt to hide the details of communication to the browser.

Ocsigen is looking like a very mature project at this point, and I’m very keen to integrate it with Mirage to specialise the into micro-kernels. A task for the hacking day tomorrow morning I think!

Mirage

I talked about Mirage, hurrah! Good questions about why we need a block device (and not just use NFS), and I replied that everything is available as the library and the programmer can choose depending on their needs (the core goal of exokernels).

A highlight for me was lunch where I finally met Richard Jones, who is one of the other OCaml and cloud hackers out there. Wide ranging conversation about what the cool stuff going in KVM and Red Hat in general. Richard also gave a short talk about how they use OCaml to generate hundreds of thousands of lines of code in libguestfs. There are bindings for pretty much every major language, and it is all generated from an executable specification. He notes that “normal” programmers love the OCaml type safety without explicit annotations, and that it is a really practical language for the working programmer. The Xen Cloud Platform also has a similar generator for XenAPI bindings, so I definitely agree with him about this!

OCaml Future

Xavier “superstar” Leroy then gave an update of OCaml development. Major new features in 3.12.0 are first-class modules, polymorphic recursion, local module opens, and richer operations over module signatures. Version 3.12.1 is coming out soon, with bug fixes (in camlp4 and ocamlbuild mainly), and better performance on x86_64: turns out a new mov instruction change improves floating point performance on x86_64.

OCaml 3.13 has no release date, but several exciting features are in the pipeline. Firstly, more lightweight first-class modules by permitting some annotations to be inferred by the context, and it introduces patterns to match and bind first-class module values. Much more exciting is support for GADTs (Generalised Algebraic Data Types). This permits more type constraints to be enforced at compile time:

	type _ t =
	  | IntLit : int -> int t
	  | Pair : 'a t * 'b t -> ('a * 'b) t
	  | App : ('a -> 'b) t * 'a t -> 'b t
	  | Abs : ('a -> 'b) -> ('a -> 'b) t
	 
	let rec eval : type s . s t -> s = function
	  | IntLit x -> x (* s = int here *)
	  | Pair (x,y) -> (eval x, eval y) (* s = 'a * 'b here *)
	  | App (f,a) -> (eval f) (eval a)
	  | Abs f -> f

In this example of a typed interpreter, the eval function is annotated with a type s . s t -> s type that lets each branch of the pattern match have a constrained type for s depending on the use. This reminded me of Edwin Brady’s partial evaluation work using dependent types, but a much more restricted version suitable for OCaml.

There are some really interesting uses for GADTs:

Enforcing invariants in data structures, as with the typed interpreter example above.
Reflecting types into values means that libraries such as our own dyntype can be expressed in the core language without lots of camlp4 hacks. Finally, this should make typed I/O generators for XML, JSON and other network formats much simpler.

The challenges in the implementation are that principle type inference is now impossible (so some annotation is required), and pattern matching warnings are also trickier.

From the IDE perspective, the third bit of work is to have the OCaml compiler save the full abstract syntax tree annotation with source locations, scoping information, types (declared and inferred) and addition user-defined annotations. This generalises the -annot flag and can help projects like OCamlSpotter, OCamlWizard, OcaIDE, etc. It also helps code-generators driven by type-generators (such as our SQL ORM or ATDgen).

The OCaml consortium has new members; MLState and MyLife, and Esterel, OCamlPro and one unnamed new member are joining. The consortium goals are to sell permissive licensing (BSD) to members, and sound off new features with the serious users. Three companies are now doing commercial development (Gerd, OCamlCore, OCamlPro) which is growing the community nicely.

JoCaml

Luc Maranget (who looks like an archetypal mad professor!) gave a great rundown on JoCaml, a distributed programming extension to OCaml. This extends the compiler with join-definitions (a compiler patch), and a small bit of runtime support (using Thread), and significant extensions for concurrent and distributed programming in a type-safe way.

It extends the syntax with three new keywords: def, spawn and reply, and new usage for or and & (you should be using || and && anyway). Binary libraries remain compatible between matching versions of JoCaml and OCaml. An example of JoCaml code is:

	let create n =
	  def st(rem) & tick() = st(rem-1)
	  or st(0) & wait() = reply to wait in
	  spawn st(n) ; { tick=tick; wait=wait; }
	
	type t = {
	  tick: unit Join.chan;
	  wait: unit -> unit;
	}

After n messages to tick, the wait barrier function will be called.

	let c = create n
	let () =
	  for k = 0 to 9 do
	   spawn begin printf "%i" k; c.tick ()
	  done;
  	  c.wait ()

Here we asynchronously print the numbers of 0 to 9, and then the wait call acts as a barrier until it finishes. JoCaml is useful for distributed fork-join parallelism tasks such as raytracing, but with the type system support of OCaml. It is a bit like MapReduce, but without the data partitioning support of Hadoop (and is more light-weight). It would be quite interesting to combine some of the JoCaml extensions with the dynamic dataflow graphs in our own CIEL distributed execution engine.

Forgetful Memoisation in OCaml

Francois Bobot talks about the problem of memoizing values so that they can be re-used (e.g. in a cache). Consider a standard memoiser:

	let memo_f =
	  let cache = H.create () in
	  fun k ->
	    try H.find cache k
	    with Not_found ->
	      let v = f k in
	      H.add cache k v;
	      v
	
	let v1 = memo_f k1
	let v2 = memo_f k2 in (* k2 = k1 in O(1) *)

If a key is not reachable from anywhere other than the heap, we want to eliminate it from the cache also. The first solution is a normal hashtable, but this results in an obvious memory leak since a key held in the cache marks it as reachable. A better solution is using OCaml weak pointers that permit references to values without holding on to them (see Weaktbl by Zheng Li who is now an OCaml hacker at Citrix). The problem with Weaktbl is that if the value points to the key, forming a cycle which will never be reclaimed.

Francois solves this by using Ephemerons from Smalltalk. They use the rule that the value can be reclaimed if the key or the ephemeron itself can be reclaimed by the GC, and have a signature like:

	module Ephemeron : sig type ('a,'b) t
	  val create : 'a -> 'b -> ('a,'b) t
	  val check : ('a,'b) t -> bool
	  val get : ('a,'b) t -> 'b option
	  val get_key : ('a,'b) t -> 'a option
	end

The implementation in OCaml patches the runtime to use a new tag for ephemerons, and the performance graphs in his slides look good. This is an interesting topic for me since we need efficient memoisation in Mirage I/O (see the effects on DNS performance in the Eurosys paper which used Weaktbl). When asked if the OCaml patch will be upstreamed, Damien Doligez did not like the worst-case complexity of long chains of ephemerons in the GC, and there are several approaches under consideration to alleviate this without too many changes to the runtime, but Francois believes the current complexity is not too bad in practise.

Oasis and website

Sylvain came on stage later to give a demonstration of OASIS, an equivalent of Cabal for Haskell or CPAN for Perl. It works with a small _oasis file that describes the project, and then the OASIS tool auto-generates ocamlbuild files from it (this reminds me of Perl’s MakeMaker). Once the files are auto-generated, it is self-contained and there is no further dependency on OASIS itself.

OASIS works with either an existing build system in a project, or can be integrated more closely with ocamlbuild by advanced users. Lots of projects are already using OASIS (from Cryptokit to Lwt to the huge Jane Street Core). He is also working on a distribution mechanism on a central website, which should make for convenient OCaml packaging when it is finished and gets more adoption from the community.

Finally, Ashish Agarwal led a discussion on how OCaml can improve its web presence for beginners. Lots of good ideas here (some of which we implemented when reworking the CUFP website last year). Looking forward to seeing what happens next year in this space! I really enjoyed the day; the quality of talks was very high, and many engaging discussions from all involved!

How many OCaml hackers does it take to change a lightbulb?

Wearing bibs at French Teppinyaki

Team Mirage cheeses it up

Of course, not all of the OCaml community action is in France. The ever-social Jake Donham organised the First Ever San Francisco User Group that I attended when I was over there a few weeks ago. Ok, admittedly it was mainly French people there too, but it was excellent to meet up with Mika, Martin, Julien, Henri and of course Jake when over there.

We should definitely have more of these fun local meetups, and a number of other OCaml hackers I mentioned it to want to attend next time in the Bay Area, if only to cry into their drinks about the state of multi-core… just kidding, OCamlPro is hard at work fixing that after all :-)

Commercial Users of Functional Programming (CUFP) 2011

2011-04-13T00:00:00+01:00

Yaron Minsky and I are co-chairing the Commercial Users of Functional Programming workshop this year. It’s going to be held in Tokyo in September alongside ICFP.

The event is definitely the highlight of my conference calendar: a very diverse set of academics, industrial types and web hackers all mingle and argue till the early hours. There is also an expanded tutorial series (where David Scott and I plan to hold a session on constructing functional operating systems!), as well as BoF sessions for nailing down ideas with a smaller group of people.

The deadline for presentation proposals is June 15th (just a short abstract and bio required), so head over to the Call for Proposals and get writing!

If you are curious about past events, check out the videos from previous years. Two of my personal favourite talks are Eugene Letuchy talking about how they used Erlang at Facebook to build the popular chat functionality, and Marius Eriksen describing Scala for the geo-database at Twitter (embedded below).

Yurts for Digital Nomads

2010-04-29T00:00:00+01:00

The App Engine data collector for Personal Containers is coming on nicely, and is on track for an alpha preview release fairly soon. Working with AppEngine has been interesting; it’s got excellent availability and you can’t beat the price (free), but coding robust Python that doesn’t trip over the tight resource limits for individual requests, asynchronous tasks and queries is tricky. While it is good for small records such as my iPhone or Find My iPhone GPS traces traces, it doesn’t work so well with my gigabytes of photographs or decades of e-mail.

This confirmed our earlier intuition that there is no one perfect solution for personal data handling; instead, we need to embrace diversity and construct an infrastructure that can cope with change over the coming decades. Mobile programming has changed beyond recognition in just a few years, and cloud providers are specialising in different ways (e.g. PiCloud for simple compute, or EC2 for fancy services like elastic load balancing).

So to recognise this, we are building components that all interoperate with your personal data, keep it secure, and ensure it persists for more than a few years. Malte Schwarzkopf came up with the term “digital yurts”, and it’s stuck. We’ve written a draft paper about it, and would love to hear your comments and feedback on the approach.

There are some interesting recent trends that make doing this particularly important:

The New York Times wrote about the data-driven life increasingly influencing our decision making. Current sensor data such as GPS traces are just harbringers for the privacy disaster that would be information such as heart rates or your consumption habits getting into the public domain. (link via Derek Murray).
Facebook has announced a brand new API platform to get access to your information. The EFF has a fantastic timeline of Facebook’s Eroding Privacy over the last five years, to demonstrate how unsafe it is to trust your data to any third-party. We’ve started developing an information dump plugin for Facebook, but the API just changed mid-way and so it has to be started again (volunteers welcome!).
In the UK, the Digital Economy Act is an extremely controversial act that makes anonymity and privacy all the more important. We’re assembling an open-source dust cloud that integrates Tor into personal containers to automatically grant you anonymity as you communicate with your friends.

If you’re interested, join our group or contact me directly. At this stage, you need desire and the ability to hack code, but things are settling down over the next few months…

Pulling together a user interface

2010-04-15T00:00:00+01:00

We’ve been hacking away on fleshing out the App Engine node for personal containers. We’re building this node first because, crucially, deploying an App Engine VM is free to anyone with a Google account. The service itself is limited since you can only respond to HTTP or XMPP requests and do HTTP fetches, and so its primary use is as an always-on data collection service with a webmail-style UI written using extjs.

Personal containers gather data from a wide variety of sources, and normalise them into a format which understands people (address book entries, with a set of services such as e-mail, phone, IM and online IDs), places (GPS, WOEID), media (photos, movies) and messages (Tweets, emails, Facebook messages). I’ll post more about the data model behind personal containers in a follow-up as the format settles.

The App Engine node has a number of plugins to gather data and aggregate them into a single view (see screenshot). Plugins include:

iPhoto extracts location (via EXIF), people present (associated via faces), and of course, the actual photograph.
Adium logs all IMs into a threaded chat view.
iPhone uses the backup files on a Mac to extract SMS messages, phone call records (and it could also get photographs and browsing history, although it currently doesn’t). An AppEngine tracker can also use FindMyIPhone to poll your iPhone regularly to keep track of your location without publishing it to Google or Yahoo (and hopefully in iPhone 4.0, we can operate as a background service at last!).
Twitter runs directly on AppEngine (authenticated via OAuth) and synchronizes with a Twitter feed.
SyncServices hooks into the MacOS X sync framework and initially subscribes to Address Book updates. This seems to be the first open-source sync alternative to the expensive Mobile Me, as far as I can tell. I’m planning to expand this to also subscribe to the full set of sync information (e.g. calendars).

I’m switching tacks briefly; we received an Amazon Research Grant recently and I’m building a node that runs as a Linux server to act as a longer-term archival and search server. This is being written in OCaml and uses Tokyo Cabinet (with Jake Donham’s excellent bindings) and so should be speedy and a useful alternative implementation of the HTTP REST interface. The plan is to automatically synchronize meta-data across all the nodes of a personal container, but store large and historical data away from expensive cloud storage such as App Engine.

There are lots more plugins in development, such as Foursquare and Gowalla OAuth collectors, an Android mobile application to upload location and contacts information, and Google GData synchronization. If you’re interested in one of these or something else, please do get in touch or just fork the project and start hacking!

Opening a website

2010-03-29T00:00:00+01:00

We’ve been working away at building a new type of database to help individuals keep reigns on their ever-increasing personal digital information. The first prototypes run freely on Google App Engine to gather your data behind-the-scenes, and we are working on more advanced versions that run on embedded devices and the cloud.

If you’re interested in keeping track of your personal data, you can start off with the installation instructions to clone your own version. After that, read up on the design of the system (which is still changing as we research new ideas around it). When you find something you want to fix, or add a new plugin data source, just clone the code and send us back fixes!

C#, F# and Other Programming Language Fun at PDC 2008

2008-11-01T00:00:00+00:00

My favourite session of PDC 2008 was by Anders Hejlsberg as he described the Future of C#. Anders is an excellent speaker, and it was educational watching how he made fairly complex type systems come across to an audience more used to simpler programming languages. I say this after attending ICFP recently, where that wasn’t true of all the presentations!

The high-level message was exactly what I wanted to hear: in order to exploit multi-core processors more effectively, Microsoft’s languages are focussing on improvements in three areas:

Declarative constructs: making the programmer define overall goals and constraints enables a tool-chain to more effectively scale computation across multiple processors. An example of this in C# 3.0 is the Language Integrated Query (LINQ) domain specific language for defining queries.
Dynamic interoperability: Although Anders is a big fan of static type systems (robust, high-performance, enable intelligent tools), he pointed out that a lot of the systems that C# programmers have to deal with are fundamentally dynamic. Examples are web interfaces such as SOAP/REST APIs, COM objects for talking to other Microsoft applications, or scripting language integration (e.g. Python or Ruby).
Concurrent programming: dealing with multiple cores requires splitting up computation into parallel threads, and the resulting parallel mass of code is often hard to debug (e.g. difficulties of reproducing obscure concurrency issues) and statically reason about.

An interesting aside was his assertion about “co-evolution”. Microsoft have a number of languages which have recently been unified under the CLR, such as Visual Basic, C# and most recently F#. Rather than have one language race ahead of the others, they like to “borrow” features as appropriate into other languages. This is obviously made easier by having a common run-time foundation, and an example of co-evolution is the introduction of lambdas into C# 3.0 as obviously seen in F#.

Concurrency

Anders observes that all the attempts to automatically parallelize software has not yielded good results in the past, and so support needs to be built into the language to do “top-down” parallelism instead of the compiler inferring it from the bottom-up. To this end, they are introducing parallel extensions into the .NET framework. This makes use of functional features to parallelize code, including LINQ queries or CPU-intensive compute.

For an example of how this code looks, check out Jurgen Van Gael’s post on using F# with the Task Parallel Library. I wanted to learn more about this, and wandered over to the Hands On Labs where F# and Visual Studio 10 were all pre-installed. I then ran into an extremely cool feature of F#… asynchronous workflows.

F# introduces an extension to the usual let ML operator in the form of let!. As Don Syme explains on his blog ), this can be interpreted as “run the asynchronous computation on the right and wait for its result. If necessary suspend the rest of the workflow as a callback awaiting some system event”. So this construct lets you write straight-line code which can potentially block, without the hassle of spawning threads or encoding continuation passing style constructs in the code. Already a nice improvement over OCaml! (although to be fair I’ve not had a chance to check out some of the parallel OCaml extensions).

Dynamic Programming

Dynamic programming is the main focus of C# 4.0. In order to support dynamic objects as first-class citizens in a statically typed world, they introduce a dynamic static type. This forces all method resolution on that object to happen at run-time and disables static checks by the compiler (aside from looking from mixups between dynamic and static objects). The dynamic type now makes it easy to wrap support for Python, Ruby and Javascript, since the relevant dispatch functions can all be hidden away in the method resolver at run-time, leaving the programmer with a single syntax for invoking methods across these different languages.

The actual definition of method resolvers is pretty straight-forward; he demonstrated custom getters and setters (similar to Python for example) by using the IDynamicObject interface to define actions to take when properties are accessed. His example did the usual dictionary wrapper which mapped setting arbitrary properties onto an internal dictionary variable.

Another improvement in this space is the addition of optional arguments and labelled arguments. Both of these have well-defined semantics (optional arguments have to come after non-optional ones, and evaluation of arguments is left-to-right) and are purely syntactic improvements with no run-time cost. One of the best examples he showed of using these around COM interoperability. In current versions of C#, due to the lack of named arguments a common function such as “Save As” might require 12 or more stub arguments to be specified as ref missing. Now, those long, repetitive lines can be folded down to only the arguments which are required.

Sort of related to the earlier co-evolution was his demonstration of how similar Javascript and C# syntax is now with this new support. He took a Silverlight 2 code snippet written in Javascript and ported it over to C# 4.0 with some very mechanical changes. The languages are definitely converging fast!

If you’re interested in checking out more on this topic, look at IronPython, the Dynamic Language Runtime (DLR), and the TL10 session on dynamic languages on .NET. The DLR is currently maintained as part of IronPython, but is being moved out to sit on top of the CLR for more languages such as Visual Basic, Javascript or COM. The DLR covers common optimizations which are useful for scripting languages, such as call-site caching, dynamic dispatch and expression trees (e.g. for LINQ). The DLR has a bunch of binders which bridge between different backends such as .NET, Silverlight, the native Python or Ruby backends, or COM applications such as Microsoft office.

Meta-programming

Later on at the Future of Programming Languages panel session, Anders talked about meta-programming as being one of the future improvement he’s looking at. Currently, there is a lot of ad-hoc code generation in place when creating Windows applications, and unifying this into the language would give safety and maintainability improvements.

In order to do this, for C# 5.0 they are rewriting the compiler to be self-hosting in C#, since it has historically been a C++ application. This permits them to switch the compiler from being a traditional “black box” compiler to a hosted .NET service which can be called directly by .NET programs in order to do dynamic run-time compilation of code. Other portions of the compiler chain are also exposed to permit incremental program construction by third-party code.

He demonstrated this with a pretty nifty C# top-level, into which he directly typed Winforms code to construct a window with a few simple buttons using the C# compiler server. Not to be outdone by this, Miguel de Icaza promptly upstaged Anders at his (fantastic) Mono 2.2 session. He demonstrated the new C# shell which is present in Mono trunk builds and can essentially be used like an OCaml or Python top-level to mess around and manipulate C# code. He also talked about embedded Mono and SIMD support which pushes their compiler ahead of Microsoft’s in the 3D performance game.

Summary

I’m firmly convinced about the potential of F# now. I had the opportunity at the Open Spaces area to quiz Scott Guthrie about whether or not F# was a toy language. He replied using the same arguments as Anders that the higher-level language approach (declarative, functional) was very important strategically to Microsoft to let their developer platform continue to survive in a multi-core world.

This boils down to the individual languages not being that important any more (as seen by the sharing of features between C# and F#), and the underlying execution layer (the CLR/DLR) adding efficient support. Now any old language can adopt higher-level features without having to re-do all the optimization grunt work again and again. Much like Xen offers a new golden age for innovative new OS research by freeing programmers from writing a million hardware device drivers, it looks like .NET is ushering in a new age of programming language innovation!

Inspired by the PDC talks, I’ve got MonoDevelop and F# up and running on my Macbook Air, and am just playing with GTK# and CocoaSharp#. If this works as well as OCaml, then it might finally be time to abandon the old stalwart and move to a new language for my day-to-day stuff!

Working through TED videos on the train

2008-10-28T00:00:00+00:00

I’m over at the Microsoft PDC in Los Angeles this week, and commuting over from Anand’s place in Camarillo using the Metrolink. The 3 hour commute gives me a great excuse to catch up on my TED talks on my iPhone.

The entire set of talks is available online via iTunes, and these really stood out:

Jared Diamond on Why Societies Collapse : he talks about lots of civilizations which have reached tipping points and have collapsed, often catastrophically. Not a funny or entertaining one, but lots of solid, well-researched facts. Also see his book which George Dunlap recommended to me but is still in my reading list.
Mih?ly Cs?kszentmih?lyi on Creativity, fulfillment and flow : again, a really well-researched talk which categorizes human motivation into a 2D space of challenge vs skill, and the different emotions which are evoked by being in different situations (e.g. apathy requires no challenge and no skill, and anxiety results if you have low skill and a high challenge ahead of you). The “flow” he argues for is the ideal pinnacle of where we should aim to get to in our lives.
Finally, what I consider to be the greatest TED talk I’ve ever seen is by the late Paul MacCready on Nature vs Humans, and what we can do about it : he starts off slowly by showing some graphs about the impact of humanity on the population of the planet (hint: we’ve taken over), and finishes off with some truly awesome demonstrations of natural planes, such as Ornithopters or solar-powered tiny fliers that do not require gas to fly. Highly recommended if you only watch one TED video at all!

There are about 80 videos left on my viewing list.. this is going to take a while!

Peeking under the hood of High Availability

2008-09-17T00:00:00+01:00

Well, the big launch of XenServer 5 has gone smoothly, and with it have arrived a flood of questions about how exactly the new High Availability functionality works. I’ll use this post to explain the overall architecture of HA in XenServer 5, and also how some of the fault detection and failure planning works.

Fundamentally, HA is about making sure important VMs are always running on a resource pool. There are two aspects to this: reliably detecting host failure, and computing a failure plan to deal with swift recovery.

Detecting host failure reliably is difficult since you need to remotely distinguish between a host disappearing for a while versus exploding in a ball of flames. If we mistakenly decide that a master host has broken down and elect a new master in its place, there may be unpredictable results if the original host were to make a comeback! Similarly, if there is a network issue and a resource pool splits into two equal halves, we need to ensure that only one half accesses the shared storage and not both simultaneously.

Heartbeating for availability

We solve all these problems in XenServer by having two mechanisms: a storage heartbeat and a network heartbeat. When you enable HA in a pool, you must nominate an iSCSI or FC storage repository to be the heartbeat SR. XenServer automatically creates a couple of small virtual disks in this SR. The first disk is used by every physical host in the resource pool as a shared quorum disk. Each host allocates itself a unique block in the shared disk and regularly writes to the block to indicate that it is alive.

I asked Dave 'highly available' Scott, the principal engineer behind HA about the startup process:

“When HA starts up, all hosts exchange data over both network and storage channels, indicating which hosts they can see over both channels; i.e. which I/O paths are working and which are not. This liveness information is exchanged until a fixed point is reached and all of the hosts are satisfied that they are in agreement about what they can see. When this happens, the HA functionality is ‘armed’ and the pool is protected.”

This HA arming process can take a few minutes to settle for larger pools, but is only required when HA is first enabled.

Once HA is active, each host regularly writes storage updates to the heartbeat virtual disk, and network packets over the management interface. It is vital to ensure that network adapters are bonded for resilience, and that storage interfaces are using dynamic multipathing where supported. This will ensure that any single adapter or wiring failures do not result in any availability issues.

The worst-case scenario for HA is the situation where a host is thought to be off-line but is actually still writing to the shared storage, since this can result in corruption of persistent data. In order to prevent this situation without requiring active power strip control, we implemented hypervisor-level fencing. This is a Xen modification which will hard-power the host off at a very low-level if it doesn’t hear regularly from a watchdog process running in the control domain. Since it is implemented at a very low-level, this also covers the case where the control domain becomes unresponsive for any reason.

Hosts will self-fence (i.e. power off and restart) in the event of any heartbeat failure unless any of the following hold true:

The storage heartbeat is present for all hosts but the network has partitioned (so that there are now two groups of hosts). In this case, all of the hosts which are members of the largest network partition stay running, and the hosts in the smaller network partition self-fence. The assumption here is that the network outage has isolated the VMs, and they ought to be restarted on a host with working networking. If the network partitions are exactly the same size, then only one of them will self-fence according to a stable selection function.
If the storage heartbeat goes away but the network heartbeat remains, then the hosts check to see if they can see all other hosts over the network. If this condition holds true, then the hosts remain running on the assumption that the storage heartbeat server has gone away. This doesn’t compromise VM safety, but any network glitches will result in fencing since that would mean both heartbeats have disappeared.

Planning for failure

The heartbeat system gives us reliable notification of host failure, and so we move onto the second step of HA: capacity planning for failure.

A resource pool consists of several physical hosts (say, 16), each with potentially different amounts of host memory and a different number of running VMs. In order to ensure that no single host failure will result in the VMs on that host being unrestartable (e.g. due to insufficient memory on any other host), the XenServer pool dynamically computes a failure plan which calculates the actions that would be taken on any host failure.

But there’s one more complexity… a single host failure plan does not cover more advanced cases such as network partitions which take out entire groups of hosts. It would be very useful to be able to create a plan that could tolerate more than a single host failure, so that administrators could ignore the first host failure and be safe in the knowledge that (for example) three more hosts could fail before the pool runs out of spare capacity.

That’s exactly what we do in XenServer… the resource pool dynamically computes a failure plan which considers the “number of host failures to tolerate” (or nhtol). This represents the number of disposable servers in a pool for a given set of protected VMs.

The planning algorithms are pretty complex, since doing a brute force search of all possible failures across all hosts across all VMs is an exponential problem. We apply heuristics to ensure we can compute a plan in a reasonably small time:

for up to 3 host failures, we do a comprehensive search which tries almost all permutations. This covers corner cases such as having hosts or VMs with very different amounts of memory (e.g. 4GB vs 128GB). Rather than calculate memory slots or otherwise approximate results, we just deal with them individually and give very accurate plans.
for greater than 3 host failures, we make conservative decisions by approximating every VM to be as large as the largest, and considering each host to be the same as the most densely packed host. We do not approximate the host memory, and so having pools with uneven amounts of host memory will be fine. However, in approximate planning mode having a single very large VM will result in a low nhtol value. If this is a problem, then try to reduce the nhtol or try to have a more even spread of VM memory sizes.

Since planning algorithms are designed for unexpected host failures, we only consider absolutely essential resource reservations which would prevent the VM from starting on the alternative host (e.g. storage is visible, and enough memory is present). We do not perform CPU reservation on the basis that it can be optimised at a later stage via live relocation once the VM is back up and running.

Overcommit protection

We now have HA armed and a failover plan for our VMs. But what if you want to make changes to your configuration after HA is enabled? This is dealt with via overcommit protection.

The XenServer pool dynamically calculates a new failover plan in response to every XenAPI call which would affect it (e.g. starting a new VM). If a new plan cannot be calculated due to insufficient resources across the pool, the XenServer will return an overcommitment error message to the client which blocks the operation.

The “What if?” Machine

This overcommit protection would be quite irritating if you have to keep trying things and seeing if a plan exists or not, and so we built in a ”What If?” machine into XenServer to facilitate counter-factual reasoning.

When reconfiguring HA via XenCenter, you can supply a hypothetical series of VM priorities, and XenServer will return a number of host failures which would be tolerated under this scheme. This lets you try various combinations of VM protections depending on your business needs, and see if the number of host failures is appropriate to the level of paranoia you desire.

This can even be done via the CLI, using the snappily named ”xe pool-ha-compute-max-host-failures-to-tolerate” when HA is enabled.

The nice thing about XenServer HA is that it is done at the XenAPI level, and so any of the standard clients (such as the xe CLI or XenCenter) or any third-party clients which use the XenAPI will all interoperate just fine. The XenServer pool dynamically recalculates plans in response to the client requests, and so no special “oracle” is required outside of the pool to figure out HA plans.

Finally, HA makes master election completely invisible. Any host in a pool can be a master host, and the pool database is constantly replicated across all nodes and also backed up to shared storage on the heartbeat SR for additional safety. Any XenAPI client can connect to any host, and a redirect is issued to the current master host.

Protection Levels

Each VM in an HA pool can be either fully protected, best-effort or unprotected. VMs which are protected are all included in the failover planning, and if no plan exists for which they can all be reliably restarted then the pool is considered to be overcommitted. Hugh Warrington (who implemented the XenCenter HA support) explained what use protection levels are:

“Best-effort VMs are not considered when calculating a failover plan, but the pool will still try to start them as a one-off if a host that is running them fails. This restart is attempted after all protected VMs are restarted, and if the attempt to start them fails then it will not be retried. This is a useful setting for test/dev VMs which aren’t critical to keep running, but would be nice to do so in a pool which also has some important VMs which absolutely must run.”

There are some advanced features which are only available via the CLI. Each protected VM in an HA pool can be assigned a numeric ha-restart-priority. If a pool is well-resourced with a high nhtol, then these restart priorities are not relevant: the VMs are all guaranteed to be started.

If more hosts fail than have been planned for, then the priorities are used to determine the order in which VMs are restarted. This ensures that in over-committed pools, the most important VMs are restarted first. Although the pool will start priority 1 VMs first, they might not finish booting before the priority 2 VMs, and so this should not be used as the basis for service ordering.

Note that it's very important to ensure that a VM is agile when protecting it by HA. If the VM is not agile (e.g has a physical CD drive mapped in from a host), then it can only be assigned Best Effort restart since it is tied to one host.

XenCenter support for HA

The best practice for HA is not to make configuration changes while it is enabled. Instead, it is intended to be the "2am safeguard" which will restart hosts in the event of a problem when there isn't a human administrator nearby. If you are actively making configuration changes such as applying patches, then HA should be disabled for the duration of these changes.

XenCenter makes some common changes under HA much more user-friendly, which I asked Ewan Mellor (the principal GUI engineer) about:

Normally a protected VM cannot be shut down via the CLI or from within the guest (a shutdown from within the guest will automatically restart it). If you try to shutdown from XenCenter, it will give you the option of unprotecting the VM and then shutting it down first. Thus, accidental in-guest shutdowns wont result in downtime, but administrators can still stop a protected guest if they really want to.
If you want to reboot a host when HA is enabled, XenCenter automatically uses the hypothetical planning calculation to determine if this would invalidate the failover plan. If it doesn’t affect it, then the host is shut down normally. If the plan would be violated, but the nhtol is greater than 1, XenCenter will give the administrator the option of lowering the nhtol value by 1. This reduces the overall resilience of the pool, but always ensures that at least one host failure will be tolerated. When the host comes back up, the plan is automatically recalculated and the original nhtol value restored if appropriate.
If you try to apply a hotfix, then XenCenter will disable HA for the duration of the pool patching wizard. It is important to manually keep an eye on hotfix application to ensure that host failures do not disrupt the operation of the pool.

So, I hope this short article has given you a taster… just kidding! This post is almost as long as my PhD thesis, but then, HA is a complex topic. Please do feel free to get back to me with comments and feedback about how we can improve it in the future releases, or if you just love it the way it is. Many thanks to Dave Scott, Richard Sharp, Ewan Mellor and Hugh Warrington for their input to this article.

Shedding some light on XenApp on XenServer performance tuning

2008-08-04T00:00:00+01:00

You won’t be surprised to hear that we spend a lot of time improving XenApp performance when running on XenServer. Although there are some good benchmark comparisons available (such as the Tolly Group report), I still get a lot of customers asking about what the “secret sauce” is. I sat down with George Dunlap, the lead XenServer performance engineer to chat about the very first optimisation we did back in XenServer 4.0 last year.

Before we dive in, we first need to explain how a normal operating system handles memory. George explains:

“Modern desktop and server processors don’t access memory directly using its physical address. They use ’virtual memory’ to separate the addresses that processes use to read and write memory from the actual memory itself. This allows operating systems to hide from processes all the dirty details of how much memory there is, where in physical memory the process needs to write to, and so on.”

“However, the actual processor still needs to translate from a virtual address to the physical memory address in order to actually read and write any memory. This translation is done with something called page tables.”

“Page tables are used to implement virtual memory by mapping virtual addresses to physical addresses. The operating system constructs page tables using physical memory addresses, and then puts the physical address of the “top-level” page table into a hardware register called the ‘base pointer’. Then the processor will read these page tables to translate virtual addresses to physical addresses as needed, before reading and writing to physical memory.”

Most modern processor types have some sort of paging mechanism, although XenServer is specifically tuned for x86-64 CPUs. An excellent book on the general topic is Modern Operating Systems by Andrew Tanenbaum. When XenServer creates Windows VMs, it takes advantage of the virtualization extensions in modern CPUs, which requires special memory handling in Xen. George explains this further:

“When we create a virtual machine, we virtualize the memory as well; that means that the guest operating system’s idea of physical memory does not match up to real physical memory on the host. Traditionally, what the guest thinks of as physical memory is called “physical memory”, and what the hypervisor thinks of as physical memory is called “machine memory”. Since this terminology is a bit confusing, Xen tends to call what the guest thinks of as physical memory as “guest physical” memory, just to help make things more clear.”

“This means that any fully-virtualized operating system, like Windows, will create page tables using guest physical memory, and will point the base pointer at the guest physical address of the top-level page table. Unfortunately, the hardware still needs to map from virtual memory address to machine addresses, not guest physical addresses.”

“In order to allow this to happen, the hypervisor sets up shadow page tables. These page tables are generated by the hypervisor are copies of the guest page tables, but with the guest physical addresses converted into machine physical addresses. The guest cannot access them directly, and they don’t reside in the guest’s physical memory; they’re generated out of a pool of memory that the hypervisor allocates when a VM is created, called “shadow page table memory”.”

“What this means is that whenever the guest operating system wants to map some new memory, after it writes the data into the page table but before it can actually use it, the hypervisor needs to translate the change to the guest page table into changes to the shadow page table. So any workload that involves a lot of this will necessarily involve the hypervisor a lot, which causes overhead.”

So shadow page tables are our mechanism of giving a guest an interface which is identical to real hardware (so it doesn’t need to be modified), but still intercepting changes before they reach the real hardware. You can find more details from the XenSummit 2006 talk or from the 2005 NSDI paper. So how is this all relevant to XenApp performance? Back to George…

“The hypervisor allocates a certain amount of memory for each VM to use for shadow page tables; this is called shadow page table memory. As new page tables are created and old ones aren’t used anymore, the hypervisor cycles through this shadow page table memory. When it needs a new page and there isn’t enough, it will ‘unshadow’ the guest page tables that haven’t been used for the longest time to reclaim shadow memory, so that it can use more.”

“We don’t know ahead of time how much shadow memory a given workload will use, but we can estimate based on the amount of memory that the VM has. We allocate enough shadow memory for each page to be mapped once, more or less, then add an extra 50% to have some slack. For all the workloads we’ve tested, that’s been enough – except XenApp.”

“XenApp is the one workload we’ve found that requires more shadow page table memory than our standard default. Because XenApp generally starts hundreds of copies of the same process, the same memory ends up mapped in hundreds of different processes. What happens when all of those processes are active is that XenServer is continually unshadowing one process’ page tables in order to shadow another process’ pagetables; only to have to re-shadow the original ones a second or two later when it runs again! This is called thrashing, when there’s not enough of a limited resource.”

Once the bottleneck was discovered, the solution was simple. In XenServer 4.1, we created a special XenServer application template called "Citrix XenApp", which has an increased shadow multiplier that reserves more shadow memory for the guest when it starts. This is also a good example of how templates hide the complexities of performance tuning from the user, but still permitting custom modifications if they are required. For example, on your XenServer host with a VM called “XenApp”, you could view the shadow multiplier by using the CLI:

# xe vm-list name-label=XenApp params=HVM-shadow-multiplier
  HVM-shadow-multiplier ( RW)    : 4.000

The same value is also available from XenCenter in the Optimization pane, but of course do remember that the default value was chosen through extensive testing and doesn’t need to be changed. Most of the other templates in XenServer also have carefully tuned settings (e.g. the hardware platform flags) to ensure smooth running, or in the case of Linux templates, to support para-virtual installation. This is why it’s so important that you not use the "Other Install Media" template in preference of a more specialised one!

I mentioned at the beginning of this post that this was the first of many XenApp optimisations. We’ve just released the public beta of the latest XenServer (“Orlando”) which is even faster. The story of what those improvements are, and the tools which George and his team uses to analyze the inner workings of Xen, are a topic for a future post. For now, get downloading XenServer and start virtualizing your XenApp installations! Or if you’re feeling inspired, go over to xen.org, check out the source, and get coding…

Installing Ubuntu on XenServer

2008-07-02T00:00:00+01:00

I thought I’d kick off my Citrix blog with a question I get pretty often from Linux enthusiasts: how to install unsupported Linux distributions on XenServer 4.1.

The most common solution people find is to use the “Other Install Media” template, insert the distribution installation CD, and find that the mouse cursor doesn’t work when they boot into X11. The reason for this is that they are using the hardware-assisted emulation mode of installing Linux. In this mode (dubbed “HVM”), all input and output is emulated, and in particular the mouse interface uses the USB tablet interface. If the distribution doesn’t include a driver for USB tablets, then no mouse will appear.

Windows guests run at high-speed in HVM mode due to the installation of the XenServer tools which install high-speed drivers, but these are not necessary for Linux distributions since they can be run in para-virtualized mode (dubbed “PV”). This involves obtaining a Xen-enabled PV kernel from the distribution, and modifying the VM record in XenServer to boot into this kernel instead of HVM mode. The XenServer built-in templates for popular distributions such as RHEL, CentOS or SUSE Linux already automate all this and are in PV mode from the installer onwards.

In the remainder of this post, I’ll explain how to take a distribution without direct support (Ubuntu 8.04), get it installed in HVM mode on XenServer 4.1, and convert it to PV mode with a XenCenter graphical console.

Download the "Alternative Installation CD". The main installation CD uses graphical mode, which won't install as well in HVM mode due to the use of esoteric 16-bit mode instructions for the graphics operations. The 16-bit emulation mechanisms vary between processors (with better support on AMD chips, and a software instruction emulator required on Intel VT chips). However, the Ubuntu alternate CD uses a text-based installer which works fine.

Create a new VM on the XenServer 4.1 host using the "Windows Server 2003" template. This template is set up with a sensible set of hardware emulation flags and default disks, and so is a good base for the HVM installation of Ubuntu as well. Attach the Ubuntu ISO you just downloaded to the VM, and proceed to install Ubuntu as normal. You should install it onto the first disk, to make the subsequent steps in this guide easier.

When the installation is finished, reboot the VM (don't forget to detach the installation ISO first). It should boot up in HVM mode into the graphical login screen. The XenCenter display will show it as not being optimized, which is fine. At this stage, I prefer to work via a remote command-line using SSH. Open up a Terminal from Ubuntu, and run "sudo apt-get install openssh-server". Then find out the VM's IP address with "ifconfig eth0", and then connect to it remotely. Alternatively, you can continue to type in the commands directly into the terminal as well.

On the Ubuntu guest, you now need to install the latest Xen version of the Ubuntu kernel:
- Install the Linux kernel virtual package with "sudo apt-get install linux-image-xen". This is a virtual package which pulls in the latest Xen kernel and modules, in my case 2.6.24.19.21.
- You now need to workaround a bug in grub. Due to the switch in recent versions of Linux to work with the hypervisor-independent paravirt_ops interface, update-grub doesn't update the grub configuration with your newly installed Xen kernel. To fix this:
  - Open /boot/grub/menu.lst in your favourite editor.
  - Scroll to the bottom to the kernel list, and find the entry which looks like:
    title Ubuntu 8.04, kernel 2.6.24-16-generic root (hd0,0) kernel /boot/vmlinuz-2.6.24-16-generic root=UUID=<uuid> ro quiet splash initrd /boot/initrd.img-2.6.24-16-generic quiet
  - Add a new entry which is similar to this, but change all references to the 2.6.24-16-generic to the Xen kernel. In /boot I have vmlinuz-2.6.24-19-xen, so my new entry looks like:
    title Ubuntu 8.04, kernel 2.6.24-19-xen root (hd0,0) kernel /boot/vmlinuz-2.6.24-19-xen root=UUID=<uuid> ro quiet splash initrd /boot/initrd.img-2.6.24-19-xen quiet
  - Also edit the default entry in the menu.lst to match the number of the kernel you just added. I set mine to 3, since it is the fourth entry in the list and the indexing starts from 0.

When this is done, shut down the guest but do not reboot it just yet. You first need to edit the VM record for your Ubuntu VM to convert it to PV boot mode. From the control domain console of your XenServer:
- Determine the UUID of the Ubuntu VM by using the xe CLI:
  - xe vm-list name-label=Ubuntu params=uuid --minimal : this will print out the UUID of the VM named "Ubuntu". If you are logged into the control domain, pressing the <tab> key will perform auto-completion of UUIDs in subsequent XE commands, so you don't need to keep typing it in every time!
  - xe vm-param-set uuid=<uuid> HVM-boot-policy= : this will clear the HVM boot mode from the VM.
  - xe vm-param-set uuid=<uuid> PV-bootloader=pygrub : this will switch the VM to using to the pygrub bootloader which starts the guest in PV mode by examining its filesystem for kernel.
  - vm vm-param-set uuid=<uuid> PV-args="console=tty0 xencons=tty" : this configures the kernel boot arguments to display the login console on the correct TTY, so that it shows up in the XenCenter console.
- Next, you need to flag the root disk of the VM as bootable so that pygrub knows where to look for the PV kernel:
  - xe vm-disk-list uuid=<uuid> and look for the UUID of the VBD for the disk. VBD stands for "Virtual Block Device" and represents how to map the virtual disk into the virtual machine.
  - xe vbd-param-set uuid=<vbd uuid> bootable=true will set the root disk VBD to be bootable.

You should be all set now! If you boot up the Ubuntu VM, it should start up in text-mode with the high-speed PV kernel. If it doesn't work due to an incorrect grub configuration, you can use the xe-edit-bootloader script in the XenServer control domain to edit the grub.conf until it works.

The next step is to install the XenServer tools within the guest, so that metrics such as the network interface IP addresses are recorded and reported from XenCenter. To do this:
- Due to a portability issues with the default shell in Ubuntu (dash), you will need to replace it by: sudo apt-get -y install bash && sudo dpkg-reconfigure dash. We've actually fixed this issue in future releases of XenServer, but for XenServer 4.1 you will need to use bash.
- Attach the XenServer Tools ISO into the VM, and mount it into the guest with sudo mount /dev/xvdd /mnt
- Install the tools with sudo dpkg -i /mnt/Linux/xe-guest-utilities_4.1.0-257_i386.deb.
- The warnings about the VM being unoptimized should disappear, and additional information such as the IP address of the guest should appear in XenCenter.

In order to access the Ubuntu installation via the graphical console, you need to configure it to run VNC on the external network interface. XenCenter polls the guest to see if it is listening on the VNC port 5900, and offers the option to switch to the graphical console if it finds it. I followed the excellent instructions on this forum post. To summarise them:
- sudo apt-get install vnc4server xinetd : to install the required packages
- Edit /etc/gdm/gdm.conf and uncomment the RemoteGreeter=/usr/lib/gdm/gdmlogin line, set the key Enable=true in the [xdcmp] section.
- Install a new service file for xinetd into /etc/xinetd.d/Xvnc with the following contents:
```
service Xvnc
{
  type = UNLISTED
  disable = no
  socket_type = stream
  protocol = tcp
  wait = no
  user = nobody
  server = /usr/bin/Xvnc
  server_args = -inetd -query localhost -geometry 1024x768  -depth 16 -cc 3 -once -SecurityTypes=none -extension XFIXES
  port = 5900
}
```
- The major difference from the forum poster is to run it on port 5900, and not to restrict it to just localhost (since XenCenter also needs to connect to it).
- Finally, restart the xinetd service by running sudo /etc/init.d/xinetd restart.

Once you're done with this installation, you can shut down the VM and convert it to a template. Any exports or clones will continue to run in PV mode, since the XenServer XVA export format records all of the metadata required to re-create the VM records.

Enjoy the Ubuntu on XenServer experience! Remember to report any issues you have with the in-guest packages on the Ubuntu support forums, or just give them positive feedback.

PS: many thanks to Andrew Peace and Ian Campbell for assistance. May their Linux beards remain long and uncut.

Parsing EXIF/IPTC photo tags using pyexiv2 on Leopard

2008-04-17T00:00:00+01:00

I’ve been looking around at refreshing my current photo gallery to make it easier to update, since I’m around 1000 photos behind over the last year. Part of making it easier to use is to adopt an Aperture-based system of marking the metadata directly in the image itself, and figuring out how to render so many pictures more effectively in my gallery.

The best library I could find to do the job is Exiv2 and its associated Python bindings pyexiv2. They use the rather complex Boost.Python bindings, which are a total pain to compile on MacOS X Leopard.

In order to get it to work:

Use the latest Fink to install:
- python25
- boost1.34.python25
- libexiv2
- scons
Download the latest pyexiv2 distribution, extract it, and drop in my exiv2 Leopard Makefile into the extracted directory.
Type in:
- make
- sudo make install.
Test it out by running python2.5 and experimenting with importing pyexiv2 (see the pyexiv2 developer guide for more information.

I’ll tidy this up into a fink package sometime, but for now I want to press on with finishing my Django experiments.

Sushi, Seattle and Seahawks

2007-04-11T00:00:00+01:00

So I’m hanging at the XenSource Redmond office for a few days, and munching on delicious sushi at Flo, and it turns out that the dude behind us was none other than Matt Hasselbeck! It seems like only two years ago that I was cheering on the Pittsburgh Steelers and the Bus as they owned the Seahawks.

Unfortunately, I only realised my brief brush with celebrity on our way out of the restaurant, which also probably saved me from saying something stupid to him. As if! But anyway, the, err, moral of this post is that you must check out Flo Sushi if you’re in Seattle and eat the spider roll, it’s awesome.

Karnatik Jazz, Live Painting, and such things...

2007-04-08T00:00:00+01:00

Had a great evening out in San Francisco on Friday, when we visited the Red Poppy Art House on Folsom Street to check out a really unique Jazz experience.

The boys from VidyA recently took up an artist-in-residence position there, and performed their fusion of traditional Indian Karnatik music with modern Jazz pieces. They’ve been making some waves in the SF Jazz scene already it seems, and to top it all of, the owner of the art house did a live painting session during the performance. The audience passed out random bits of stuff, and he incorporated it into a piece done over the course of an evening. Talk about artistic overload!

Love the area as well; there’s a nice friendly coffee-house down the road (which is pretty much my only criteria, apart from not getting shot and such).

My school was never this cool.

The area was a mix of Latino and African, it seemed

And in the Red Poppy Art House, eclectic art abounds

Core stability exercising

2007-04-08T00:00:00+01:00

It’s been a wierd exercise year; I went from hard-core powerlifting (go 220lb bench!), to playing a lot of badminton and tennis, and then ending off the PhD era with breaking my knee (details not suitable for blog publication).

As anyone it’s happened to knows, a knee injury is really annoying to recover from as it feels really “wobbly” for ages and can’t quite be trusted. I’ve been doing physio on it myself for about half a year now, and can finally run around for 5km or so without too much discomfort, although yoga and Capoeria are perhaps a few more months away.

But then, the Forrest Gump-class runner and ex-officemate Alex Ho pointed out these amazing exercises by the captain of the CUH&H running team in Cambridge. They all emphasise smooth movement and core strength without the need for heavy weights, so can be done while travelling as well. So highly recommended, that even little Ajay (right) is trying them out!