anil.recoil.org

Yurts for Digital Nomads

2010-04-29T00:00:00+01:00

The App Engine data collector for Personal Containers is coming on nicely, and is on track for an alpha preview release fairly soon. Working with AppEngine has been interesting; it’s got excellent availability and you can’t beat the price (free), but coding robust Python that doesn’t trip over the tight resource limits for individual requests, asynchronous tasks and queries is tricky. While it is good for small records such as my iPhone or Find My iPhone GPS traces traces, it doesn’t work so well with my gigabytes of photographs or decades of e-mail.

This confirmed our earlier intuition that there is no one perfect solution for personal data handling; instead, we need to embrace diversity and construct an infrastructure that can cope with change over the coming decades. Mobile programming has changed beyond recognition in just a few years, and cloud providers are specialising in different ways (e.g. PiCloud for simple compute, or EC2 for fancy services like elastic load balancing).

So to recognise this, we are building components that all interoperate with your personal data, keep it secure, and ensure it persists for more than a few years. Malte Schwarzkopf came up with the term “digital yurts”, and it’s stuck. We’ve written a draft paper about it, and would love to hear your comments and feedback on the approach.

There are some interesting recent trends that make doing this particularly important:

The New York Times wrote about the data-driven life increasingly influencing our decision making. Current sensor data such as GPS traces are just harbringers for the privacy disaster that would be information such as heart rates or your consumption habits getting into the public domain. (link via Derek Murray).
Facebook has announced a brand new API platform to get access to your information. The EFF has a fantastic timeline of Facebook’s Eroding Privacy over the last five years, to demonstrate how unsafe it is to trust your data to any third-party. We’ve started developing an information dump plugin for Facebook, but the API just changed mid-way and so it has to be started again (volunteers welcome!).
In the UK, the Digital Economy Act is an extremely controversial act that makes anonymity and privacy all the more important. We’re assembling an open-source dust cloud that integrates Tor into personal containers to automatically grant you anonymity as you communicate with your friends.

If you’re interested, join our group or contact me directly. At this stage, you need desire and the ability to hack code, but things are settling down over the next few months…

Pulling together a user interface

2010-04-15T00:00:00+01:00

We’ve been hacking away on fleshing out the App Engine node for personal containers. We’re building this node first because, crucially, deploying an App Engine VM is free to anyone with a Google account. The service itself is limited since you can only respond to HTTP or XMPP requests and do HTTP fetches, and so its primary use is as an always-on data collection service with a webmail-style UI written using extjs.

Personal containers gather data from a wide variety of sources, and normalise them into a format which understands people (address book entries, with a set of services such as e-mail, phone, IM and online IDs), places (GPS, WOEID), media (photos, movies) and messages (Tweets, emails, Facebook messages). I’ll post more about the data model behind personal containers in a follow-up as the format settles.

The App Engine node has a number of plugins to gather data and aggregate them into a single view (see screenshot). Plugins include:

iPhoto extracts location (via EXIF), people present (associated via faces), and of course, the actual photograph.
Adium logs all IMs into a threaded chat view.
iPhone uses the backup files on a Mac to extract SMS messages, phone call records (and it could also get photographs and browsing history, although it currently doesn’t). An AppEngine tracker can also use FindMyIPhone to poll your iPhone regularly to keep track of your location without publishing it to Google or Yahoo (and hopefully in iPhone 4.0, we can operate as a background service at last!).
Twitter runs directly on AppEngine (authenticated via OAuth) and synchronizes with a Twitter feed.
SyncServices hooks into the MacOS X sync framework and initially subscribes to Address Book updates. This seems to be the first open-source sync alternative to the expensive Mobile Me, as far as I can tell. I’m planning to expand this to also subscribe to the full set of sync information (e.g. calendars).

I’m switching tacks briefly; we received an Amazon Research Grant recently and I’m building a node that runs as a Linux server to act as a longer-term archival and search server. This is being written in OCaml and uses Tokyo Cabinet (with Jake Donham’s excellent bindings) and so should be speedy and a useful alternative implementation of the HTTP REST interface. The plan is to automatically synchronize meta-data across all the nodes of a personal container, but store large and historical data away from expensive cloud storage such as App Engine.

There are lots more plugins in development, such as Foursquare and Gowalla OAuth collectors, an Android mobile application to upload location and contacts information, and Google GData synchronization. If you’re interested in one of these or something else, please do get in touch or just fork the project and start hacking!

Opening a website

2010-03-29T00:00:00+01:00

We’ve been working away at building a new type of database to help individuals keep reigns on their ever-increasing personal digital information. The first prototypes run freely on Google App Engine to gather your data behind-the-scenes, and we are working on more advanced versions that run on embedded devices and the cloud.

If you’re interested in keeping track of your personal data, you can start off with the installation instructions to clone your own version. After that, read up on the design of the system (which is still changing as we research new ideas around it). When you find something you want to fix, or add a new plugin data source, just clone the code and send us back fixes!

C#, F# and Other Programming Language Fun at PDC 2008

2008-11-01T00:00:00+00:00

My favourite session of PDC 2008 was by Anders Hejlsberg as he described the Future of C#. Anders is an excellent speaker, and it was educational watching how he made fairly complex type systems come across to an audience more used to simpler programming languages. I say this after attending ICFP recently, where that wasn’t true of all the presentations!

The high-level message was exactly what I wanted to hear: in order to exploit multi-core processors more effectively, Microsoft’s languages are focussing on improvements in three areas:

Declarative constructs: making the programmer define overall goals and constraints enables a tool-chain to more effectively scale computation across multiple processors. An example of this in C# 3.0 is the Language Integrated Query (LINQ) domain specific language for defining queries.
Dynamic interoperability: Although Anders is a big fan of static type systems (robust, high-performance, enable intelligent tools), he pointed out that a lot of the systems that C# programmers have to deal with are fundamentally dynamic. Examples are web interfaces such as SOAP/REST APIs, COM objects for talking to other Microsoft applications, or scripting language integration (e.g. Python or Ruby).
Concurrent programming: dealing with multiple cores requires splitting up computation into parallel threads, and the resulting parallel mass of code is often hard to debug (e.g. difficulties of reproducing obscure concurrency issues) and statically reason about.

An interesting aside was his assertion about “co-evolution”. Microsoft have a number of languages which have recently been unified under the CLR, such as Visual Basic, C# and most recently F#. Rather than have one language race ahead of the others, they like to “borrow” features as appropriate into other languages. This is obviously made easier by having a common run-time foundation, and an example of co-evolution is the introduction of lambdas into C# 3.0 as obviously seen in F#.

Concurrency

Anders observes that all the attempts to automatically parallelize software has not yielded good results in the past, and so support needs to be built into the language to do “top-down” parallelism instead of the compiler inferring it from the bottom-up. To this end, they are introducing parallel extensions into the .NET framework. This makes use of functional features to parallelize code, including LINQ queries or CPU-intensive compute.

For an example of how this code looks, check out Jurgen Van Gael’s post on using F# with the Task Parallel Library. I wanted to learn more about this, and wandered over to the Hands On Labs where F# and Visual Studio 10 were all pre-installed. I then ran into an extremely cool feature of F#… asynchronous workflows.

F# introduces an extension to the usual let ML operator in the form of let!. As Don Syme explains on his blog ), this can be interpreted as “run the asynchronous computation on the right and wait for its result. If necessary suspend the rest of the workflow as a callback awaiting some system event”. So this construct lets you write straight-line code which can potentially block, without the hassle of spawning threads or encoding continuation passing style constructs in the code. Already a nice improvement over OCaml! (although to be fair I’ve not had a chance to check out some of the parallel OCaml extensions).

Dynamic Programming

Dynamic programming is the main focus of C# 4.0. In order to support dynamic objects as first-class citizens in a statically typed world, they introduce a dynamic static type. This forces all method resolution on that object to happen at run-time and disables static checks by the compiler (aside from looking from mixups between dynamic and static objects). The dynamic type now makes it easy to wrap support for Python, Ruby and Javascript, since the relevant dispatch functions can all be hidden away in the method resolver at run-time, leaving the programmer with a single syntax for invoking methods across these different languages.

The actual definition of method resolvers is pretty straight-forward; he demonstrated custom getters and setters (similar to Python for example) by using the IDynamicObject interface to define actions to take when properties are accessed. His example did the usual dictionary wrapper which mapped setting arbitrary properties onto an internal dictionary variable.

Another improvement in this space is the addition of optional arguments and labelled arguments. Both of these have well-defined semantics (optional arguments have to come after non-optional ones, and evaluation of arguments is left-to-right) and are purely syntactic improvements with no run-time cost. One of the best examples he showed of using these around COM interoperability. In current versions of C#, due to the lack of named arguments a common function such as “Save As” might require 12 or more stub arguments to be specified as ref missing. Now, those long, repetitive lines can be folded down to only the arguments which are required.

Sort of related to the earlier co-evolution was his demonstration of how similar Javascript and C# syntax is now with this new support. He took a Silverlight 2 code snippet written in Javascript and ported it over to C# 4.0 with some very mechanical changes. The languages are definitely converging fast!

If you’re interested in checking out more on this topic, look at IronPython, the Dynamic Language Runtime (DLR), and the TL10 session on dynamic languages on .NET. The DLR is currently maintained as part of IronPython, but is being moved out to sit on top of the CLR for more languages such as Visual Basic, Javascript or COM. The DLR covers common optimizations which are useful for scripting languages, such as call-site caching, dynamic dispatch and expression trees (e.g. for LINQ). The DLR has a bunch of binders which bridge between different backends such as .NET, Silverlight, the native Python or Ruby backends, or COM applications such as Microsoft office.

Meta-programming

Later on at the Future of Programming Languages panel session, Anders talked about meta-programming as being one of the future improvement he’s looking at. Currently, there is a lot of ad-hoc code generation in place when creating Windows applications, and unifying this into the language would give safety and maintainability improvements.

In order to do this, for C# 5.0 they are rewriting the compiler to be self-hosting in C#, since it has historically been a C++ application. This permits them to switch the compiler from being a traditional “black box” compiler to a hosted .NET service which can be called directly by .NET programs in order to do dynamic run-time compilation of code. Other portions of the compiler chain are also exposed to permit incremental program construction by third-party code.

He demonstrated this with a pretty nifty C# top-level, into which he directly typed Winforms code to construct a window with a few simple buttons using the C# compiler server. Not to be outdone by this, Miguel de Icaza promptly upstaged Anders at his (fantastic) Mono 2.2 session. He demonstrated the new C# shell which is present in Mono trunk builds and can essentially be used like an OCaml or Python top-level to mess around and manipulate C# code. He also talked about embedded Mono and SIMD support which pushes their compiler ahead of Microsoft’s in the 3D performance game.

Summary

I’m firmly convinced about the potential of F# now. I had the opportunity at the Open Spaces area to quiz Scott Guthrie about whether or not F# was a toy language. He replied using the same arguments as Anders that the higher-level language approach (declarative, functional) was very important strategically to Microsoft to let their developer platform continue to survive in a multi-core world.

This boils down to the individual languages not being that important any more (as seen by the sharing of features between C# and F#), and the underlying execution layer (the CLR/DLR) adding efficient support. Now any old language can adopt higher-level features without having to re-do all the optimization grunt work again and again. Much like Xen offers a new golden age for innovative new OS research by freeing programmers from writing a million hardware device drivers, it looks like .NET is ushering in a new age of programming language innovation!

Inspired by the PDC talks, I’ve got MonoDevelop and F# up and running on my Macbook Air, and am just playing with GTK# and CocoaSharp#. If this works as well as OCaml, then it might finally be time to abandon the old stalwart and move to a new language for my day-to-day stuff!

Working through TED videos on the train

2008-10-28T00:00:00+00:00

I’m over at the Microsoft PDC in Los Angeles this week, and commuting over from Anand’s place in Camarillo using the Metrolink. The 3 hour commute gives me a great excuse to catch up on my TED talks on my iPhone.

The entire set of talks is available online via iTunes, and these really stood out:

Jared Diamond on Why Societies Collapse : he talks about lots of civilizations which have reached tipping points and have collapsed, often catastrophically. Not a funny or entertaining one, but lots of solid, well-researched facts. Also see his book which George Dunlap recommended to me but is still in my reading list.
Mih?ly Cs?kszentmih?lyi on Creativity, fulfillment and flow : again, a really well-researched talk which categorizes human motivation into a 2D space of challenge vs skill, and the different emotions which are evoked by being in different situations (e.g. apathy requires no challenge and no skill, and anxiety results if you have low skill and a high challenge ahead of you). The “flow” he argues for is the ideal pinnacle of where we should aim to get to in our lives.
Finally, what I consider to be the greatest TED talk I’ve ever seen is by the late Paul MacCready on Nature vs Humans, and what we can do about it : he starts off slowly by showing some graphs about the impact of humanity on the population of the planet (hint: we’ve taken over), and finishes off with some truly awesome demonstrations of natural planes, such as Ornithopters or solar-powered tiny fliers that do not require gas to fly. Highly recommended if you only watch one TED video at all!

There are about 80 videos left on my viewing list.. this is going to take a while!

Peeking under the hood of High Availability

2008-09-17T00:00:00+01:00

Well, the big launch of XenServer 5 has gone smoothly, and with it have arrived a flood of questions about how exactly the new High Availability functionality works. I’ll use this post to explain the overall architecture of HA in XenServer 5, and also how some of the fault detection and failure planning works.

Fundamentally, HA is about making sure important VMs are always running on a resource pool. There are two aspects to this: reliably detecting host failure, and computing a failure plan to deal with swift recovery.

Detecting host failure reliably is difficult since you need to remotely distinguish between a host disappearing for a while versus exploding in a ball of flames. If we mistakenly decide that a master host has broken down and elect a new master in its place, there may be unpredictable results if the original host were to make a comeback! Similarly, if there is a network issue and a resource pool splits into two equal halves, we need to ensure that only one half accesses the shared storage and not both simultaneously.

Heartbeating for availability

We solve all these problems in XenServer by having two mechanisms: a storage heartbeat and a network heartbeat. When you enable HA in a pool, you must nominate an iSCSI or FC storage repository to be the heartbeat SR. XenServer automatically creates a couple of small virtual disks in this SR. The first disk is used by every physical host in the resource pool as a shared quorum disk. Each host allocates itself a unique block in the shared disk and regularly writes to the block to indicate that it is alive.

I asked Dave 'highly available' Scott, the principal engineer behind HA about the startup process:

“When HA starts up, all hosts exchange data over both network and storage channels, indicating which hosts they can see over both channels; i.e. which I/O paths are working and which are not. This liveness information is exchanged until a fixed point is reached and all of the hosts are satisfied that they are in agreement about what they can see. When this happens, the HA functionality is ‘armed’ and the pool is protected.”

This HA arming process can take a few minutes to settle for larger pools, but is only required when HA is first enabled.

Once HA is active, each host regularly writes storage updates to the heartbeat virtual disk, and network packets over the management interface. It is vital to ensure that network adapters are bonded for resilience, and that storage interfaces are using dynamic multipathing where supported. This will ensure that any single adapter or wiring failures do not result in any availability issues.

The worst-case scenario for HA is the situation where a host is thought to be off-line but is actually still writing to the shared storage, since this can result in corruption of persistent data. In order to prevent this situation without requiring active power strip control, we implemented hypervisor-level fencing. This is a Xen modification which will hard-power the host off at a very low-level if it doesn’t hear regularly from a watchdog process running in the control domain. Since it is implemented at a very low-level, this also covers the case where the control domain becomes unresponsive for any reason.

Hosts will self-fence (i.e. power off and restart) in the event of any heartbeat failure unless any of the following hold true:

The storage heartbeat is present for all hosts but the network has partitioned (so that there are now two groups of hosts). In this case, all of the hosts which are members of the largest network partition stay running, and the hosts in the smaller network partition self-fence. The assumption here is that the network outage has isolated the VMs, and they ought to be restarted on a host with working networking. If the network partitions are exactly the same size, then only one of them will self-fence according to a stable selection function.
If the storage heartbeat goes away but the network heartbeat remains, then the hosts check to see if they can see all other hosts over the network. If this condition holds true, then the hosts remain running on the assumption that the storage heartbeat server has gone away. This doesn’t compromise VM safety, but any network glitches will result in fencing since that would mean both heartbeats have disappeared.

Planning for failure

The heartbeat system gives us reliable notification of host failure, and so we move onto the second step of HA: capacity planning for failure.

A resource pool consists of several physical hosts (say, 16), each with potentially different amounts of host memory and a different number of running VMs. In order to ensure that no single host failure will result in the VMs on that host being unrestartable (e.g. due to insufficient memory on any other host), the XenServer pool dynamically computes a failure plan which calculates the actions that would be taken on any host failure.

But there’s one more complexity… a single host failure plan does not cover more advanced cases such as network partitions which take out entire groups of hosts. It would be very useful to be able to create a plan that could tolerate more than a single host failure, so that administrators could ignore the first host failure and be safe in the knowledge that (for example) three more hosts could fail before the pool runs out of spare capacity.

That’s exactly what we do in XenServer… the resource pool dynamically computes a failure plan which considers the “number of host failures to tolerate” (or nhtol). This represents the number of disposable servers in a pool for a given set of protected VMs.

The planning algorithms are pretty complex, since doing a brute force search of all possible failures across all hosts across all VMs is an exponential problem. We apply heuristics to ensure we can compute a plan in a reasonably small time:

for up to 3 host failures, we do a comprehensive search which tries almost all permutations. This covers corner cases such as having hosts or VMs with very different amounts of memory (e.g. 4GB vs 128GB). Rather than calculate memory slots or otherwise approximate results, we just deal with them individually and give very accurate plans.
for greater than 3 host failures, we make conservative decisions by approximating every VM to be as large as the largest, and considering each host to be the same as the most densely packed host. We do not approximate the host memory, and so having pools with uneven amounts of host memory will be fine. However, in approximate planning mode having a single very large VM will result in a low nhtol value. If this is a problem, then try to reduce the nhtol or try to have a more even spread of VM memory sizes.

Since planning algorithms are designed for unexpected host failures, we only consider absolutely essential resource reservations which would prevent the VM from starting on the alternative host (e.g. storage is visible, and enough memory is present). We do not perform CPU reservation on the basis that it can be optimised at a later stage via live relocation once the VM is back up and running.

Overcommit protection

We now have HA armed and a failover plan for our VMs. But what if you want to make changes to your configuration after HA is enabled? This is dealt with via overcommit protection.

The XenServer pool dynamically calculates a new failover plan in response to every XenAPI call which would affect it (e.g. starting a new VM). If a new plan cannot be calculated due to insufficient resources across the pool, the XenServer will return an overcommitment error message to the client which blocks the operation.

The “What if?” Machine

This overcommit protection would be quite irritating if you have to keep trying things and seeing if a plan exists or not, and so we built in a ”What If?” machine into XenServer to facilitate counter-factual reasoning.

When reconfiguring HA via XenCenter, you can supply a hypothetical series of VM priorities, and XenServer will return a number of host failures which would be tolerated under this scheme. This lets you try various combinations of VM protections depending on your business needs, and see if the number of host failures is appropriate to the level of paranoia you desire.

This can even be done via the CLI, using the snappily named ”xe pool-ha-compute-max-host-failures-to-tolerate” when HA is enabled.

The nice thing about XenServer HA is that it is done at the XenAPI level, and so any of the standard clients (such as the xe CLI or XenCenter) or any third-party clients which use the XenAPI will all interoperate just fine. The XenServer pool dynamically recalculates plans in response to the client requests, and so no special “oracle” is required outside of the pool to figure out HA plans.

Finally, HA makes master election completely invisible. Any host in a pool can be a master host, and the pool database is constantly replicated across all nodes and also backed up to shared storage on the heartbeat SR for additional safety. Any XenAPI client can connect to any host, and a redirect is issued to the current master host.

Protection Levels

Each VM in an HA pool can be either fully protected, best-effort or unprotected. VMs which are protected are all included in the failover planning, and if no plan exists for which they can all be reliably restarted then the pool is considered to be overcommitted. Hugh Warrington (who implemented the XenCenter HA support) explained what use protection levels are:

“Best-effort VMs are not considered when calculating a failover plan, but the pool will still try to start them as a one-off if a host that is running them fails. This restart is attempted after all protected VMs are restarted, and if the attempt to start them fails then it will not be retried. This is a useful setting for test/dev VMs which aren’t critical to keep running, but would be nice to do so in a pool which also has some important VMs which absolutely must run.”

There are some advanced features which are only available via the CLI. Each protected VM in an HA pool can be assigned a numeric ha-restart-priority. If a pool is well-resourced with a high nhtol, then these restart priorities are not relevant: the VMs are all guaranteed to be started.

If more hosts fail than have been planned for, then the priorities are used to determine the order in which VMs are restarted. This ensures that in over-committed pools, the most important VMs are restarted first. Although the pool will start priority 1 VMs first, they might not finish booting before the priority 2 VMs, and so this should not be used as the basis for service ordering.

Note that it's very important to ensure that a VM is agile when protecting it by HA. If the VM is not agile (e.g has a physical CD drive mapped in from a host), then it can only be assigned Best Effort restart since it is tied to one host.

XenCenter support for HA

The best practice for HA is not to make configuration changes while it is enabled. Instead, it is intended to be the "2am safeguard" which will restart hosts in the event of a problem when there isn't a human administrator nearby. If you are actively making configuration changes such as applying patches, then HA should be disabled for the duration of these changes.

XenCenter makes some common changes under HA much more user-friendly, which I asked Ewan Mellor (the principal GUI engineer) about:

Normally a protected VM cannot be shut down via the CLI or from within the guest (a shutdown from within the guest will automatically restart it). If you try to shutdown from XenCenter, it will give you the option of unprotecting the VM and then shutting it down first. Thus, accidental in-guest shutdowns wont result in downtime, but administrators can still stop a protected guest if they really want to.
If you want to reboot a host when HA is enabled, XenCenter automatically uses the hypothetical planning calculation to determine if this would invalidate the failover plan. If it doesn’t affect it, then the host is shut down normally. If the plan would be violated, but the nhtol is greater than 1, XenCenter will give the administrator the option of lowering the nhtol value by 1. This reduces the overall resilience of the pool, but always ensures that at least one host failure will be tolerated. When the host comes back up, the plan is automatically recalculated and the original nhtol value restored if appropriate.
If you try to apply a hotfix, then XenCenter will disable HA for the duration of the pool patching wizard. It is important to manually keep an eye on hotfix application to ensure that host failures do not disrupt the operation of the pool.

So, I hope this short article has given you a taster… just kidding! This post is almost as long as my PhD thesis, but then, HA is a complex topic. Please do feel free to get back to me with comments and feedback about how we can improve it in the future releases, or if you just love it the way it is. Many thanks to Dave Scott, Richard Sharp, Ewan Mellor and Hugh Warrington for their input to this article.

Shedding some light on XenApp on XenServer performance tuning

2008-08-04T00:00:00+01:00

You won’t be surprised to hear that we spend a lot of time improving XenApp performance when running on XenServer. Although there are some good benchmark comparisons available (such as the Tolly Group report), I still get a lot of customers asking about what the “secret sauce” is. I sat down with George Dunlap, the lead XenServer performance engineer to chat about the very first optimisation we did back in XenServer 4.0 last year.

Before we dive in, we first need to explain how a normal operating system handles memory. George explains:

“Modern desktop and server processors don’t access memory directly using its physical address. They use ’virtual memory’ to separate the addresses that processes use to read and write memory from the actual memory itself. This allows operating systems to hide from processes all the dirty details of how much memory there is, where in physical memory the process needs to write to, and so on.”

“However, the actual processor still needs to translate from a virtual address to the physical memory address in order to actually read and write any memory. This translation is done with something called page tables.”

“Page tables are used to implement virtual memory by mapping virtual addresses to physical addresses. The operating system constructs page tables using physical memory addresses, and then puts the physical address of the “top-level” page table into a hardware register called the ‘base pointer’. Then the processor will read these page tables to translate virtual addresses to physical addresses as needed, before reading and writing to physical memory.”

Most modern processor types have some sort of paging mechanism, although XenServer is specifically tuned for x86-64 CPUs. An excellent book on the general topic is Modern Operating Systems by Andrew Tanenbaum. When XenServer creates Windows VMs, it takes advantage of the virtualization extensions in modern CPUs, which requires special memory handling in Xen. George explains this further:

“When we create a virtual machine, we virtualize the memory as well; that means that the guest operating system’s idea of physical memory does not match up to real physical memory on the host. Traditionally, what the guest thinks of as physical memory is called “physical memory”, and what the hypervisor thinks of as physical memory is called “machine memory”. Since this terminology is a bit confusing, Xen tends to call what the guest thinks of as physical memory as “guest physical” memory, just to help make things more clear.”

“This means that any fully-virtualized operating system, like Windows, will create page tables using guest physical memory, and will point the base pointer at the guest physical address of the top-level page table. Unfortunately, the hardware still needs to map from virtual memory address to machine addresses, not guest physical addresses.”

“In order to allow this to happen, the hypervisor sets up shadow page tables. These page tables are generated by the hypervisor are copies of the guest page tables, but with the guest physical addresses converted into machine physical addresses. The guest cannot access them directly, and they don’t reside in the guest’s physical memory; they’re generated out of a pool of memory that the hypervisor allocates when a VM is created, called “shadow page table memory”.”

“What this means is that whenever the guest operating system wants to map some new memory, after it writes the data into the page table but before it can actually use it, the hypervisor needs to translate the change to the guest page table into changes to the shadow page table. So any workload that involves a lot of this will necessarily involve the hypervisor a lot, which causes overhead.”

So shadow page tables are our mechanism of giving a guest an interface which is identical to real hardware (so it doesn’t need to be modified), but still intercepting changes before they reach the real hardware. You can find more details from the XenSummit 2006 talk or from the 2005 NSDI paper. So how is this all relevant to XenApp performance? Back to George…

“The hypervisor allocates a certain amount of memory for each VM to use for shadow page tables; this is called shadow page table memory. As new page tables are created and old ones aren’t used anymore, the hypervisor cycles through this shadow page table memory. When it needs a new page and there isn’t enough, it will ‘unshadow’ the guest page tables that haven’t been used for the longest time to reclaim shadow memory, so that it can use more.”

“We don’t know ahead of time how much shadow memory a given workload will use, but we can estimate based on the amount of memory that the VM has. We allocate enough shadow memory for each page to be mapped once, more or less, then add an extra 50% to have some slack. For all the workloads we’ve tested, that’s been enough – except XenApp.”

“XenApp is the one workload we’ve found that requires more shadow page table memory than our standard default. Because XenApp generally starts hundreds of copies of the same process, the same memory ends up mapped in hundreds of different processes. What happens when all of those processes are active is that XenServer is continually unshadowing one process’ page tables in order to shadow another process’ pagetables; only to have to re-shadow the original ones a second or two later when it runs again! This is called thrashing, when there’s not enough of a limited resource.”

Once the bottleneck was discovered, the solution was simple. In XenServer 4.1, we created a special XenServer application template called "Citrix XenApp", which has an increased shadow multiplier that reserves more shadow memory for the guest when it starts. This is also a good example of how templates hide the complexities of performance tuning from the user, but still permitting custom modifications if they are required. For example, on your XenServer host with a VM called “XenApp”, you could view the shadow multiplier by using the CLI:

# xe vm-list name-label=XenApp params=HVM-shadow-multiplier
  HVM-shadow-multiplier ( RW)    : 4.000

The same value is also available from XenCenter in the Optimization pane, but of course do remember that the default value was chosen through extensive testing and doesn’t need to be changed. Most of the other templates in XenServer also have carefully tuned settings (e.g. the hardware platform flags) to ensure smooth running, or in the case of Linux templates, to support para-virtual installation. This is why it’s so important that you not use the "Other Install Media" template in preference of a more specialised one!

I mentioned at the beginning of this post that this was the first of many XenApp optimisations. We’ve just released the public beta of the latest XenServer (“Orlando”) which is even faster. The story of what those improvements are, and the tools which George and his team uses to analyze the inner workings of Xen, are a topic for a future post. For now, get downloading XenServer and start virtualizing your XenApp installations! Or if you’re feeling inspired, go over to xen.org, check out the source, and get coding…

Installing Ubuntu on XenServer

2008-07-02T00:00:00+01:00

I thought I’d kick off my Citrix blog with a question I get pretty often from Linux enthusiasts: how to install unsupported Linux distributions on XenServer 4.1.

The most common solution people find is to use the “Other Install Media” template, insert the distribution installation CD, and find that the mouse cursor doesn’t work when they boot into X11. The reason for this is that they are using the hardware-assisted emulation mode of installing Linux. In this mode (dubbed “HVM”), all input and output is emulated, and in particular the mouse interface uses the USB tablet interface. If the distribution doesn’t include a driver for USB tablets, then no mouse will appear.

Windows guests run at high-speed in HVM mode due to the installation of the XenServer tools which install high-speed drivers, but these are not necessary for Linux distributions since they can be run in para-virtualized mode (dubbed “PV”). This involves obtaining a Xen-enabled PV kernel from the distribution, and modifying the VM record in XenServer to boot into this kernel instead of HVM mode. The XenServer built-in templates for popular distributions such as RHEL, CentOS or SUSE Linux already automate all this and are in PV mode from the installer onwards.

In the remainder of this post, I’ll explain how to take a distribution without direct support (Ubuntu 8.04), get it installed in HVM mode on XenServer 4.1, and convert it to PV mode with a XenCenter graphical console.

Download the "Alternative Installation CD". The main installation CD uses graphical mode, which won't install as well in HVM mode due to the use of esoteric 16-bit mode instructions for the graphics operations. The 16-bit emulation mechanisms vary between processors (with better support on AMD chips, and a software instruction emulator required on Intel VT chips). However, the Ubuntu alternate CD uses a text-based installer which works fine.

Create a new VM on the XenServer 4.1 host using the "Windows Server 2003" template. This template is set up with a sensible set of hardware emulation flags and default disks, and so is a good base for the HVM installation of Ubuntu as well. Attach the Ubuntu ISO you just downloaded to the VM, and proceed to install Ubuntu as normal. You should install it onto the first disk, to make the subsequent steps in this guide easier.

When the installation is finished, reboot the VM (don't forget to detach the installation ISO first). It should boot up in HVM mode into the graphical login screen. The XenCenter display will show it as not being optimized, which is fine. At this stage, I prefer to work via a remote command-line using SSH. Open up a Terminal from Ubuntu, and run "sudo apt-get install openssh-server". Then find out the VM's IP address with "ifconfig eth0", and then connect to it remotely. Alternatively, you can continue to type in the commands directly into the terminal as well.

On the Ubuntu guest, you now need to install the latest Xen version of the Ubuntu kernel:
- Install the Linux kernel virtual package with "sudo apt-get install linux-image-xen". This is a virtual package which pulls in the latest Xen kernel and modules, in my case 2.6.24.19.21.
- You now need to workaround a bug in grub. Due to the switch in recent versions of Linux to work with the hypervisor-independent paravirt_ops interface, update-grub doesn't update the grub configuration with your newly installed Xen kernel. To fix this:
  - Open /boot/grub/menu.lst in your favourite editor.
  - Scroll to the bottom to the kernel list, and find the entry which looks like:
    title Ubuntu 8.04, kernel 2.6.24-16-generic root (hd0,0) kernel /boot/vmlinuz-2.6.24-16-generic root=UUID=<uuid> ro quiet splash initrd /boot/initrd.img-2.6.24-16-generic quiet
  - Add a new entry which is similar to this, but change all references to the 2.6.24-16-generic to the Xen kernel. In /boot I have vmlinuz-2.6.24-19-xen, so my new entry looks like:
    title Ubuntu 8.04, kernel 2.6.24-19-xen root (hd0,0) kernel /boot/vmlinuz-2.6.24-19-xen root=UUID=<uuid> ro quiet splash initrd /boot/initrd.img-2.6.24-19-xen quiet
  - Also edit the default entry in the menu.lst to match the number of the kernel you just added. I set mine to 3, since it is the fourth entry in the list and the indexing starts from 0.

When this is done, shut down the guest but do not reboot it just yet. You first need to edit the VM record for your Ubuntu VM to convert it to PV boot mode. From the control domain console of your XenServer:
- Determine the UUID of the Ubuntu VM by using the xe CLI:
  - xe vm-list name-label=Ubuntu params=uuid --minimal : this will print out the UUID of the VM named "Ubuntu". If you are logged into the control domain, pressing the <tab> key will perform auto-completion of UUIDs in subsequent XE commands, so you don't need to keep typing it in every time!
  - xe vm-param-set uuid=<uuid> HVM-boot-policy= : this will clear the HVM boot mode from the VM.
  - xe vm-param-set uuid=<uuid> PV-bootloader=pygrub : this will switch the VM to using to the pygrub bootloader which starts the guest in PV mode by examining its filesystem for kernel.
  - vm vm-param-set uuid=<uuid> PV-args="console=tty0 xencons=tty" : this configures the kernel boot arguments to display the login console on the correct TTY, so that it shows up in the XenCenter console.
- Next, you need to flag the root disk of the VM as bootable so that pygrub knows where to look for the PV kernel:
  - xe vm-disk-list uuid=<uuid> and look for the UUID of the VBD for the disk. VBD stands for "Virtual Block Device" and represents how to map the virtual disk into the virtual machine.
  - xe vbd-param-set uuid=<vbd uuid> bootable=true will set the root disk VBD to be bootable.

You should be all set now! If you boot up the Ubuntu VM, it should start up in text-mode with the high-speed PV kernel. If it doesn't work due to an incorrect grub configuration, you can use the xe-edit-bootloader script in the XenServer control domain to edit the grub.conf until it works.

The next step is to install the XenServer tools within the guest, so that metrics such as the network interface IP addresses are recorded and reported from XenCenter. To do this:
- Due to a portability issues with the default shell in Ubuntu (dash), you will need to replace it by: sudo apt-get -y install bash && sudo dpkg-reconfigure dash. We've actually fixed this issue in future releases of XenServer, but for XenServer 4.1 you will need to use bash.
- Attach the XenServer Tools ISO into the VM, and mount it into the guest with sudo mount /dev/xvdd /mnt
- Install the tools with sudo dpkg -i /mnt/Linux/xe-guest-utilities_4.1.0-257_i386.deb.
- The warnings about the VM being unoptimized should disappear, and additional information such as the IP address of the guest should appear in XenCenter.

In order to access the Ubuntu installation via the graphical console, you need to configure it to run VNC on the external network interface. XenCenter polls the guest to see if it is listening on the VNC port 5900, and offers the option to switch to the graphical console if it finds it. I followed the excellent instructions on this forum post. To summarise them:
- sudo apt-get install vnc4server xinetd : to install the required packages
- Edit /etc/gdm/gdm.conf and uncomment the RemoteGreeter=/usr/lib/gdm/gdmlogin line, set the key Enable=true in the [xdcmp] section.
- Install a new service file for xinetd into /etc/xinetd.d/Xvnc with the following contents:
```
service Xvnc
{
  type = UNLISTED
  disable = no
  socket_type = stream
  protocol = tcp
  wait = no
  user = nobody
  server = /usr/bin/Xvnc
  server_args = -inetd -query localhost -geometry 1024x768  -depth 16 -cc 3 -once -SecurityTypes=none -extension XFIXES
  port = 5900
}
```
- The major difference from the forum poster is to run it on port 5900, and not to restrict it to just localhost (since XenCenter also needs to connect to it).
- Finally, restart the xinetd service by running sudo /etc/init.d/xinetd restart.

Once you're done with this installation, you can shut down the VM and convert it to a template. Any exports or clones will continue to run in PV mode, since the XenServer XVA export format records all of the metadata required to re-create the VM records.

Enjoy the Ubuntu on XenServer experience! Remember to report any issues you have with the in-guest packages on the Ubuntu support forums, or just give them positive feedback.

PS: many thanks to Andrew Peace and Ian Campbell for assistance. May their Linux beards remain long and uncut.

Parsing EXIF/IPTC photo tags using pyexiv2 on Leopard

2008-04-17T00:00:00+01:00

I’ve been looking around at refreshing my current photo gallery to make it easier to update, since I’m around 1000 photos behind over the last year. Part of making it easier to use is to adopt an Aperture-based system of marking the metadata directly in the image itself, and figuring out how to render so many pictures more effectively in my gallery.

The best library I could find to do the job is Exiv2 and its associated Python bindings pyexiv2. They use the rather complex Boost.Python bindings, which are a total pain to compile on MacOS X Leopard.

In order to get it to work:

Use the latest Fink to install:
- python25
- boost1.34.python25
- libexiv2
- scons
Download the latest pyexiv2 distribution, extract it, and drop in my exiv2 Leopard Makefile into the extracted directory.
Type in:
- make
- sudo make install.
Test it out by running python2.5 and experimenting with importing pyexiv2 (see the pyexiv2 developer guide for more information.

I’ll tidy this up into a fink package sometime, but for now I want to press on with finishing my Django experiments.

Sushi, Seattle and Seahawks

2007-04-11T00:00:00+01:00

So I’m hanging at the XenSource Redmond office for a few days, and munching on delicious sushi at Flo, and it turns out that the dude behind us was none other than Matt Hasselbeck! It seems like only two years ago that I was cheering on the Pittsburgh Steelers and the Bus as they owned the Seahawks.

Unfortunately, I only realised my brief brush with celebrity on our way out of the restaurant, which also probably saved me from saying something stupid to him. As if! But anyway, the, err, moral of this post is that you must check out Flo Sushi if you’re in Seattle and eat the spider roll, it’s awesome.

Karnatik Jazz, Live Painting, and such things...

2007-04-08T00:00:00+01:00

Had a great evening out in San Francisco on Friday, when we visited the Red Poppy Art House on Folsom Street to check out a really unique Jazz experience.

The boys from VidyA recently took up an artist-in-residence position there, and performed their fusion of traditional Indian Karnatik music with modern Jazz pieces. They’ve been making some waves in the SF Jazz scene already it seems, and to top it all of, the owner of the art house did a live painting session during the performance. The audience passed out random bits of stuff, and he incorporated it into a piece done over the course of an evening. Talk about artistic overload!

Love the area as well; there’s a nice friendly coffee-house down the road (which is pretty much my only criteria, apart from not getting shot and such).

My school was never this cool.

The area was a mix of Latino and African, it seemed

And in the Red Poppy Art House, eclectic art abounds

Core stability exercising

2007-04-08T00:00:00+01:00

It’s been a wierd exercise year; I went from hard-core powerlifting (go 220lb bench!), to playing a lot of badminton and tennis, and then ending off the PhD era with breaking my knee (details not suitable for blog publication).

As anyone it’s happened to knows, a knee injury is really annoying to recover from as it feels really “wobbly” for ages and can’t quite be trusted. I’ve been doing physio on it myself for about half a year now, and can finally run around for 5km or so without too much discomfort, although yoga and Capoeria are perhaps a few more months away.

But then, the Forrest Gump-class runner and ex-officemate Alex Ho pointed out these amazing exercises by the captain of the CUH&H running team in Cambridge. They all emphasise smooth movement and core strength without the need for heavy weights, so can be done while travelling as well. So highly recommended, that even little Ajay (right) is trying them out!

Trac spamming is taking down Melange

2007-04-02T00:00:00+01:00

After loads of reports that the Melange site keeps going 505 and crashing, I took at look at why. Turns out several spam crawlers were going mental and repeatedly adding tickets with spam links. Around 600 tickets and thousands of comments later, the process decided it had enough and terminated.

I’ve deleted all tickets (even the valid ones were spammed into oblivion) and turned off comment creation and modification for anonymous users. A look on the Trac Wiki shows that there are some SpamFilter extensions being developed which I’ll investigate at some point.

Melange hits the euro-spotlight

2007-04-01T00:00:00+01:00

It’s been a busy old March (release management of the new XenEnterprise sucked up most of it). I did take a break and go over to EuroSys 2007 in Portugal to present the language and compiler I implemented as part of my PhD work (read it here).

Although the talk I gave was was a bit underwhelming (more preparation and time-practise next time!), I met a whole bunch of really interesting people. My argument about rewriting whole applications also didn’t get laughed out the room as I thought it might, as people recognise that retro-fitting safety enhancements on existing languages is a bit of a dead-end road to go down. It has definitely inspired me to make more time to spend on polishing up the Melange applications for a proper release in 2007.

In a pleasant surprise, it also won the Best Student Paper award of the conference as well!

Hans Rosling at TED last year on global development

2007-04-01T00:00:00+01:00

Having coffee with AliB and discussing lecturing reminded me of one of the best presentations I’ve ever seen: Hans Rosling at TED and debunking myths about the developing world.

Even if you’re not interested in the subject, it’s worth checking out just to see how cool his presentation of otherwise rather boring statistics is. His company, Gapminder also got bought by Google a few weeks ago, so chances are Analytics will get more interesting soon.

Deens, welcome to the Internet!

2006-12-30T00:00:00+00:00

Inspired by finishing my PhD corrections (!) today, I decided to hook up the DNS server from our Melange project up to the Internet. The authoritative server is called deens (since the co-author is one Tim Deegan, geddit?), and is written in pure OCaml.

This is all rather experimental, to put it mildly, but I stuck in the zone file below, hooked it up as a delegate to our main name-servers, checked it against the DNS Report, and it all seems to be working!

$ORIGIN deens.recoil.org. ;
$TTL    240
deens.recoil.org. 604800 IN SOA  (
    deens.recoil.org. anil.recoil.org.
    2006122401 3600 1800 3024000 1800
)
        IN  NS     ns1.deens.recoil.org.
        IN  NS     deensns.recoil.org.
ns1     IN  A      194.70.3.132
dynamic IN  CNAME  dynamic.recoil.org.
static  IN  CNAME  static.recoil.org.
anil    IN  CNAME  dynamic
stats   IN  CNAME  dynamic

I also modified stats.recoil.org to be an alias to stats.deens.recoil.org, so all the requests for that domain will go via the deens setup. You actually need a user/pass to access the site, but that doesn’t matter; if it gets that far, the DNS bit has worked.

There’s still an awful lot of tedious work to get the server into a production-ready state, such as proper logging, more error handling and recovery, etc., but I really hope to find the time in 2007 to polish this up somewhat. Performance is excellent already; faster than BIND by quite a lot, and it can optionally use more memory to cache responses to shoot up to crazy levels.

Incidentally, the dig replacement utility also seems to be working fairly well, and David Scott has been messing around with a Bonjour implementation that will get finished sometime in 2007 as well (honest!).

Google Webmaster tools

2006-12-28T00:00:00+00:00

The conversion of the Recoil web services to external FastCGI pinned our Trac installation at Melange as the source of the CPU hogging. It turned out the Google crawler was indexing the entire source tree via Trac, causing it to go ballistic.

I then stumbled on the latest cool Googlism: the Google Webmaster Tool, which lets you register your sites and displays options, diagnostics and statistics about how the Google crawler views your website. I turned down the frequency at which Google hits the Trac installation (as well as installing a suitable robots.txt file). This solved the immediate problem, but some of the search statistics were fun to check out as well.

It turns out the gallery is pretty highly ranked for image searches. My trips to Japan seems to have made it big, with popular searches including ”Shibuya”, ”tokyo at night”, and ”japanese roof”. My random pictures of indian buffaloes, smoggy skylines and fried ice-cream seem especially popular as well. It’s a wierd old Internet eh?

The gallery has fallen a bit by the wayside in recent months. I’ll update it when I get back to Cambridge!

Mercurial FastCGI module

2006-12-27T00:00:00+00:00

Our lighttpd setup has been very unstable in recent months, probably brought on by the load of the large Mercurial repositories hosted on Recoil since the Google Summer of Code mentoring.

The source of the instability was really hard to track down, but it seems to be the automatic spawning of FastCGI processes by the web-server, and lighttpd failing to handle a SIGCHLD somewhere when a child process crashes. To sort this out, I just converted all the Ruby on Rails setups (this blog and Nick’s) to use an external spawn.

This only leaves our Mercurial vhost hg.recoil.org to switch to using FastCGI, and I couldn’t find a module for this anywhere and so lashed up some Python glue to do the job.

You can download the small distribution for Mercurial 0.9 (hg-fcgi-0.9.tar.gz). It has a FastCGI library written by someone else, the Python files to glue the Mercurial and FastCGI libraries together, and a simple rc script to launch the external web process. Instructions are for lighttpd… install the Python files somewhere, modify them to point to the Mercurial directory, run the rc script to start the daemon, and then add something similar to the following to your lighttpd config file:

fastcgi.server = (
  ".fcgi" => ( "localhost" =>
    ( "socket" => "/var/cache/fcgi/sites/hg.recoil.org/dirsock" )),
  ".hg" => ( "localhost" =>
    ( "socket" => "/var/cache/fcgi/sites/hg.recoil.org/sock" )),
)

Also add “index.fcgi” to index-file.names in the config file, and touch it in the vhost directory to create an empty file (this is to avoid getting a 404 error and instead pass it through to the FastCGI process). Similarly, touch a .hg file for every repository you want to serve. You could do this differently by passing through a URL prefix and modifying the Python appropriately, but I prefer finer control over what we’re serving.

Hope this is useful; I won’t bother submitting it back to the Mercurial list as it looks like the official hg repo has a different code layout; I’ll check it out later on when I have a bit more time and integrate properly.

I have no idea whether or not this will actually improve our stability, but it’s at least easier to move onto a different web-server now that everything is FastCGI. All I need now is an OpenBSD/php5-fastcgi port, which doesn’t seem to exist (yet).

Looking my Spam statistics

2006-12-27T00:00:00+00:00

The switch to qpsmtpd does seem to have reduced my spam intake somewhat, so out of curiousity I looked at the statistics from 2 years of procmail logs to see what’s been happening in terms of filtering effectiveness.

A quick import and bug-fix of Log::Procmail into OpenBSD, and some lashed up Perl and gnuplot later, the graph on the right showed up. The red and green are ham and spam respectively, as classified by SpamAssassin.

The large amount of ham in 2004 was not actually real mail, but mostly postmaster bounces from forged spam; I am currently forced to destroy all domain bounces without even reading them due to the sheer volume. This is something that Sender Permitted From promises to help solve once we determine if any our users send @recoil.org mail from sources other than our mail server.

Since the turn of this year the amount of spam has jumped, but more concerningly, SpamAssassin has been missing increasing amounts, and it’s been flowing through straight to my Inbox (despite sa-update running daily). I’m going to do these graphs again in a few months and see just how much the switch to the new paranoid SMTP has helped.

Christmas Spam Cleanup

2006-12-25T00:00:00+00:00

It’s Christmas Day, I’ve eaten far too much, and am lounging around doing the now-traditional Annual Recoil Cleanup as the year’s todo list has grown ever larger. I’ve been meaning to switch from our venerable qmail-smtpd for some years now, and finally made the move over to qpsmtpd.

qpsmtpd is a drop-in replacement for the SMTP portion of qmail, and is written in Perl with a number of plug-ins which lets us increase our paranoia levels considerably. It’s a pity we have to do this, but the policy of ‘accept anything’ has been under increasing stress for the last few years, and when I looked at my e-mail stats last night, I realised over 99.99% of my incoming e-mail was some kind of virus or spam. Even a 1% miss rate on SpamAssassin is enough to chuck 100s of mails into my inbox!

So now the new e-mail setup at Recoil includes virus scanning via the wonderful clamav, reverse DNS RBL looksup via rfc-ignorant.org, and even early-chatter detection of viruses which blindly blast messages before the initial SMTP greeting has completed. I’m hoping to enable global SpamAssassin checking soon if all else is stable and I don’t get bleating about missing mail from our users.

I played with Greylisting as well to see if it had improved from my earlier experiments a couple of years ago. Unfortunately, it still looks as if there are many broken MTAs out there which don’t cope well with rejection, and manual whitelists are required, which sounds a bit unreliable for setups like ours which sometimes don’t get looked at for years on end (ahem).

So it’s with a tear in my eye that I wave goodbye to qmail-smtpd, the first ever network-facing service deployed on Recoil back in 1998, and incredibly, the only one I’ve never had to upgrade in the 8 years since.