# Shedding light on XenApp on XenServer performance tuning

*2008-08-04 — note*


You won’t be surprised to hear that we spend a lot of time improving
[XenApp](http://www.citrix.com/XenApp) performance when running on
[XenServer](http://www.citrix.com/XenServer). Although there are some
good benchmark comparisons available (such as the [Tolly
Group](http://community.citrix.com/x/_4ENAg) report), I still get a lot
of customers asking about what the “secret sauce” is. I sat down with
George Dunlap, the lead XenServer performance engineer to chat about the
very first optimisation we did back in XenServer 4.0 last year.


Before we dive in, we first need to explain how a normal operating
system handles memory. George explains:

> Modern desktop and server processors don’t access memory directly
> using its physical address. They use ‘[virtual
> memory](http://en.wikipedia.org/Virtual_Memory)’ to separate the
> addresses that processes use to read and write memory from the actual
> memory itself. This allows operating systems to hide from processes
> all the dirty details of how much memory there is, where in physical
> memory the process needs to write to, and so on.
> 
> However, the actual processor still needs to translate from a
> [virtual address](http://en.wikipedia.org/wiki/Virtual_address) to the
> physical memory address in order to actually read and write any
> memory. This translation is done with something called [page
> tables](http://en.wikipedia.org/wiki/Page_tables).
> 
> Page tables are used to implement virtual memory by mapping virtual
> addresses to physical addresses. The operating system constructs page
> tables using physical memory addresses, and then puts the physical
> address of the “top-level” page table into a hardware register called
> the ‘base pointer’. Then the processor will read these page tables to
> translate virtual addresses to physical addresses as needed, before
> reading and writing to physical memory.

Most modern processor types have some sort of paging mechanism, although
XenServer is specifically tuned for
[x86-64](http://en.wikipedia.org/wiki/X86-64) CPUs. An excellent book on
the general topic is [Modern Operating
Systems](http://en.wikipedia.org/wiki/Special:BookSources/0130313580) by
[Andrew Tanenbaum](http://www.cs.vu.nl/~ast/). When XenServer creates
Windows VMs, it takes advantage of the [virtualization
extensions](http://en.wikipedia.org/wiki/X86_virtualization) in modern
CPUs, which requires special memory handling in Xen. George explains
this further:

> When we create a virtual machine, we virtualize the memory as well;
> that means that the guest operating system’s idea of physical memory
> does not match up to real physical memory on the host. Traditionally,
> what the guest thinks of as physical memory is called “physical
> memory”, and what the hypervisor thinks of as physical memory is
> called “machine memory”. Since this terminology is a bit confusing,
> Xen tends to call what the guest thinks of as physical memory as
> “guest physical” memory, just to help make things more clear.

> This means that any fully-virtualized operating system, like Windows,
> will create page tables using guest physical memory, and will point
> the base pointer at the guest physical address of the top-level page
> table. Unfortunately, the hardware still needs to map from virtual
> memory address to machine addresses, not guest physical addresses.

> In order to allow this to happen, the hypervisor sets up **shadow
> page tables**. These page tables are generated by the hypervisor are
> copies of the guest page tables, but with the guest physical addresses
> converted into machine physical addresses. The guest cannot access
> them directly, and they don’t reside in the guest’s physical memory;
> they’re generated out of a pool of memory that the hypervisor
> allocates when a VM is created, called shadow page table memory.

> What this means is that whenever the guest operating system wants to
> map some new memory, after it writes the data into the page table but
> before it can actually use it, the hypervisor needs to translate the
> change to the guest page table into changes to the shadow page table.
> So any workload that involves a lot of this will necessarily involve
> the hypervisor a lot, which causes overhead.

So shadow page tables are our mechanism of giving a guest an interface
which is identical to real hardware (so it doesn’t need to be modified),
but still intercepting changes before they reach the real hardware. You
can find more details from the [XenSummit 2006
talk](http://www.xensource.com/files/summit_3/XenSummit_Shadow2.pdf) or
from the 2005 [NSDI
paper](http://www.cl.cam.ac.uk/research/srg/netos/papers/2005-migration-nsdi-pre.pdf).
So how is this all relevant to XenApp performance? Back to George…

> The hypervisor allocates a certain amount of memory for each VM to
> use for shadow page tables; this is called **shadow page table
> memory**. As new page tables are created and old ones aren’t used
> anymore, the hypervisor cycles through this shadow page table memory.
> When it needs a new page and there isn’t enough, it will ‘unshadow’
> the guest page tables that haven’t been used for the longest time to
> reclaim shadow memory, so that it can use more.

> We don’t know ahead of time how much shadow memory a given workload
> will use, but we can estimate based on the amount of memory that the
> VM has. We allocate enough shadow memory for each page to be mapped
> once, more or less, then add an extra 50% to have some slack. For all
> the workloads we’ve tested, that’s been enough – except XenApp.

> XenApp is the one workload we’ve found that requires more shadow page
> table memory than our standard default. Because XenApp generally
> starts hundreds of copies of the same process, the same memory ends up
> mapped in hundreds of different processes. What happens when all of
> those processes are active is that XenServer is continually
> unshadowing one process’ page tables in order to shadow another
> process’ pagetables; only to have to re-shadow the original ones a
> second or two later when it runs again! This is called
> [thrashing](http://en.wikipedia.org/wiki/Thrash_\(computer_science\)),
> when there’s not enough of a limited resource.

Once the bottleneck was discovered, the solution was simple. In
XenServer 4.1, we created a special XenServer application template
called *“Citrix XenApp”*, which has an increased shadow multiplier that
reserves more shadow memory for the guest when it starts. This is also a
good example of how templates hide the complexities of performance
tuning from the user, but still permitting custom modifications if they
are required. For example, on your XenServer host with a VM called
“XenApp”, you could view the shadow multiplier by using the CLI:

```bash
# xe vm-list name-label=XenApp params=HVM-shadow-multiplier
  HVM-shadow-multiplier ( RW)    : 4.000
```

The same value is also available from XenCenter in the Optimization
pane, but of course do remember that the default value was chosen
through extensive testing and doesn’t need to be changed. Most of the
other templates in XenServer also have carefully tuned settings (e.g.
the hardware platform flags) to ensure smooth running, or in the case of
Linux templates, to support [para-virtual
installation](http://docs.xensource.com/XenServer/4.1.0/1.0/en_gb/sdk.html#id2553443).
This is why it’s so important that you not use the *“Other Install
Media”* template in preference of a more specialised one\!

I mentioned at the beginning of this post that this was the first of
many XenApp optimisations. We’ve just released the [public
beta](https://www.citrix.com/English/ss/downloads/details.asp?downloadId=1679827&productId=683148)
of the latest XenServer (“Orlando”) which is even faster. The story of
what those improvements are, and the tools which George and his team
uses to analyze the inner workings of Xen, are a topic for a future
post. For now, get downloading XenServer and start virtualizing your
XenApp installations! Or if you’re feeling inspired, go over to
[xen.org](http://xen.org/), check out the source, and get coding…
Synopsis: Optimize XenApp performance on XenServer with expert tuning tips.
Words: 1253
DOI: 10.59350/48sxx-hjt49

## Related

- [Xen Hypervisor](https://anil.recoil.org/projects/xen) (project, 2002-01-01)

---
Canonical: https://anil.recoil.org/notes/shedding-some-light-on-xenapp-on-xenserver-performance-tuning
Type: note
License: CC BY 4.0 <https://creativecommons.org/licenses/by/4.0/>
Tags: xen, systems, opensource