Building Xenopsd

What is xenopsd?

Xenopsd is the "domain management" component within the XAPI toolstack. When a VM is started, xenopsd is the one who issues the hypercalls to xen (via the xenctrl library), builds the initial guest memory layout (via the xenguest library) and generally looks after the VM while it is running (or rebooting, suspending, resuming, migrating etc). Xenopsd is a single-host service. The only time xenopsd talks over the network is when it is sending or receiving a VM during a cross-host migration.

Note xenopsd knows nothing about "resource pools" or "clouds": these concepts belong to higher-level layers (xapi and {open,cloud}stack).

Who is this document for?

This document is for people who want to build/run/test the latest development version of the xenopsd component. For production use, you should use a pre-packaged stable version from your distribution vendor. If you want to

1. test the latest development version before a release;
2. reproduce a bug;
3. test a patch; or
4. have a bit of fun on your laptop

then the development version of xenopsd is for you!

Note this document is quite new and is still being debugged. If you find a problem, first see if you can fix it (this is a wiki after all). If the fix isn't obvious then send an email to <xen-api@lists.xen.org>.

Build from source prerequisites

You will need:

1. the normal build tools (on Debian or Ubuntu install "build-essential git rsync wget m4")
2. the Xen development headers (on Debian or Ubuntu install "libxen-dev")
3. the OCaml compiler (at least version 3.12.1)
4. the OCaml package manager, OPAM

Assuming a Debian wheezy or Ubuntu 12.10 system (other systems may differ), first install the system packages:

 $ sudo apt-get install build-essential git rsync wget m4 libxen-dev ocaml camlp4-extra ocaml-native-compilers autoconf libvirt0 libvirt-dev

Then install the OPAM tool:

 $ wget http://github.com/OCamlPro/opam/archive/0.9.2.tar.gz
 $ tar -xvzf 0.9.2.tar.gz
 $ cd opam-0.9.2
 $ ./configure
 $ make
 $ sudo make install

Next initialise OPAM, this will read in the definitions of common OCaml packages including "uuidm" (for handling uuids), "syslog" (for writing to the system logger), "cmdliner" (for parsing arguments and generating man pages), "rpc" (for generating rpc stubs and skeletons), "type_conv" (for generating nice marshalling/unmarshalling code from data type definitions).

 $ opam init

Since OPAM packages are installed by default into a custom directory in $HOME/.opam, it's necessary to update some paths in your environment. The instructions will be printed to your terminal and typically look like the following:

To update $PATH, $MANPATH, $OCAML_TOPLEVEL_PATH, $CAML_LD_LIBRARY_PATH; you can now run:
           
   $ which opam && eval `opam config -env`

and add this in your ~/.profile

Let's build xenopsd (and its friends)

To make building easy, the development versions of xenopsd (and associated tools) are packaged into OPAM packages in a custom repository. To build, first add this repository as a remote:

 $ opam remote add xen-dev git://github.com/xapi-project/opam-repo-dev

The new packages found in this remote will be listed on your terminal and may be queried via "opam list". At the time of writing this list looked like:

The following NEW packages are available:
- fd-send-recv.1.0.0
- forkexecd.1.0.0
- message_switch.1.0.0
- shared-memory-ring.1.0.0
- xcp.1.0.0
- xen-api-client.1.0.0
- xenctrl.4.1.0
- xenops-cli.1.0.0
- xenopsd.1.0.0
- xenstore.2.0.0
- xenstore_transport.1.0.0

Finally, to build xenopsd (and its CLI) and all their dependencies, use the following command:

 $ opam install xenopsd xenops-cli

The xenospd binary will end up in .opam/system/bin and should be on your $PATH (if you followed the earlier instructions to update your environment)

Running xenopsd

When xenopsd starts it will scan its environment for all the helper programs and services it might possibly need. If something is missing it will give you a, hopefully helpful, error message. One of the dependencies you need is called "vncterm" and is used to provide a VNC console for PV guests.

If you're running Debian or Ubuntu you should be able to use the existing stable packages:

 $ sudo apt-get install xcp-vncterm xcp-eliloader

When xenopsd wants to start another program (such as vncterm or qemu) it starts it indirectly via a "fork/exec" service, rather than fork/exec it directly. There are 2 reasons for this odd behaviour:

1. it works around some bugs in older glibcs where calling exec() in a threaded program with a pthread_atfork() handler would enter an infinite loop (xenopsd is multithreaded)
2. it would be nice to evolve the fork/exec service into a general-purpose "supervisor" process, capable of monitoring children and implementing on-error recovery actions (in the style of Erlang OTP)

Anyway, for the moment we must start the fork/exec service:

 $ sudo ./.opam/system/bin/xcp-fe

The build process also creates a basic configuration file with paths to all the script dependencies filled in, pointing at their installed locations. Start xenopsd using:

 $ sudo ./.opam/system/bin/xenopsd -config ./.opam/system/etc/xenopsd.conf

by default it will not daemonize and continue writing debug messages to the terminal.

If all has gone well, then congratulations! I recommend a grabbing a celebratory coffee before continuing...

Starting a simple VM

Xenopsd's primary interface is over a JSONRPC API. The easiest way to talk this API is via the xenops-cli tool.

To read the CLI help, try:

 $ xenops-cli
 $ xenops-cli add --help

The xenops-cli is able to read and write VM config files in xm/xl syntax. First create a file with contents like:

name='myfirstvm'
builder='hvmloader'
boot='dc'
vcpus=1
memory=256
disk=[ 'phy:/dev/loop0,hda,w', ',hdb:cdrom,r' ]
vif=[ 'mac=00:16:3E:74:3d:77,model=rtl8139,bridge=xenbr0' ]
pci=[  ]
pci_msitranslate=1
pci_power_mgmt=0

Note the use of "phy:" above -- this is most likely to work for now. When we manage to separate the storage infrastructure from xapi it will be possible to name a disk using a GUID and have it be automatically managed.

Also note the use of the bridge definition in the vif config. Xenopsd will assume this bridge has already been created, thus it is up to the user to do this ahead of time.

Xenopsd is designed to only manage VMs which have been "registered" or "added" to it explicitly. This is good because if you're testing on your development machine, xenopsd will (almost) ignore all your important VMs (such as the ones hosting your xenopsd compiler and your dom0) unless you explicitly tell it about them. Hopefully this makes it easier to develop and test on a single machine (such as a laptop). I used the word "almost" because the one case when xenopsd will notice a VM which hasn't been explicitly registered is when it host a driver backend (blkback, netback) for one of the registered VMs. In this case it will manage the device backend, but still leave the backend VM alone.

OK, so now we have to register our first VM with xenopsd:

$ sudo ./.opam/system/bin/xenops-cli add myvm.cfg

Now you should be able to see this VM in xenopsd's registered VM list:

$ sudo ./.opam/system/bin/xenops-cli list

You should be able to start the VM as follows:

$ sudo ./.opam/system/bin/xenops-cli start myfirstvm

Since it's running HVM, you will be able to connect to the qemu VNC display like this:

$ vncviewer localhost:1

Future

There's lots of stuff to do, including:

1. bugfix all the things that are currently broken
2. add support for libxl (in progress: Rob Hoes is working on it)
3. stabilise xenopsd and create a release
4. update the Debian packages
5. package up squeezed, rrdd and xapi in the same way
6. separate the storage code from xapi so it can be used without xapi

Development

If you're interested in taking part or influencing development:

1. join the mailing list <xen-api@lists.xen.org>
2. join the IRC channel #xen-api on freenode
3. check out the code on http://github.com/djs55/xenopsd