XenBus

Needs Additions Currently focuses on Linux kernel side details, could be generalized

Writing Xen Drivers: Using XenBus and XenStore

XenBus provides a bus abstraction for paravirtualized drivers to communicate between domains. In practice, the bus is used for configuration negotiation, leaving most data transfer to be done via an interdomain channel composed of a shared page and an event channel.

Xenstore is a centralized configuration database that is accessible by all domains. Management tools configure and control virtual devices by writing values into the database that trigger events in drivers.

Driver and tool developers should use this document as a guide for bringing Xenbus and Xenstore functionality into their code.

Xenbus

Device Drivers

All conventional Xen virtual device drivers should register themselves with the XenBus at initialization. This is done by passing an appropriately initialized struct xenbus_driver to the xenbus_register_frontend() function. Most internal initialization and setup should be postponed until the Xenbus calls the probe callback function.

In the case of a traditional split Xen driver, such as the block driver, the communication between the front and back ends should be established when the probe callback is executed. The block driver is a good example for developers wishing to see an actual implementation (see linux/drivers/block/xen-blkfront.c)

Other drivers

Some drivers, such as the balloon driver, don't fit into the conventional split-driver mold (meaning they don't have front and back ends). These drivers don't register with the Xenbus in the normal way, and therefore don't ever get a probe callback to initialize themselves once the store is up. These drivers should instead call register_xenstore_notifier() to register a notifier callback that can perform store-related initialisation.

Xenstore

Xenstore: Drivers and Tools

The Xenstore is a filesystem-like database that is used by Xen applications and drivers to communicate and store configuration information. Applications and tools should use the store to configure drivers by writing information into keys in the database; drivers should set watches on the appropriate keys and respond to changes appropriately.

As a general rule, users of the store should attempt to use human-readable values whenever possible. This allows a generic browser tool (think gconf-editor) to be used to examine the contents of the store. For example, a toolstack could instruct the shutdown driver to power off by writing a "0" into the appropriate store key. This works, but makes it hard for a viewer to know what "0" means. Instead, toolstacks write "poweroff" or "reboot" into the key, which makes it clear to an observer what action is intended.

Reading and Writing data

The Xenstore API provides methods for manipulating data in the store that resemble standard C functions. In a Linux kernel this interface is defined in include/xen/xenbus.h.

xenbus_printf(DIR, NODE, FORMAT, ...)
xenbus_scanf(DIR, NODE, FORMAT, ...)
xenbus_rm(DIR, NODE)
xenbus_read(DIR, NODE, &LEN)

Developers should use these functions to read and write data in the store. Nodes that do not already exist will be created by a xenbus_printf(). When reading a variable-length string value from a store key, xenbus_read() should be used, which returns a kmalloc'd buffer containing the string. This string should be kfree'd after use.

Transactions

Transactions provide developers with a method for ensuring that multiple operations on the Xenstore are seen as a single atomic operation. Any time multiple operations must be performed before any changes are seen by watchers, a transaction must be used to encapsulate the changes. For example:

xenbus_transaction_start("mydir");
xenbus_printf("mydir", "command", "%s", "do_something");
xenbus_printf("mydir", "arg", "%s", "14");
xenbus_transaction_end(0);

Watches

A "watch" can be placed on a key in the XenStore which causes a callback function to be executed whenever something changes at or below the level where the watch was placed. This allows drivers or applications to respond immediately to changes in the store. For example, the balloon driver watches "memory/target" and immediately attempts to balloon the domain's memory whenever a new target is written to the key:

static struct xenbus_watch xb_watch = {
        .node = "memory",
        .callback = watch_target;
};

ret = register_xenbus_watch(&xb_watch);
if(IS_ERR(ret)) {
        IPRINTK("Failed to initialize balloon watcher\n");
} else {
        IPRINTK("Balloon xenbus watcher initialized\n");
}

Rules and Standards

There are several things developers need to be aware of when utilizing the Xenstore:

General:

• No data that is written into the Xenstore from DomU kernels or tools should be trusted or considered correct by the Dom0 kernel or system management tools.
• When writing drivers, care must be given to the Dom0 case. When Dom0 boots, the Xenbus and Xenstore are not active; therefore, they must not be used.
• Directories and (when possible) keys should be created by the tools, instead of the kernel.

Store organization:

• All routines should specify relative paths in Xenstore calls, which results in a path relative to the "home directory" of the domain within the store.

Permissions

Any guest can read any part of the store, but only if it has permission to do so. Domain 0 may read or write anywhere in the store, regardless of permissions, and permissions are set up by the tools in domain 0, or by Xenstored when it first starts up.

The permission semantics for Xenstore are a bit strange; here they are:

There are calls in the Python layer -- xstransact.SetPermissions and xstransact.set_permissions -- and a corresponding C layer call -- xs_set_permissions -- each of which takes a path, and a list of (domid, permissions) pairs. I shall use the Python syntax, as this is clearer. In this case, read and write flags are specified, which are packed into the "permissions" field in the tuple.

xstransact.SetPermissions(path, { 'dom'   : dom1,
                                  'read'  : True,
                                  'write' : False },
                                { 'dom'   : dom2,
                                  'read'  : True,
                                  'write' : True },
                                { 'dom'   : dom3,
                                  'read'  : True,
                                  'write' : True })

This looks clear, but actually the semantics of this are strange. The first element in this list specifies the owner of the path, plus the read and write access flags for every domain not explicitly specified subsequently. The owner always has read and write access to their nodes and the ability to change permissions. The subsequent entries are normal capabilities.

The example above, therefore, sets the permissions in the path to be such that domain 0 (being privileged), dom1 (being the owner), and domains dom2 and dom3 (being explicitly specified) can _all_ write to the node. Any other domain can only read, as specified by the first pair of 'read' and 'write' flags.

Permissions in the store are inherited from parent to child, but only when the child is created. This means that if you do

write('/tool/mytool/foo', 'Hi')
write('/tool/mytool/bar', 'Hello')
set_perms('/tool/mytool', { 'dom'   : 0,
                            'read'  : True,
                            'write' : True })

then foo and bar will not necessarily be read/write everybody! They will instead have the permissions of /tool/mytool at the time of the writes, so if /tool/mytool did not exist at the time, then foo and bar will be unreadable by anyone except dom0 and the domain that created them.

Toolstacks are careful to do

rm('/local/domain/<domid>')
mkdir('/local/domain/<domid>')
set_perms('/local/domain/<domid>', { 'dom' : <domid> })

for this very reason.