Archive/XCP DirectDiskAccess

The content of this page comes from http://techblog.conglomer.net/sata-direct-local-disk-access-on-xenserver/ and all credit goes to the original author.

There are several reasons why you may wish to attach a physical disk to your existing Xenserver installation, in our case we use it for both a permanent storage solution as well as moving large sets of data around (such as copying out virtual disks or data intensive backups). It’ll treat the SATA (or better yet eSATA) disk similarly to plugging in an external USB drive to the Xenserver host, but at much higher throughput.

One of our servers utilizes a 3 disk RAID5 array which is primarily used for VM and VM data storage. However we do have a NAS virtual machine that stores quite a lot of backup data; and rather then use up a bunch of space on our RAID array and subsequently have to transfer the data off again anyways, we prefer to use a single hard disk formatted in EXT3. This disk can be treated like any typical hard disk data store and is not Xenserver dependent. Once the drive reaches capacity, we can detach from the virtual machine, pull it out and swap in a new one. (And of course mount it on any Linux box and access the data directly).

In order to get Xenserver to recognize the SATA drive we’ll need to create some symbolic links in the hypervisor’s /dev/xapi/block directory by editing the /etc/udev/rules.d/50-udev.rules file. (The udev daemon in the kernel properly abstracts USB and SATA while placing the appropriate node into the /dev directory).

Using /dev/sdb as the drive we wish to attach, add the following lines to /etc/udev/rules.d/50-udev.rules which are modified from similar USB mass storage detection rules.

ACTION=="add", KERNEL=="sdb", SYMLINK+="xapi/block/%k", RUN+="/bin/sh -c '/opt/xensource/libexec/local-device-change %k 2>&1 >/dev/null&'"
ACTION=="remove", KERNEL=="sdb", RUN+="/bin/sh -c '/opt/xensource/libexec/local-device-change %k 2>&1 >/dev/null&'"

You can add additional lines for additional devices (such as sdc, sdd) if you wish.

Save and reboot Xenserver. Your attached SATA hard disk should now be recognized as a removable storage device, which can now be attached to a virtual machine via the XenCenter console. It’ll be using an emulated SCSI controller instead of USB, which means full read/write throughput! Don’t forget to attach/detach the drive via XenCenter to/from the VM of your choice – while it’s physically “hot swapable”, you can’t just yank the physical drive out of a running VM that’s using it.

This change will be persistent through reboot, but remember that any product updates to the core Xenserver installation will likely overwrite these modifications, please bookmark this page for future reference.

Troubleshooting

In CentOS this works fine, but there is something to be aware of. You must attach the storage after you start the VM, though only for the first attachment. If you do not, and attach the storage before you turn on the VM, it seems that the kernel will not release its hold on the storage and give it to the OS. When you try to do anything to the storage (write, mount, fsck, etc) it will respond that the storage is either mounted, busy or in use by the system. As long as you attach it after the boot process, you will avoid this, and on future reboots this problem will not occur. This has been confirmed in CentOS 5.8 and 6.3 and there's a good chance other Linux distros will have this issue as well.