RHEL5 CentOS5 Xen Intel SR-IOV NIC Virtual Function VF PCI Passthru Tutorial

From Xen
Jump to navigationJump to search

Intel SR-IOV NIC Virtual Function (VF) PCI passthru with RHEL5/CentOS5 Xen

Requirements:

  • You need at least RHEL 5.9 / CentOS 5.9 for the dom0 system. Earlier EL5 versions have some bugs that prevent SR-IOV VF passthru from working properly. For example in EL 5.8 PCI passthru to HVM guests is broken, and in EL 5.7 VF passthru works only once, and fails on the second time. See the end of this wiki page for more information about the bugs and links to the various bugzilla entries.
  • You need a system with hardware IOMMU for PCI passthru (Intel VT-d). IOMMU needs to be supported by the CPU, chipset, BIOS/firmware and Xen.

Components used in this tutorial:

  • Dell R510 server, BIOS version: 1.10.2 (04/27/2012).
  • Intel Xeon CPU L5640.
  • Virtualization technology (Intel VT-x/VMX) enabled in BIOS.
  • IOMMU/VT-d enabled in BIOS.
  • Intel 82599EB 10 Gbit/sec dual-port SR-IOV Server NIC.
  • CentOS 5.9 x86_64 DVD1.
  • Stock Xen rpms from CentOS 5.9.
  • Stock kernel-xen from CentOS 5.9 as dom0 kernel.

Xen SR-IOV VF passthru to VMs

In this tutorial we'll configure el5 Xen SR-IOV Virtual Function (VF) interface passthru to the following VMs, using Intel 82599 10 Gbit/sec (ixgbe) NIC:

  • RHEL5 / CentOS5 x64 PV domU.
  • RHEL5 / CentOS5 x64 HVM guest.
  • RHEL6 / CentOS6 x64 HVM guest.

VMs will have direct access to the 10 Gbit/sec Virtual Function PCI device, offering high performance and low latency for network traffic.

EL5 Host installation and configuration

  • Install RHEL 5.9 or CentOS 5.9 x64 (64 bit) host using the "Server" profile from DVD1.
  • Both RHEL 5.9 and CentOS 5.9 have been verified to work identically, but in this tutorial we'll use CentOS because it's publicly available for free.
  • Use LVM disk configuration, and leave free space to the LVM volumegroup, so you can later create new LVM volumes and install VMs using LVM volumes as disks.
  • After installation disable SElinux in "/etc/selinux/config", change to "SELINUX=disabled".
  • Do all the usual network/IP, hostname, DNS etc configuration.
  • Make sure there is free unallocated space in the LVM volume group:
[root@dom0 ~]# vgdisplay
  --- Volume group ---
  VG Name               VolGroup00
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  38
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                27
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               1.82 TB
  PE Size               32.00 MB
  Total PE              59551
  Alloc PE / Size       55670 / 1.70 TB
  Free  PE / Size       3881 / 121.28 GB
  VG UUID               QuouSc-MXj2-Hhdo-Ut9k-RLLO-jk9N-RjQD6E
  • Note the "Free PE / Size" line on the vgdisplay command output above.
  • Update the system with "yum update".
  • Disable some of the extra services that are not needed in this tutorial:
chkconfig isdn off
chkconfig mcstrans off
chkconfig haldaemon off
chkconfig hidd off
chkconfig autofs off
chkconfig avahi-daemon off
chkconfig xfs off
chkconfig bluetooth off
chkconfig pcscd off
chkconfig iptables off
chkconfig ip6tables off
  • Install Xen and related packages:
yum install xen xen-libs kernel-xen libvirt virt-viewer python-virtinst xorg-x11-xauth
  • Edit "/etc/xen/xend-config.sxp" and disable (comment out) Xen network-script line:
#(network-script network-bridge)
  • We want to configure dom0 networking settings and bridges ourselves.
  • Modify "/boot/grub/grub.conf" and add all the usual options for Xen hypervisor:
default=0
timeout=10
splashimage=(hd0,0)/grub/splash.xpm.gz
#hiddenmenu
title CentOS (2.6.18-348.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-348.el5 dom0_mem=2048M loglvl=all
        module /vmlinuz-2.6.18-348.el5xen ro root=/dev/VolGroup00/LogVol00
        module /initrd-2.6.18-348.el5xen.img
  • Add "dom0_mem=2048M loglvl=all" options on the xen.gz line, adjust the dom0_mem as you wish/need.
  • Make sure the Xen entry is the default in Grub
  • Reboot the system to Xen.
  • When the system has booted up, verify Xen works OK:
[root@dom0 ~]# xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0     2048    12 r-----    788.0
  • Verify from the "xm list" output that dom0 is using the amount of memory you specified in Grub.
[root@dom0 ~]# xm info
host                   : dom0.localdomain
release                : 2.6.18-348.el5xen
version                : #1 SMP Tue Jan 8 18:35:04 EST 2013
machine                : x86_64
nr_cpus                : 12
nr_nodes               : 1
sockets_per_node       : 1
cores_per_socket       : 6
threads_per_core       : 2
cpu_mhz                : 2266
hw_caps                : bfebfbff:2c100800:00000000:00000940:029ee3ff:00000000:00000001
total_memory           : 49139
free_memory            : 46145
node_to_cpu            : node0:0-11
xen_major              : 3
xen_minor              : 1
xen_extra              : .2-348.el5
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 20080704 (Red Hat 4.1.2-54)
cc_compile_by          : mockbuild
cc_compile_domain      : (none)
cc_compile_date        : Tue Jan  8 17:45:10 EST 2013
xend_config_format     : 2
  • Versions of xen related rpms:
[root@dom0 ~]# rpm -qa | grep xen
xen-libs-3.0.3-142.el5
xen-libs-3.0.3-142.el5
kernel-xen-2.6.18-348.el5
xen-3.0.3-142.el5
  • Note that the actual Xen hypervisor is version 3.1.2 (+ a lot of patches from later Xen versions), only the userland tools are 3.0.3 based. Xen hypervisor is included in kernel-xen rpm in RHEL5/CentOS5.

Intel SR-IOV NIC

  • In this tutorial we're using the following Intel SR-IOV capable 10 Gbit/sec 82599 NIC:
[root@dom0 ~]# lspci | grep 10-Gig
03:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
  • which uses the Intel ixgbe driver, as can be seen from the "ethtool -i" output:
[root@dom0 ~]# ethtool -i eth2
driver: ixgbe
version: 3.4.8-k
firmware-version: 0.9-3
bus-info: 0000:03:00.0

[root@dom0 ~]# ethtool -i eth3
driver: ixgbe
version: 3.4.8-k
firmware-version: 0.9-3
bus-info: 0000:03:00.1

Enabling IOMMU support in Xen command line options

  • Edit "/boot/grub/grub.conf" and add "iommu=1" option for Xen, and "pci_pt_e820_access=on" option for dom0 Linux kernel (vmlinuz). Both options are required:
default=0
timeout=10
splashimage=(hd0,0)/grub/splash.xpm.gz
#hiddenmenu
title CentOS (2.6.18-348.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-348.el5 dom0_mem=2048M loglvl=all iommu=1
        module /vmlinuz-2.6.18-348.el5xen ro root=/dev/VolGroup00/LogVol00 pci_pt_e820_access=on
        module /initrd-2.6.18-348.el5xen.img

"pci_pt_e820_access=on" is EL5 kernel-xen specific option to enable MMCONF access method for the PCI configuration space, required for SR-IOV Virtual Function (VF) device support. See RHEL5 technical release notes for more information: https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/5/html/5.9_Technical_Notes/Known_Issues-kernel-xen.html .

Enable Intel SR-IOV NIC Virtual Functions (VFs) in the ixgbe driver options

  • Add the following line to end of "/etc/modprobe.conf":
options ixgbe max_vfs=8
  • 8 is just an example here, 82599 NIC supports up to 64 VFs.
  • Blacklist the Intel VF driver (ixgbevf) in dom0 so that dom0 kernel doesn't try to use the Virtual Functions (we want to PCI passthru them to Xen VMs!). Create "/etc/modprobe.d/blacklist-ixgbevf.conf" file with the following contents:
# intel ixgbe sr-iov vf (virtual function) driver
blacklist ixgbevf
  • Now reboot the system to activate the changes.
  • After reboot check that the IOMMU support gets enabled in Xen hypervisor from the Xen dmesg log:
[root@dom0 ~]# xm dmesg | grep -i vt-d | grep -i enable
(XEN) Intel VT-d has been enabled
(XEN) Intel VT-d snoop control enabled
(XEN) [VT-D]iommu.c:619: iommu_enable_translation: iommu->reg = ffff828bfff58000
  • Check for I/O virtualisation in Xen log:
[root@dom0 ~]# xm dmesg | grep "I/O virt"
(XEN) I/O virtualisation enabled
  • If VT-d IOMMU doesn't get enabled, read the full "xm dmesg | less" log for more information.
  • Also verify you can see the VFs in "lspci" output:
[root@dom0 ~]# lspci | grep "Virtual Function"
03:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  • Why 16 VFs when we configured max_vfs=8? There's 8 per NIC/port, so 16 total for 2 ports.
  • Also check the dom0 Linux kernel dmesg for the ixgbe driver logs:
[root@dom0 ~]# dmesg | grep ixgbe
ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.4.8-k
ixgbe: Copyright (c) 1999-2011 Intel Corporation.
ixgbe 0000:03:00.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
ixgbe 0000:03:00.0: (PCI Express:5.0GT/s:Width x8) 00:3b:31:77:9e:1c
ixgbe 0000:03:00.0: MAC: 2, PHY: 8, SFP+: 3, PBA No: E81283-002
ixgbe 0000:03:00.0: eth2: IOV is enabled with 8 VFs
ixgbe 0000:03:00.0: eth2: IOV: VF 0 is enabled MAC fe:71:c2:a8:4d:28
ixgbe 0000:03:00.0: eth2: IOV: VF 1 is enabled MAC a2:cd:8b:d7:a0:3e
ixgbe 0000:03:00.0: eth2: IOV: VF 2 is enabled MAC 2a:e4:d6:69:a2:9a
ixgbe 0000:03:00.0: eth2: IOV: VF 3 is enabled MAC 5a:56:e0:a4:7d:36
ixgbe 0000:03:00.0: eth2: IOV: VF 4 is enabled MAC ae:f5:bb:ff:f4:bc
ixgbe 0000:03:00.0: eth2: IOV: VF 5 is enabled MAC 2a:be:89:3e:53:3b
ixgbe 0000:03:00.0: eth2: IOV: VF 6 is enabled MAC 1a:61:ce:c8:a7:c0
ixgbe 0000:03:00.0: eth2: IOV: VF 7 is enabled MAC 96:d7:3f:c4:f2:aa
ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
ixgbe 0000:03:00.1: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
ixgbe 0000:03:00.1: (PCI Express:5.0GT/s:Width x8) 00:3b:31:77:9e:1d
ixgbe 0000:03:00.1: MAC: 2, PHY: 8, SFP+: 4, PBA No: E81283-002
ixgbe 0000:03:00.1: eth3: IOV is enabled with 8 VFs
ixgbe 0000:03:00.1: eth3: IOV: VF 0 is enabled MAC ee:b0:2b:65:c4:6c
ixgbe 0000:03:00.1: eth3: IOV: VF 1 is enabled MAC 0a:f0:de:c0:b1:4d
ixgbe 0000:03:00.1: eth3: IOV: VF 2 is enabled MAC 9a:ab:7f:4a:d6:ce
ixgbe 0000:03:00.1: eth3: IOV: VF 3 is enabled MAC ee:f3:59:15:1b:02
ixgbe 0000:03:00.1: eth3: IOV: VF 4 is enabled MAC be:cd:68:3c:bc:f2
ixgbe 0000:03:00.1: eth3: IOV: VF 5 is enabled MAC a2:66:eb:00:25:80
ixgbe 0000:03:00.1: eth3: IOV: VF 6 is enabled MAC 6e:3e:da:74:84:63
ixgbe 0000:03:00.1: eth3: IOV: VF 7 is enabled MAC de:06:3d:af:8c:ce
ixgbe 0000:03:00.1: Intel(R) 10 Gigabit Network Connection

Configuring Xen pciback for VF PCI passthru

  • First check available PCI devices for Xen PCI passthru:
[root@dom0 ~]# xm pci-list-assignable-devices
Error: pciback not loaded?
  • The error message makes sense, we haven't yet configured any PCI devices for passthru, and we haven't loaded the Xen pciback module in dom0 kernel.
  • Edit "/etc/modprobe.conf" and add options for the Xen pciback dom0 kernel module:
options pciback hide=(03:10.0)(03:10.1)(03:10.2)(03:10.3)(03:10.4)(03:10.5)(03:10.6)(03:10.7)(03:11.0)(03:11.1)(03:11.2)(03:11.3)(03:11.4)(03:11.5)(03:11.6)(03:11.7)
  • Here we are "hiding" all the Virtual Function PCI device IDs, so they can be used for Xen PCI passthru.
  • Now load the "pciback" module in dom0 kernel:
modprobe pciback
  • After loading "pciback" module check the dom0 Linux kernel dmesg:
pciback 0000:03:10.0: seizing device
pciback 0000:03:10.2: seizing device
pciback 0000:03:10.4: seizing device
pciback 0000:03:10.6: seizing device
pciback 0000:03:11.0: seizing device
pciback 0000:03:11.2: seizing device
pciback 0000:03:11.4: seizing device
pciback 0000:03:11.6: seizing device
pciback 0000:03:10.1: seizing device
pciback 0000:03:10.3: seizing device
pciback 0000:03:10.5: seizing device
pciback 0000:03:10.7: seizing device
pciback 0000:03:11.1: seizing device
pciback 0000:03:11.3: seizing device
pciback 0000:03:11.5: seizing device
pciback 0000:03:11.7: seizing device
PCI: Enabling device 0000:03:11.7 (0000 -> 0002)
PCI: Enabling device 0000:03:11.5 (0000 -> 0002)
PCI: Enabling device 0000:03:11.3 (0000 -> 0002)
PCI: Enabling device 0000:03:11.1 (0000 -> 0002)
PCI: Enabling device 0000:03:10.7 (0000 -> 0002)
PCI: Enabling device 0000:03:10.5 (0000 -> 0002)
PCI: Enabling device 0000:03:10.3 (0000 -> 0002)
PCI: Enabling device 0000:03:10.1 (0000 -> 0002)
PCI: Enabling device 0000:03:11.6 (0000 -> 0002)
PCI: Enabling device 0000:03:11.4 (0000 -> 0002)
PCI: Enabling device 0000:03:11.2 (0000 -> 0002)
PCI: Enabling device 0000:03:11.0 (0000 -> 0002)
PCI: Enabling device 0000:03:10.6 (0000 -> 0002)
PCI: Enabling device 0000:03:10.4 (0000 -> 0002)
PCI: Enabling device 0000:03:10.2 (0000 -> 0002)
PCI: Enabling device 0000:03:10.0 (0000 -> 0002)
  • Now check again what PCI devices are ready for Xen PCI passthru:
[root@dom0 ~]# xm pci-list-assignable-devices
0000:03:10.1
0000:03:10.0
0000:03:10.2
0000:03:10.3
0000:03:10.4
0000:03:10.5
0000:03:10.6
0000:03:10.7
0000:03:11.0
0000:03:11.1
0000:03:11.2
0000:03:11.3
0000:03:11.4
0000:03:11.5
0000:03:11.6
0000:03:11.7
  • Enable the SR-IOV NIC Physical Function (main PCI device) in dom0, otherwise all the VFs will be in "down" state and cannot be used in domUs/VMs!
[root@dom0 ~]# ifconfig eth2 up
[root@dom0 ~]# ifconfig eth3 up
  • Enable automatic loading of Xen pciback module in dom0 kernel after system reboot
[root@dom0 ~]# echo "modprobe pciback" > /etc/sysconfig/modules/xen-pciback.modules
[root@dom0 ~]# chmod +x /etc/sysconfig/modules/xen-pciback.modules


Xen SR-IOV VF PCI passthru to CentOS 5 x64 PV domU, guest installation

  • Create a new LVM volume for the new VM:
[root@dom0 ~]# lvcreate -L18G -nc59x64pv /dev/VolGroup00
  Logical volume "c59x64pv" created
  • Install a new CentOS 5.9 (or later) x64 PV domU, for example with virt-install. ssh with X11 forwarding enabled to dom0, so you can view the GUI installer (VNC client virt-viewer session) on your local desktop/X-server:
[root@dom0 ~]# virt-install --debug -b virbr0 -n c59x64pv -r 1024 --vcpus=2 -f /dev/VolGroup00/c59x64pv --vnc -p -l "http://ftp.funet.fi/pub/mirrors/centos.org/5.9/os/x86_64"
  • Replace "ftp.funet.fi" with your local CentOS mirror site.
  • Note that virt-install will attach the VM to "virbr0" ethernet bridge, which provides DHCP+NAT services, so the VM can connect to the Internet thru dom0.
  • Install the VM with minimal package selection profile, no need to choose any extra packages, GUI, or anything.
  • When the installation finishes, and after the VM is automatically rebooted, login to the console as root, and check the IP address of the VM. It'll probably be something in the 192.168.122.0/24 private subnet from the virbr0 dnsmasq setup. Use "ifconfig eth0" to verify the IP of the domU.
  • Verify you can ssh to the VM from dom0.
  • Run "lspci" in the VM to verify there are no PCI devices before we configure the Xen PCI passthru:
[root@c59x64pv ~]# lspci
[root@c59x64pv ~]#
  • Install CentOS updates:
[root@c59x64pv ~]# yum update
  • Reboot the VM to verify everything works after the updates are installed.
  • Again connect to the VM with ssh.
  • Check the kernel version:
[root@c59x64pv ~]# uname -a
Linux c59x64pv.localdomain 2.6.18-348.el5xen #1 SMP Tue Jan 8 18:35:04 EST 2013 x86_64 x86_64 x86_64 GNU/Linux
  • Shutdown the domU.

Xen SR-IOV VF PCI passthru to CentOS 5.9 x64 PV domU, guest configuration

  • RHEL / CentOS 5.8 and 5.9 both seem to work OK as domU. 5.7 and earlier won't work as domU. Dom0 is required to be at least 5.9 or later!
  • Check the output of "xm pci-list-assignable-devices" in dom0 and decide which VF you want to passthru to the VM.
  • In this example we'll use the PCI device "03:10.0", which is the first VF on the first NIC port.
  • Edit "/etc/xen/c59x64pv" configuration file for the VM and add a line to enable PCI passthru for the chosen VF:
pci = [ '03:10.0' ]
  • So the whole "/etc/xen/c59x64pv" domU cfgfile looks like:
name = "c59x64pv"
uuid = "ac23d84e-9807-8e3a-5a84-494aa881fd67"
maxmem = 1024
memory = 1024
vcpus = 2
bootloader = "/usr/bin/pygrub"
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
vfb = [ "type=vnc,vncunused=1,keymap=fi" ]
disk = [ "phy:/dev/VolGroup00/c58x64pv,xvda,w" ]
vif = [ "mac=00:16:3e:64:99:f3,bridge=virbr0,script=vif-bridge" ]
pci = [ '03:10.0' ]
  • Now start the VM:
[root@dom0 xen]# xm create -f /etc/xen/c59x64pv
Using config file "/etc/xen/c59x64pv".
Using <class 'grub.GrubConf.GrubConfigFile'> to parse /grub/menu.lst
Started domain c59x64pv
  • Connect to the VM with ssh (you checked the IP earlier).
  • Check the domU Linux kernel log with "dmesg" :
[root@c59x64pv ~]# dmesg|grep ixg
ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.1.0-k
  • Check the domU "lspci" output:
[root@c59x64pv ~]# lspci
00:00.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  • Check "ethtool -i" output in domU:
[root@c59x64pv ~]# ethtool -i eth1
driver: ixgbevf
version: 2.1.0-k
firmware-version: N/A
bus-info: 0000:00:00.0
  • Enable and configure eth1 VF interface:
[root@c59x64pv ~]# ifconfig eth1 <ip> netmask <netmask> up
  • Link status etc with ethtool:
[root@c59x64pv ~]# ethtool eth1
Settings for eth1:
        Supported ports: [ ]
        Supported link modes:   10000baseT/Full
        Supports auto-negotiation: No
        Advertised link modes:  Not reported
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: Unknown! (255)
        PHYAD: 0
        Transceiver: Unknown!
        Auto-negotiation: off
        Current message level: 0x00000007 (7)
        Link detected: yes
  • Check that interrupt (IRQ) counters are increasing for the VF:
[root@c59x64pv ~]# grep eth1 /proc/interrupts
253:         24          0        Phys-irq  eth1:mbx
254:         14          0        Phys-irq  eth1-tx-0
255:        661          0        Phys-irq  eth1-rx-0


  • Note that "eth0" will be the Xen PV NIC (using xennet driver), and "eth1" will the SR-IOV VF using Intel "ixgbevf" driver.
  • Configure the network/IP settings in "/etc/sysconfig/network-scripts/ifcfg-eth1" if you want them to persist across VM reboots.
  • Enjoy the 10 Gbit/sec VF in the Xen PV domU!

Xen SR-IOV VF PCI passthru to CentOS 5 x64 HVM guest, installation

  • Create a new LVM volume for the VM:
[root@dom0 ~]# lvcreate -L18G -nc59x64hvm /dev/VolGroup00
  Logical volume "c59x64hvm" created
  • Download CentOS 5.9 x64 DVD1 .iso from your local CentOS mirror site:
[root@dom0 ~]# mkdir ~/iso
[root@dom0 ~]# cd ~/iso
[root@dom0 iso]# wget ftp://ftp.funet.fi/pub/mirrors/centos.org/5.9/isos/x86_64/CentOS-5.9-x86_64-bin-DVD-1of2.iso
  • Create a Xen cfgfile for the new VM, edit "/etc/xen/c59x64hvm":
kernel = "/usr/lib/xen/boot/hvmloader"
builder='hvm'
device_model = '/usr/lib64/xen/bin/qemu-dm'
name = "c59x64hvm"
memory = 1024
shadow_memory = 8
vcpus=2
pae=1
acpi=1
apic=1
vif = [ 'mac=00:26:5f:14:18:19, bridge=virbr0, model=e1000' ]
disk = [ 'phy:/dev/VolGroup00/c59x64hvm,hda,w', 'file:/root/iso/CentOS-5.9-x86_64-bin-DVD-1of2.iso,hdc:cdrom,r' ]
boot='cd'
xen_platform_pci=1
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'
sdl=0
vnc=1
vncpasswd=
stdvga=0
serial='pty'
tsc_mode=0
usb=1
usbdevice='tablet'
keymap='fi'
  • Remember to edit and change the MAC address on the vif-line, every VM/vif needs to have a unique static MAC !
  • Start the VM and CentOS 5.9 installation:
[root@dom0 ~]# xm create -f /etc/xen/c59x64hvm
Using config file "/etc/xen/c59x64hvm".
Started domain c59x64hvm
  • Connect to the HVM guest console with the GUI VNC client:
[root@dom0 ~]# virt-viewer c59x64hvm
  • Install using the defaults.
  • Choose DHCP for eth0.
  • Set the system hostname manually to "c59x64hvm".
  • Deselect "Desktop - Gnome" package group, it's not needed in the VM.
  • When the installation is complete, and the installer reboots the VM, you need to launch virt-viewer again to reconnect to the VM console.
  • Login to the console as root.
  • Check the IP of eth0 with "ifconfig eth0". If there's no IP yet, enable the network with "ifup eth0" and then re-check the IP.
  • Connect to the VM with ssh from dom0.
  • Make sure basic tools are available:
[root@c59x64hvm ~]# yum install pciutils vim nano wget tcpdump screen ethtool
  • Install the CentOS updates with:
[root@c59x64hvm ~]# yum update
  • After the updates are installed reboot the VM.
  • Reconnect to the VM with ssh.
  • Verify the kernel version:
[root@c59x64hvm ~]# uname -a
Linux c59x64hvm.localdomain 2.6.18-348.el5 #1 SMP Tue Jan 8 17:53:53 EST 2013 x86_64 x86_64 x86_64 GNU/Linux
  • Verify the PCI devices in the VM:
[root@c59x64hvm ~]# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:01.3 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 SCSI storage controller: XenSource, Inc. Xen Platform Device (rev 01)
00:04.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)
  • Note that because this is an HVM (=fully virtualized) guest there are some Xen Qemu-dm emulated PCI devices visible in the lspci output.
  • Shutdown the VM.
[root@c59x64hvm ~]# poweroff


Xen SR-IOV VF PCI passthru to CentOS 5 x64 HVM guest, configuration

  • NOTE! There is a bug in RHEL 5.9 / CentOS 5.9 and earlier where kudzu+udev interaction might end up renaming the VF ethX interface to something like "__tmp254339888", which triggers a bug in the Intel ixgbevf driver, which crashes the guest kernel. This bug is fixed in a driver update in RHEL 5.10 / CentOS 5.10. The driver bug is still present in the el5.9 GA kernel (2.6.18-348.el5). For more information: https://bugzilla.redhat.com/show_bug.cgi?id=862862 .


  • Check the output of "xm pci-list-assignable-devices" in dom0 and decide which VF you want to passthru to the VM.
  • In this example we'll use the PCI device "03:10.1", which is the first VF on the second NIC port.
  • Edit "/etc/xen/c59x64hvm" configuration file for the VM and add a line to enable PCI passthru for the chosen VF:
pci = [ '03:10.1' ]
  • So the whole "/etc/xen/c59x64hvm" VM cfgfile looks like:
kernel = "/usr/lib/xen/boot/hvmloader"
builder='hvm'
device_model = '/usr/lib64/xen/bin/qemu-dm'
name = "c59x64hvm"
memory = 1024
shadow_memory = 8
vcpus=2
pae=1
acpi=1
apic=1
vif = [ 'mac=00:26:5f:14:18:19, bridge=virbr0, model=e1000' ]
disk = [ 'phy:/dev/VolGroup00/c59x64hvm,hda,w', 'file:/root/iso/CentOS-5.9-x86_64-bin-DVD-1of2.iso,hdc:cdrom,r' ]
boot='cd'
xen_platform_pci=1
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'
sdl=0
vnc=1
vncpasswd=
stdvga=0
serial='pty'
tsc_mode=0
usb=1
usbdevice='tablet'
keymap='fi'
pci = [ '03:10.1' ]
  • Start the VM:
[root@dom0 ~]# xm create -f /etc/xen/c59x64hvm
Using config file "/etc/xen/c59x64hvm".
Started domain c59x64hvm
  • Connect to the VM with ssh.
  • Check "lspci" output for the VF:
[root@c59x64hvm ~]# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:01.3 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 SCSI storage controller: XenSource, Inc. Xen Platform Device (rev 01)
00:04.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)
00:06.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  • Check kernel dmesg log for VF driver entries:
[root@c59x64hvm ~]# dmesg | grep ixgbevf
ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.1.0-k
ixgbevf 0000:00:06.0: setting latency timer to 64
  • Check "ethtool -i" output:
[root@c59x64hvm ~]# ethtool -i eth1
driver: ixgbevf
version: 2.1.0-k
firmware-version: N/A
bus-info: 0000:00:06.0
  • Check and verify "/etc/modprobe.conf"
[root@c59x64hvm ~]# cat /etc/modprobe.conf
alias eth0 e1000
alias scsi_hostadapter ata_piix
alias eth1 ixgbevf

The emulated e1000 NIC is eth0, so make sure modprobe.conf has "alias eth0 e1000". VF is eth1, so make sure modprobe.conf has "alias eth1 ixgbevf". If these aliases are the other way around (as kudzu sometimes writes them..) udev will start renaming the ethX interfaces at boot time, and that might trigger the ixgbevf driver bug mentioned earlier, which will crash the guest kernel!

  • Check and verify "ifcfg-eth0" file in "/etc/sysconfig/network-scripts" directory.
# cat /etc/sysconfig/network-scripts/ifcfg-eth0
# Intel Corporation 82540EM Gigabit Ethernet Controller
DEVICE=eth0
BOOTPROTO=dhcp
HWADDR=00:26:5F:14:18:19
ONBOOT=yes
DHCP_HOSTNAME=c59x64hvm.localdomain
  • Remove the VF MAC address from "ifcfg-eth1" in "/etc/sysconfig/network-scripts" directory because the VF MAC is random, and it'll change on every host reboot.
[root@c59x64hvm ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
# Intel Corporation 82599 Ethernet Controller Virtual Function
DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes


  • Enable and configure eth1 VF interface:
[root@c59x64hvm ~]# ifconfig eth1 <ip> netmask <netmask> up
  • Check link status etc with ethtool:
[root@c59x64hvm ~]# ethtool eth1
Settings for eth1:
        Supported ports: [ ]
        Supported link modes:   10000baseT/Full 
        Supports auto-negotiation: No
        Advertised link modes:  Not reported
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: Unknown! (255)
        PHYAD: 0
        Transceiver: Unknown!
        Auto-negotiation: off
        Current message level: 0x00000007 (7)
        Link detected: yes
  • Check that interrupt (IRQ) counters are increasing for the VF:
[root@c59x64hvm ~]# grep eth1 /proc/interrupts 
193:      37696        726       PCI-MSI-X  eth1-rx-0
201:        721          0       PCI-MSI-X  eth1-tx-0
209:         13          0       PCI-MSI-X  eth1:mbx


  • Enjoy the 10 Gbit/sec VF in the Xen HVM guest!


Xen SR-IOV VF PCI passthru to CentOS 6 PV domU

RHEL6 / CentOS6 uses Linux 2.6.32 based kernel, which includes Xen domU support via the upstream Linux "pvops" framework. xen-pcifront PCI frontend driver was added to upstream Linux 2.6.37, so it's not included in Linux 2.6.32, which means xen-pcifront is not in el6 kernel either. This means you can't do Xen PCI passthru to an el6 PV domU, because the el6 kernel lacks the required pci frontend driver, so it's not possible to use any PCI devices in the el6 PV domU.

Xen SR-IOV VF PCI passthru to CentOS 6 HVM guest, installation

  • Create a new LVM volume for the VM:
[root@dom0 ~]# lvcreate -L18G -nc63x64hvm /dev/VolGroup00
  Logical volume "c63x64hvm" created
  • Download CentOS 6.3 x64 DVD1 .iso from your local CentOS mirror site:
[root@dom0 ~]# mkdir ~/iso
[root@dom0 ~]# cd ~/iso
[root@dom0 iso]# wget ftp://ftp.funet.fi/pub/mirrors/centos.org/6.3/isos/x86_64/CentOS-6.3-x86_64-bin-DVD1.iso
  • Create a Xen cfgfile for the new VM, edit "/etc/xen/c63x64hvm":
kernel = "/usr/lib/xen/boot/hvmloader"
builder='hvm'
device_model = '/usr/lib64/xen/bin/qemu-dm'
name = "c63x64hvm"
memory = 1024
shadow_memory = 8
vcpus=2
pae=1
acpi=1
apic=1
vif = [ 'mac=00:26:5f:12:12:11, bridge=virbr0, model=e1000' ]
disk = [ 'phy:/dev/VolGroup00/c63x64hvm,hda,w', 'file:/root/iso/CentOS-6.3-x86_64-bin-DVD1.iso,hdc:cdrom,r' ]
boot='cd'
xen_platform_pci=1
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'
sdl=0
vnc=1
vncpasswd=
stdvga=0
serial='pty'
tsc_mode=0
usb=1
usbdevice='tablet'
keymap='fi'
  • Remember to edit and change the MAC address on the vif-line, every VM/vif needs to have a unique static MAC !
  • Start the VM and CentOS 6.3 installation:
[root@dom0 ~]# xm create -f /etc/xen/c63x64hvm
Using config file "/etc/xen/c63x64hvm".
Started domain c63x64hvm
  • Connect to the HVM guest console with the GUI VNC client:
[root@dom0 ~]# virt-viewer c63x64hvm
  • Install using the "minimal" package selection profile.
  • When the installation is complete, and the installer reboots the VM, you need to launch virt-viewer again to reconnect to the VM console.
  • Login to the console as root.
  • Check the IP of eth0 with "ifconfig eth0". If there's no IP yet, enable the network with "ifup eth0" and then re-check the IP.
  • Connect to the VM with ssh from dom0.
  • Install some required extra packages that are missing from the "minimal" installation:
[root@c63x64hvm ~]# yum install pciutils vim nano wget tcpdump
  • Edit "/etc/sysconfig/network-scripts/ifcfg-eth0" and change ONBOOT="yes" and NM_CONTROLLED="no". Now the network will be started automatically after VM reboot.
  • Also we need to make sure eth0 stays as eth0, and won't get renamed in the future, so check the HWADDR line in "/etc/sysconfig/network-scripts/ifcfg-eth0", and copy the MAC address.
  • Edit "/etc/udev/rules.d/70-persistent-net.rules" and add these lines, replacing ATTR{address} with your actual eth0 MAC address:
# eth0
SUBSYSTEM=="net", ACTION=="add", DRIVERS="?*", ATTR{address}="00:26:5f:12:12:11", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
  • Install the CentOS updates with:
[root@c63x64hvm ~]# yum update
  • After the updates are installed reboot the VM.
  • Reconnect to the VM with ssh.
  • Verify the kernel version:
[root@c63x64hvm ~]# uname -a
Linux c63x64hvm.localdomain 2.6.32-279.5.1.el6.x86_64 #1 SMP Tue Aug 14 23:54:45 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
  • Verify the PCI devices in the VM:
[root@c63x64hvm ~]# lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:01.3 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 SCSI storage controller: XenSource, Inc. Xen Platform Device (rev 01)
  • Note that because this is an HVM (=fully virtualized) guest there are some Xen Qemu-dm emulated PCI devices visible in the lspci output.
  • Shutdown the VM.
[root@c63x64hvm ~]# poweroff

Xen SR-IOV VF PCI passthru to CentOS 6 x64 HVM guest, configuration

  • NOTE! You need at least RHEL 6.4 / CentOS 6.4 in the VM, because there is a known bug in EL6.3 that makes it impossible to enable SR-IOV VF interfaces in an HVM guest. The bug is fixed in kernel-2.6.32-318.el6 and later versions. For more information: https://bugzilla.redhat.com/show_bug.cgi?id=849223 .


  • Check the output of "xm pci-list-assignable-devices" in dom0 and decide which VF you want to passthru to the VM.
  • In this example we'll use the PCI device "03:10.1", which is the first VF on the second NIC port.
  • Edit "/etc/xen/c63x64hvm" configuration file for the VM and add a line to enable PCI passthru for the chosen VF:
pci = [ '03:10.1' ]
  • So the whole "/etc/xen/c63x64hvm" VM cfgfile looks like:
kernel = "/usr/lib/xen/boot/hvmloader"
builder='hvm'
device_model = '/usr/lib64/xen/bin/qemu-dm'
name = "c63x64hvm"
memory = 1024
shadow_memory = 8
vcpus=2
pae=1
acpi=1
apic=1
vif = [ 'mac=00:26:5f:12:12:11, bridge=virbr0, model=e1000' ]
disk = [ 'phy:/dev/VolGroup00/c63x64hvm,hda,w', 'file:/root/iso/CentOS-6.3-x86_64-bin-DVD1.iso,hdc:cdrom,r' ]
boot='cd'
xen_platform_pci=1
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'
sdl=0
vnc=1
vncpasswd=
stdvga=0
serial='pty'
tsc_mode=0
usb=1
usbdevice='tablet'
keymap='fi'
pci = [ '03:10.1' ]
  • Start the VM:
[root@dom0 ~]# xm create -f /etc/xen/c63x64hvm
Using config file "/etc/xen/c63x64hvm".
Started domain c63x64hvm
  • Connect to the VM with ssh.
  • Check "lspci" output for the VF:

[root@c63x64hvm ~]# lspci 

00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:01.3 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 SCSI storage controller: XenSource, Inc. Xen Platform Device (rev 01)
00:06.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  • Check kernel dmesg log for VF driver entries:
[root@c63x64hvm ~]# dmesg | grep ixgbevf
ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.2.0-k
ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
ixgbevf 0000:00:06.0: setting latency timer to 64
ixgbevf 0000:00:06.0: irq 48 for MSI/MSI-X
ixgbevf 0000:00:06.0: irq 49 for MSI/MSI-X
ixgbevf 0000:00:06.0: irq 50 for MSI/MSI-X
  • Check "ethtool -i" output:
[root@c63x64hvm ~]# ethtool -i eth1
driver: ixgbevf
version: 2.2.0-k
firmware-version:
bus-info: 0000:00:06.0
  • Enable and configure eth1 VF interface:
[root@c63x64hvm ~]# ifconfig eth1 <ip> netmask <netmask> up
  • Check link status etc with ethtool:
[root@c63x64hvm ~]# ethtool eth1
Settings for eth1:
        Supported ports: [ ]
        Supported link modes:   10000baseT/Full
        Supports auto-negotiation: No
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: Other
        PHYAD: 0
        Transceiver: Unknown!
        Auto-negotiation: off
        Current message level: 0x00000007 (7)
        Link detected: yes
  • Enjoy the 10 Gbit/sec VF in the Xen HVM guest!

Passing thru multiple VFs to a single VM

  • PCI devices in dom0:
[root@dom0 ~]# lspci | grep 82599
03:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
03:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
03:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
  • The ordering of the Virtual Function PCI devices is the following:
03:10.0: first VF of the first PF (03:00.0).
03.10.1: first VF of the second PF (03:00.1).
03.10.2: second VF of the first PF (03:00.0).
03.10.3: second VF of the second PF (03:00.1).
03.10.4: third VF of the first PF (03:00.0).
03.10.5: thid VF of the second PF (03:00.1).
  • VF = Virtual Function.
  • PF = Physical Function (physical card/port).
  • To passthru multiple VFs to a single VM use the following syntax for pci=[]
pci = [ '03:10.0', '03:10.1', '03:10.2', '03:10.3' ]


VF MAC addresses

Intel "ixgbe" PF driver will randomize the VF MAC addresses, so on every host reboot the VF MACs will change! There's no way to change this behaviour in RHEL5 / CentOS5 dom0. In later versions of "iproute" you can control VF MAC addresses from dom0 and set/change static MACs for the VFs, but unfortunately that's not possible in the EL5 version of iproute.

Bugzilla entries

For reference here are some RHEL5 / RHEL6 Redhat bugzilla entries related to Xen SR-IOV VF passthru:

  • "PCI Virtual Function Passthrough - SR-IOV, Paravirt Guest fails to obtain IRQ after reboot / passthrough works only once":

https://bugzilla.redhat.com/show_bug.cgi?id=688673

  • "SR-IOV VF doesn't work in EL6.3 HVM guest / EL6 kernel bug":

https://bugzilla.redhat.com/show_bug.cgi?id=849223

  • "RHEL5 Xen dom0 qemu-dm: NIC unplug messes up MSI-X functionality of passed-through devices":

https://bugzilla.redhat.com/show_bug.cgi?id=861349

  • "Xen qemu-dm removes wrong iomem range when unplugging emulated NIC":

https://bugzilla.redhat.com/show_bug.cgi?id=861352

  • "Only 2 VF can be seen in RHEL5.9 PV guest":

https://bugzilla.redhat.com/show_bug.cgi?id=865736

  • "RHEL 5.8 HVM guest kernel crash when enabling SR-IOV VF PCI passthru NIC in the VM":

https://bugzilla.redhat.com/show_bug.cgi?id=862862

  • "RFE: rhel5 iproute: add support for SR-IOV VF operations / wontfix":

https://bugzilla.redhat.com/show_bug.cgi?id=894747

Retrieved from "https://wiki.xenproject.org/index.php?title=RHEL5_CentOS5_Xen_Intel_SR-IOV_NIC_Virtual_Function_VF_PCI_Passthru_Tutorial&oldid=6049"