XenDom0VLANstoDomUVirtualNICs
Introduction
|
To Do:
Describe XenDom0VLANstoDomUVirtualNICs here. |
Examples
These are very simple instructions for mapping VLANs on the Dom0 through to the DomU NICs. These are Redhat/CentOS specific, and in this example the NICs were mapped through to a HVM Windoze box.
Other descriptions to help being picked up by search engines is:
- VLAN trunk to virtual machine mapping
- VLANs to DomU interfaces
Basic Flow:
- Create a bridge group for each .1q interface you want, and add the interfaces to the bridge.
- Map each virtual NIC to the bridge group.
- Don't forget to set the MTU in Windoze
- Configure the bond interface, and create VLAN interfaces on the bond interface.
- Create a bridge and map each vlan interface into the bridge.
All the files for this are at /etc/sysconfig/network-scripts/. (This examples creates VLAN 900-903.) You will need to change the HWADDR to match you NIC cards.
These are all the related files:
/etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1 /etc/sysconfig/network-scripts/ifcfg-bond0 /etc/sysconfig/network-scripts/ifcfg-bond0.900 /etc/sysconfig/network-scripts/ifcfg-bond0.901 /etc/sysconfig/network-scripts/ifcfg-bond0.902 /etc/sysconfig/network-scripts/ifcfg-bond0.903 /etc/sysconfig/network-scripts/ifcfg-br900 /etc/sysconfig/network-scripts/ifcfg-br901 /etc/sysconfig/network-scripts/ifcfg-br902 /etc/sysconfig/network-scripts/ifcfg-br903
1.1 These files configure the real ethernet ports to be part of an active/standby 'bond' interface.
/etc/sysconfig/network-scripts/ifcfg-eth0
# Intel Corporation 82541GI Gigabit Ethernet Controller DEVICE=eth0 BOOTPROTO=none HWADDR=00:15:17:28:f6:34 ONBOOT=yes TYPE=Ethernet #ETHTOOL_OPTS="autoneg off speed 100 duplex full" MASTER=bond0 SLAVE=yes USERCTL=no
/etc/sysconfig/network-scripts/ifcfg-eth1
# Intel Corporation 82566DM-2 Gigabit Network Connection DEVICE=eth1 BOOTPROTO=none HWADDR=00:15:17:28:f6:36 ONBOOT=yes TYPE=Ethernet #ETHTOOL_OPTS="autoneg off speed 100 duplex full" MASTER=bond0 SLAVE=yes USERCTL=no
1.2 This is the bond interface configuration.
Here the bond0 interface, in Dom0, is actually getting a IP via dhcp. You may want to set BOOTPROTO=none if you don't want this. The 'virtual mac' address is set here. Make sure this is unique.
/etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0 BOOTPROTO=dhcp ONBOOT=yes USERCTL=no #PEERDNS=no IPV6INIT=no MACADDR=52:54:00:12:45:56 BONDING_OPTS="mode=1 miimon=100 primary=eth0"
1.3 Now create the bridges.
/etc/sysconfig/network-scripts/ifcfg-br900
DEVICE=br900 TYPE=Bridge #STP=on ONBOOT=yes BOOTPROTO=none
...
DEVICE=br903 TYPE=Bridge #STP=on ONBOOT=yes BOOTPROTO=none
1.4 Now the creation of all the VLAN interfaces, which are put into the bridges.
/etc/sysconfig/network-scripts/ifcfg-bond0.900
DEVICE=bond0.900 PHYSDEV=bond0 ONBOOT=yes BOOTPROTO=none VLAN=yes BRIDGE=br900
...
DEVICE=bond0.903 PHYSDEV=bond0 ONBOOT=yes BOOTPROTO=none VLAN=yes BRIDGE=br903
2. Load the bond-ing module.The main thing to add is below (rest of the bond is set in the file above). Reload or modprobe.
alias bond0 bonding
3. Finally, the important line in the Xen xm config file is:
vif = [ 'type=ioemu, bridge=br900', 'type=ioemu, bridge=br901', 'type=ioemu, bridge=br902', 'type=ioemu, bridge=br903' ]
4. Don't forget to set the MTU to 1494 using the 'regedit' if you are using windoze.
If you are using Centos/Redhat you can also use 'virsh' which has a XML virtual machine definition. 'virsh' seems to be strongly supported by Redhat, and XML is fairly clean, so instead of just using the 'xm' style files do this:
<domain type='xen'>
<name>vm_name</name>
<memory>256000</memory>
<bootloader>/usr/bin/pygrub</bootloader>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<vcpu>2</vcpu>
<devices>
<disk type='file' device='disk'>
<driver name='tap'/>
<source file='/home/xen/my_vm_centos52.img'/>
<target dev='xvda' bus='xen'/>
</disk>
<interface type='bridge'>
<source bridge='br900'/>
<mac address='00:16:2e:62:f0:1c'/>
<script path='vif-bridge'/>
</interface>
<interface type='bridge'>
<source bridge='br901'/>
<mac address='00:16:2e:62:f1:1c'/>
<script path='vif-bridge'/>
</interface>
<interface type='bridge'>
<source bridge='br902'/>
<mac address='00:16:2e:62:31:1c'/>
<script path='vif-bridge'/>
</interface>
<interface type='bridge'>
<source bridge='br903'/>
<mac address='00:16:2e:62:31:3c'/>
<script path='vif-bridge'/>
</interface>
</devices>
</domain>
Do 'virsh create vm_name.xml', 'virsh list', and 'virsh autostart vm_name'
regards, Dave Seddon
Here are some additional notes on roughly the same implementation using QEMU/KVM. The key here is again, setting up the correct bridge groups and linking the virtual NICs to these.
These are some KVM/QEMU instructions for giving virtual machines multiple NICs bound to multiple physical NICs. Eg.
Eth0 - Br0 - tap0 - Virtual NIC 0 Eth1 - Br1 - tap1 - Virtual NIC 1
Linux host configuration (debian/ubuntu)
This setups up the two bridge interfaces (no STP)
# The primary network interface
auto br0
iface br0 inet dhcp
bridge_stp off
bridge_ports eth0
iface eth0 inet static
address 0.0.0.0
netmask 0.0.0.0
auto eth0
iface eth1 inet static
address 0.0.0.0
netmask 0.0.0.0
auto eth1
auto br1
iface br1 inet static
address 0.0.0.0
netmask 0.0.0.0
bridge_stp off
bridge_ports eth1
To avoid KVM having to create the tap devices when it starts up (which required root), setup an init script to generate the interfaces with the correct ownership.
You will need to change 'das' to whatever username you are using to run KVM. Recommend starting this script after the /etc/init.d/network runs.
root@das-hp:/home/das/qemu# cat /etc/init.d/make_tap_devices.sh #!/bin/sh #make tap0 /usr/sbin/tunctl -u das #make tap1 /usr/sbin/tunctl -u das
KVM normally calls this script when creating tap interfaces:
(this script matches the bridge/switch on the default route - so usually
br0, for example)
root@das-hp:/etc/kvm# cat kvm-ifup
#!/bin/sh
switch=$(ip route ls | awk '/^default / { for(i=0;i<NF;i++) { if ($(i)
== "dev") print $(i+1) }}')
/sbin/ifconfig $1 0.0.0.0 up
/usr/sbin/brctl addif ${switch} $1
exit 0
However, we need a script that will match the first argument and change
this to the correct bridge/switch:
e.g. This script is called via kvm when creating a new tap interface
kvm-ifup-tap tap0 -> add to bridge br0
kvm-ifup-tap tap1 -> add to bridge br1
root@das-hp:/etc/kvm# cat kvm-ifup-tap
#!/bin/sh
#this script is called via kvm when creating a new tap interface
# $1 = tap0, or tap1, or whatever tap device
bridge=$(echo $1 | sed -e 's/tap//')
#bring the tap interface up
/sbin/ifconfig $1 0.0.0.0 up
#add the tap interface to the bridge
/usr/sbin/brctl addif br${bridge} $1
exit 0
Then finally the command to run all this is:
sudo /usr/bin/kvm -m 1100 \
-localtime \
-usbdevice tablet \
-net nic,vlan=0,model=e1000,macaddr=52:54:00:12:34:57 -net tap,vlan=0,ifname=tap0,script=/etc/kvm/kvm-ifup-tap \
-net nic,vlan=1,model=e1000,macaddr=52:54:00:12:34:58 -net tap,vlan=1,ifname=tap1,script=/etc/kvm/kvm-ifup-tap \
/home/das/qemu/winxp10g.img.qcow
Notes: 1. Make sure the 'vlan=X' the Xes match. This is how the virtual nic is associated with the tap interface. The NIC does not generate tagged frames. If the second NIC line above also had vlan=0, then all the NICs are effectively part of the same VLAN (and causes loops) 2. The above command is staring windoze, however it will work for other OSes too (e.g. Juniper olives) 3. The MAC addresses are hard coded to make sure DHCP address assignment is consistent) Alternatively, you can replace the existing kvm-ifup script (so you don't need to specify this to kvm):
mv kvm-ifup not.kvm-ifup
ln -s kvm-ifup-tap kvm-ifup
{{{
root@das-hp:/etc/kvm# ls -la
total 28
drwxr-xr-x 3 root root 4096 2009-04-16 11:20 .
drwxr-xr-x 174 root root 12288 2009-04-16 10:44 ..
lrwxrwxrwx 1 root root 12 2009-04-16 11:20 kvm-ifup -> kvm-ifup-tap
-rwxr-xr-x 1 root root 294 2009-04-16 11:19 kvm-ifup-tap
-rwxr-xr-x 1 root root 181 2008-04-23 04:35 not.kvm-ifup
drwxr-xr-x 2 root root 4096 2008-10-06 13:13 utils
KVM command becomes:
sudo /usr/bin/kvm -m 1100 \
-localtime \
-usbdevice tablet \
-net nic,vlan=0,model=e1000,macaddr=52:54:00:12:34:57 -net tap,vlan=0,ifname=tap0 \
-net nic,vlan=1,model=e1000,macaddr=52:54:00:12:34:58 -net tap,vlan=1,ifname=tap1 \
/home/das/qemu/winxp10g.img.qcow
Notes: 1. Make sure the 'vlan=X' the Xes match. This is how the virtual nic is associated with the tap interface. The NIC does not generate tagged frames. If the second NIC line above also had vlan=0, then all the NICs are effectively part of the same VLAN (and causes loops) 2. The above command is staring windoze, however it will work for other OSes too (e.g. Juniper olives) 3. The MAC addresses are hard coded to make sure DHCP address assignment is consistent)
Some other interesting output:
root@das-hp:/etc/kvm# ifconfig -a
br0 Link encap:Ethernet HWaddr 00:1a:4b:5a:b4:96
inet addr:XX.XX.XX.36 Bcast:XX.XX.XX.255 Mask:255.255.255.0
inet6 addr: fe80::21a:4bff:fe5a:b496/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6892185 errors:0 dropped:0 overruns:0 frame:0
TX packets:682 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:896362303 (854.8 MB) TX bytes:274918 (268.4 KB)
br1 Link encap:Ethernet HWaddr 00:ff:eb:19:39:9b
inet6 addr: fe80::1211:ff:fe00:477/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:576 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:85748 (83.7 KB) TX bytes:468 (468.0 B)
eth0 Link encap:Ethernet HWaddr 00:1a:4b:5a:b4:96
inet6 addr: fe80::21a:4bff:fe5a:b496/64 Scope:Link
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:3014 errors:0 dropped:0 overruns:0 frame:0
TX packets:936 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:422764 (412.8 KB) TX bytes:219484 (214.3 KB)
Interrupt:16
eth1 Link encap:Ethernet HWaddr 10:11:00:00:04:77
inet6 addr: fe80::1211:ff:fe00:477/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:291 errors:3464688 dropped:0 overruns:0 frame:0
TX packets:291 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:43297 (42.2 KB) TX bytes:47029 (45.9 KB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:16002 errors:0 dropped:0 overruns:0 frame:0
TX packets:16002 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:12053229 (11.4 MB) TX bytes:12053229 (11.4 MB)
tap0 Link encap:Ethernet HWaddr 00:ff:62:c1:3b:12
inet6 addr: fe80::2ff:62ff:fec1:3b12/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:174 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:25728 (25.1 KB) TX bytes:468 (468.0 B)
tap1 Link encap:Ethernet HWaddr 00:ff:eb:19:39:9b
inet6 addr: fe80::2ff:ebff:fe19:399b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:90 errors:0 dropped:0 overruns:0 frame:0
TX packets:90 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:13110 (12.8 KB) TX bytes:13086 (12.7 KB)
root@das-hp:/etc/kvm# brctl show
bridge name bridge id STP enabled interfaces
br0 8000.001a4b5ab496 no eth0
tap0
br1 8000.00ffeb19399b no eth1
tap1
