Xen ARM with Virtualization Extensions/Ibox3399

Bootloader

For ARM64 Xen requires to be booted in EL2 mode by the bootloader. By default Rockchip RK3399 Ibox3399 development board comes with u-boot which is not very friendly for booting Xen.

Please update the u-boot with provided by Rockchip rockchip-linux/u-boot github. Build instructions: rockchip-linux/u-boot github README.

Prepequairnments

u-boot-utils

Cross compilers used: gcc-linaro-7.2.1 for Xen and aarch64-linux-android-4.9 for kernel

Building Xen

Get Xen and switch to xen-4.10.1 tag:

 git clone git://xenbits.xen.org/xen.git
 cd xen
 git checkout RELEASE-4.10.1 -b 'xen-rk3399'

Enable earlyprintk configuration for rk3399 boards:

 diff --git a/xen/arch/arm/Rules.mk b/xen/arch/arm/Rules.mk
 index b66c19f..b1cf319 100644
 --- a/xen/arch/arm/Rules.mk
 +++ b/xen/arch/arm/Rules.mk
 @@ -38,6 +38,7 @@ EARLY_PRINTK_lager          := scif,0xe6e60000
  EARLY_PRINTK_midway         := pl011,0xfff36000
  EARLY_PRINTK_omap5432       := 8250,0x48020000,2
  EARLY_PRINTK_rcar3          := scif,0xe6e88000
 +EARLY_PRINTK_rk3399         := 8250,0xff1a0000,2
  EARLY_PRINTK_seattle        := pl011,0xe1010000
  EARLY_PRINTK_sun6i          := 8250,0x01c28000,2
  EARLY_PRINTK_sun7i          := 8250,0x01c28000,2

Configure Hypervisor:

 cd xen/xen
 export PATH=$PATH:<path_to_gcc-linaro>/bin/
 make menuconfig CROSS_COMPILE=aarch64-linux-gnu- XEN_TARGET_ARCH=arm64

Enable debugging option:

 Debugging Options  --->
   [*] Developer Checks

Build Hypervisor:

 cd ../
 make dist-xen CROSS_COMPILE=aarch64-linux-gnu- XEN_TARGET_ARCH=arm64 debug=y CONFIG_EARLY_PRINTK=rk3399

Make image suitable to be booted by u-boot:

 mkimage -A arm64 -T kernel -a 0x02000000 -e 0x02000000 -C none -d ./xen/xen xen4.10-uImage

Collect xen4.10-uImage file at this step.

Dom0 Kernel/FDT

 git clone https://github.com/ihormatushchak/k9tripod.git
 export PATH=$PATH:<path_to_aarch64_linux_android>/bin/
 cd k9tripod/

 make ARCH=arm64 x3399-xen-dom0_defconfig
 make -j4 CROSS_COMPILE=aarch64-linux-android- ARCH=arm64

After build rename file "Image" (located at arch/arm64/boot/) to "dom0-Image" and collect it at this step.

Collect x3399-dom0-development-board.dtb file (located at arch/arm64/boot/dts/rockchip/) at this step.

Dom0 rootfs

Let's use Buildroot to build rootfs for dom0.

 git clone git://git.buildroot.net/buildroot
 cd buildroot

At the moment version of buildroot was 3.0.0

 git checkout c83f2c14f8fde96bbdc1ee351d62f3b2346d20ff -b 'xen-domains-rootfs'

Configure buildroot before build:

 make menuconfig

Here is minimal configuration for dom0:

 Target options  --->
   Target Architecture --->
       (X) AArch64 (little endian)
   Target Architecture Variant --->
       (X) cortex-A72/A53 big.LITTLE
 
 Toolchain  --->
   Kernel Headers  --->
       (X) Linux 4.4.x kernel headers
   [*] Enable WCHAR support
 
 System configuration  --->
   Init system  --->
       (X) systemV
   /bin/sh  --->
       (X) bash
 
 Target packages  --->
   System tools  --->
       [*] xen
       [ ]   Xen hypervisor (NEW)
       [*]   Xen tools
 
 Filesystem images  --->
   [*] ext2/3/4 root filesystem
       ext2/3/4 variant  --->
           (X) ext2 (rev1)
   (300M) exact size
   [ ] tar the root filesystem

Now we can build the rootfs:

 make -j4

As a result we should get buildroot/output/images/rootfs.ext2 file here.

Preparing SD card to boot

Prepare SD card with next layout:

 +-------------+-------------+-------------+
 | partition 1 | partition 2 |             |
 +-------------+-------------+-------------+
 |             |             |             |
 | 100M (vfat) | 300M (ext2) | (Unmapped)  |
 |             |             |             |
 +-------------+-------------+-------------+
 |    boot     | dom0-roots  |             |
 +-------------+-------------+-------------+

Format partition 1 with vfat labeled with "boot":

 sudo mkfs.vfat /dev/sdX1 -n boot

NOTICE: replace /dev/sdX1 with your one.

Copy Xen and dom0 images to the boot partition (obtained in previous sections):

 mount /dev/sdX1 /mnt
 mkdir /mnt/boot
 cp xen4.10-uImage dom0-Image x3399-dom0-development-board.dtb /mnt/boot/
 umount /mnt

NOTICE: replace /dev/sdX1 with your one.

Copy the rootfs (obtained from buildroot):

 dd if=buildroot/output/images/rootfs.ext2 of=/dev/sdX2; sync

NOTICE: replace /dev/sdX2 with your one.

Now SD card ready to boot Xen with Dom0.

Booting

Plug in prepared SD card, power on the board and enter u-boot shell using next settings:

 uart2: 1500000 8N1

edit bootcmd var in u-boot with:

 fatload mmc 1:1 0x02000000 boot/xen4.10-uImage; fatload mmc 1:1 0x01f00000 boot/x3399-dom0-development-board.dtb; fatload mmc 1:1 0x03F80000 boot/dom0-Image;bootm 0x02000000 - 0x01f00000

Save changes

 saveenv 

and reboot the board.

Xen, dom0 should boot now.