Xen Project and OPW
The Xen Project Advisory Board will be sponsoring one intern for Round 7 of the Gnome Outreach Program For Women which runs from December 2013 to March 2014. This program is an internship program specifically targeted at women: our goal is to increase womens' participation in the Xen Project. It is a continuation of the very successful GNOME Outreach Program for Women and we are running the program in conjunction with GNOME and other prominent open source projects.
Information For Applicants
How To Apply
The official application period for OPW Round 7 is October 1st to November 11th. Please fill our your initial application and complete your Xen patch by November 11th. Applicants that do not complete the first patch will not be considered for an internship. Please take a look at our application FAQ for more info on how to fill our your initial application. Applicants will be notified by November 25th if they have been accepted.
If you are interested in being a Xen intern, please:
- Join the xen-devel mailing list. Depending on the project you choose, you may also need to join xen-api or cl-mirage (do check the information in the project). Check the list directory.
- Join the #opw IRC channel on irc.gnome.org
- Join the #xen-opw IRC channel on freenode.net
- Read our instructions for applying, and apply by Nov 11th.
- Ask one of our friendly developers on for a bite size bug or work item. This can be reviewing somebodies code, submitting a patch, or similar and will need to be done by Nov 11th.
Schedule
- October 1: program announced and application form made available
- October 1 - November 11: applicants need to get in touch with at least one project and make a contribution to it
- November 11: application deadline
- November 25: accepted participants announced
- December 10 - March 10: internship period
This section contains projects that have been reviewed by Xen Maintainers and Committers. Community members are free to add their own project ideas. We have unreviewed project ideas in the following places:
|
This section contains peer reviewed projects that have been selected based on the following criteria:
- A diverse list of projects, covering different level of difficulties and required skills
- Well written (in particular have a well written description)
- Contain steps, outcomes, skills required, ... all written down
- Are peer reviewed and debated
- Contain a diverse set of mentors
If your project did not make it into this list, it does not mean it will be excluded. It merely is not one of the projects that were ready for the program. Please add projects into Unreviewed Project Ideas.
|
Introducing PowerClamp-like driver for Xen
Date of insert: 01/22/2013; Verified:  Not specified, date when created; GSoC: Yes |
| Mentor: George Dunlap <george.dunlap@eu.citrix.com> |
| Difficulty: Medium to difficult |
| Skills Needed: C programming. A solid knowledge of how to use spinlocks to avoid race conditions and deadlock. Ability to read diffs, ability to use git and mercurial (hg). |
| Description: PowerClamp was introduced to Linux in late 2012 in order to allow users to set a system-wide maximum power usage limit. This is particularly useful for data centers, where there may be a need to reduce power consumption based on availability of electricity or cooling. A more complete writeup is available at LWN.
These same arguments apply to Xen. The purpose of this project would be to implement a similar functionality in Xen, and to make it interface as well as possible with the Linux PowerClamp tools, so that the same tools could be used for both.
The basic mechanism for PowerClamp in Linux is to monitor the percentage of time spent idling. When this time goes below a user-specified threshold, it activates a high-priority real-time process to force the CPU to idle for the specified amount of time. The intern would have to figure out how to apply this to Xen's main scheduler, the Credit scheduler.
The idea is fairly straightforward, but working with the scheduler involves dealing with very tricky race conditions and deadlock. This should be fairly straightforward project for a very clever intern looking for a fun problem to solve. It should also provide a good taste of what operating-system level programming is like. |
Outcomes:
- Mechanism to enforce a certain percentage of idle time in Xen
- An appropriate way to access this; either using Xen's xl command-line, or the PowerClamp tools for Linux (if any), or both.
|
| Steps: The first step would be to apply the idea to the main Xen scheduler, the Credit1 scheduler. What is the best way to implement this? Adding an extra priority level? Re-using the existing credit mechanism?
The second step would be to design an appropriate interface. Are there any PowerClamp userspace tools for Linux? Does it make sense to try to integrate those tools with Xen, or should we just have this be a separate Xen feature, accessible via Xen's xl command-line interface? |
| References: LWN Article on PowerClamp |
|
{{GSoC Project
|Project=Refactor Linux hotplug scripts
|Anchor=linux-hotplug-scripts
|Date=15/11/2012
|Contact=Roger Pau Monné <roger.pau@citrix.com>
|Difficulty=Medium
|Skills=Knowledge of C and good level of shell scripting
|Desc=
Current Linux hotplug scripts are all entangled, which makes them really difficult to understand or modify. The reason of hotplug scripts is to give end-users the chance to "easily" support different configuration for Xen devices.
Linux hotplug scripts should be analyzed, providing a good description of what each hotplug script is doing. After this, scripts should be cleaned, putting common pieces of code in shared files across all scripts. A Coding style should be applied to all of them when the refactoring is finished.
Also, a new hotplug implementation is currently under review [1], which will allow the user to create more complex hotplug scripts that offer extended functionality. Optionally the intern can implement support for other backends using the new hotplug interface (GlusterFS, Ceph...).
|Steps=The work on the project can be subdivided in the following phases:
- Phase 1: analyze hotplug scripts and determine what each script does internally in order to attach the device
- Phase 2: move common bits of code to shared files, providing a sane API
- Phase 3: refactor hotplug scripts to use this new API, and clean the code applying an uniform coding style
- Phase 4 [Optional]: create hotplug scripts for new backends (GlusterFS, Ceph)
|Outcomes=The candidate is expected to produce at least a series of patches, that contains the new internal hotplug API and the old scripts refactoring, send them to the Xen development mailing list and follow all the typical Open Source process for having them upstreamed in Xen.
|References=Source of current scripts
|Review=(delete as addressed)
Ijc 09:49, 11 February 2013 (UTC): Can we include a specific requirement to not just analyze but also document the behavior of the scripts, both the high-level semantics of each class of script (vif, block etc) but also the specifics of each (e.g. vif-{bridge,route,etc})? Ideally this would integrate with existing pages like Xen Networking. Should there also be a focus on customizability? I think it is expected that people will customize the scripts to suit there environments but due to the complexity a lot of folks don't. A refactoring project is not inherently that exiting, so I'm not sure how much it would appeal to interns, perhaps Phase 4 could be non-optional and require the creation of at least one new set of hotplug scripts to be created, as a kind of concrete end goal to all the refactoring? Not sure if that explodes the scope/time required out too far though. Ideally a new network script would be included to (to cover both main sets of bases) but we already cover most of the interesting cases there I think, openvswitch perhaps? I'm a little bit concerned that this project might also be chasing a moving target as the hotplug mechanism is refactored, but perhaps much of that will be finished by the time GSoC starts and having the person doing that refactoring also mentor the project should help minimise problems.
}}
XL to XCP VM motion
| Date of insert: 15/11/12; Verified: Not specified, date when created; GSoC: Yes |
| Mentor: Ian Campbell <ian.campbell@citrix.com> |
| Difficulty: Medium |
| Skills Needed: Knowledge of either C or oCaml (or both) or another suitable language. |
| Description: Currently xl (the toolstack supplied alongside Xen) and xapi (the XCP toolstack) have very different concepts about domain configuration, disk image storage etc. In the XCP model domain configuration is persistent and stored in a data base while under xl domain configuration is written in configuration files. Likewise disk images are stored as VDIs in Storage Repositories while under xl disk images are simply files or devices in the dom0 filesystem. For more information on xl see XL. For more information on XCP see XCP Overview.
This project is to produce one or more command-line tools which support migrating VMs between these toolstacks.
One tool should be provided which takes an xl configuration file and details of an XCP pool. Using the XenAPI XML/RPC interface It should create a VM in the pool with a close approximation of the same configuration and stream the configured disk image into a selected Storage Repository.
A second tool should be provided which performs the opposite operation, i.e. give a reference to a VM residing in an XCP pool it should produce an XL compatible configuration file and stream the disk image(s) our of Xapi into a suitable format.
These tools could be reasonably bundled as part of either toolstack and by implication could be written in either C, Ocaml or some other suitable language.
The tools should work on both PV and HVM domains. The subset of properties which are common to both toolstacks which are to be considered required for successful completion of the project will be determined early on in the project.
The tool need not operate on a live VM but that could be considered a stretch goal.
An acceptable alternative to the proposed implementation would be to implement a tool which converts between a commonly used VM container format which is supported by XCP (perhaps OVF or similar) and the xl toolstack configuration file and disk image formats. |
| Outcomes: Code submitted to xen-devel@ and/or xen-api@ for tools to migrate Virtual Machines between toolstacks. Must include documentation |
Steps: A suggested set of steps for completion of the project is:
- Setup both toolstacks and create suitable virtual machines on both
- Investigation of both toolstacks to determine what existing import/export functionality is present
- Documentation of mapping between Virtual Machine properties of both toolstacks
- Evaluate mechanisms for conversion, including disk image format conversion and which VM properties are required for success
- Documentation of how to perform a manual (i.e. by hand) conversion in either direction
- Implementation and documentation of tool(s) to automate the conversion process
- Post patches for review
- Iterate patches until acceptance
|
| References: XAPI, XL, XCP Overview |
|
VM Snapshots
| Date of insert: 16/01/2013; Verified: Not specified, date when created; GSoC: Yes |
| Mentor: Anthony Perard <anthony.perard@citrix.com> |
| Difficulty: Medium |
| Skills Needed: C programming |
| Description: Although xl is capable of saving and restoring a running VM, it is not currently possible to create a snapshot of the disk together with the rest of the VM.
QEMU is capable of creating, listing and deleting disk snapshots on QCOW2 and QED files, so even today, issuing the right commands via the QEMU monitor, it is possible to create disk snapshots of a running Xen VM. xl and libxl don't have any knowledge of these snapshots, don't know how to create, list or delete them.
This project is about implementing disk snapshots support in libxl, using the QMP protocol to issue commands to QEMU. Users should be able to manage the entire life-cycle of their disk snapshots via xl.
The candidate should also explore ways to integrate QEMU disk snapshots and disk mirroring into the regular Xen save/restore mechanisms and provide a solid implementation for xl/libxl. |
| Outcomes:
Basic goal: disk snapshots can be handled entirely by xl.
Stretch goals: xl can automatically save a disk snapshot at the time of saving a VM. xl can also mirror the disk of a VM between two hosts and can do that automatically at the time of VM migration. |
| Steps:
Basic steps:
- Study libxl APIs for storage
- Study QEMU QMP commands for VM snapshots
- Implement support for QMP snapshots commands in libxl
- Implement VM snapshots functionalities in libxl using the QMP functions previously written
- Add VM snapshot commands to XL
Stretch goals:
- Add VM snapshot functionalities to libxl save/restore and migration functions
- Evaluate QEMU QMP disk mirroring capabilities (QMP command "drive-mirror")
- Implement support for QMP drive-mirror command in libxl
- Hook disk mirroring into libxl VM save/restore and migration functions (migrating a VM from one host to another is also capable of migrating the VM disk from the two hosts).
|
| References: XL, QEMU |
(delete as addressed)
- Ijc 10:05, 11 February 2013 (UTC): Although this project is specifically targeting the QEMU snapshot mechanism we should require that the libxl API which is exposed is general enough to be applied to other disk backends (blktap3, lvm snapshot, btrfs, etc)
|
Fuzz testing Xen with Mirage
| Date of insert: 28/11/2012; Verified: Not specified, date when created; GSoC: Yes |
| Mentor: Anil Madhavapeddy <anil@recoil.org> - IMPORTANT also join the cl-mirage list |
| Difficulty: Medium |
| Skills Needed: OCaml programming. C programming. |
| Description: MirageOS (http://openmirage.org) is a type-safe exokernel written in OCaml which generates highly specialised "appliance" VMs that run directly on Xen without requiring an intervening guest kernel. We would like to use the Mirage/Xen libraries to fuzz test all levels of a typical cloud toolstack. Mirage has low-level bindings for Xen hypercalls, mid-level bindings for domain management, and high-level bindings to XCP for cluster management. This project would build a QuickCheck-style fuzzing mechanism that would perform millions of random operations against a real cluster, and identify bugs with useful backtraces. |
| Outcomes:
|
Steps:
- Set up a mirage build environment, test by building and running the examples (http://github.com/mirage/mirage-skeleton)
- Make a simple fuzzer with the existing hypercall bindings
- Extend the set of bindings, and extend the fuzzer to match
|
| References:
|
(delete as addressed)
- Ijc 10:05, 11 February 2013 (UTC): There are some interesting challenges which aren't mentioned here, specifically:
- reproducability of a given run leading to a crash
- how to handle guests which crash themselves while fuzzing, e.g. management of random seeds and respawning, measuring progress and perhaps snapshotting and restarting along multiple paths (so a single crash doesn't wipe out all the interesting state built up by the fuzzer up to that point)
- logging of what is going on in the face of hosts which may crash when the fuzzer "succeeds".
- Ijc 10:05, 11 February 2013 (UTC): It would also be useful to take inspiration from the trinity Linux system call fuzzer which encodes a certain level of knowledge of what the inputs to each system/hypercall should look like such that it can probe "interesting" (i.e. limits) values with more than random probability and also provide plausible input for some arguments so as to not continually mask errors in the other options (e.g. with some probability pass a valid socket to the int fd argument of a call which expects a socket, so that the other arguments have some chance of even being evaluated). Likewise for calls which take a pointer you would want to make sure the fuzzer would occasionally (or even mostly) pass in valid pointers such that the contents of the pointed to struct can also be fuzzed.
|
Towards a multi-language unikernel substrate for Xen
| Date of insert: 28/11/2012; Verified: Not specified, date when created; GSoC: Yes |
| Mentor: Anil Madhavapeddy <anil@recoil.org> - IMPORTANT also join the cl-mirage list |
| Difficulty: Difficult |
| Skills Needed: Familiarity with C and at least 1 other of OCaml,Haskell,Erlang and Java. |
| Description: There are several languages available that compile directly to Xen microkernels, instead of running under an intervening guest OS. We're dubbing such specialised binaries as "unikernels". Examples include:
Each of these is in a different state of reliability and usability. We would like to survey all of them, build some common representative benchmarks to evaluate them, and build a common toolchain that will make it easier to share code across such efforts. This project will require a reasonable grasp of several programming languages and runtimes, and should be an excellent project to learn more about the innards of popular languages. |
Outcomes:
- One document per runtime describing how it works
- A proposed common library which would allow all the runtimes to share the low-level routines
|
Steps:
- Set up a test and dev environment capable of building at least one of the above "unikernels" in the language you are most familiar with.
- Create a simple (less than 1000 words) description of how the code is structured. Include a description of how the VM starts, how the runtime starts and how the application runtime starts.
- Get the code for the other runtimes, and create one document for each of them, using the one you're familiar with as a reference.
- Create a C library and build environment containing code which could be used by all the runtimes.
|
| References: see above |
|
Testing PV and HVM installs of Debian using debian-installer
| Date of insert: 2013-01-23; Verified: Not specified, date when created; GSoC: Yes |
| Mentor: Ian Jackson <ian.jackson@eu.citrix.com> |
| Difficulty: Basic |
| Skills Needed: Knowledge of Perl, some familiarity with Debian preseeding |
| Description: The testing system "osstest" which is used for the push gate for the xen and related trees should have Debian PV and HVM guest installations, based on the standard Debian installer, in its repertoire. Also it currently always tests
kernels as host and guest in the same installation. |
| Outcomes: Code for guest installation in live osstest.git; public report on which Linux branches to deploy for; new test cases enabled in production. |
Steps:
- Task 1: Generalise the functions in osstest which generate debian-installer preseed files and manage the installation, to teach them how to set up PV and HVM guests, and provide an appropriate ts-* invocation script.
- Task 1b: Upstream this code into the production osstest.git.
- Task 2: Extend the guest installer from task 1 to be able to install a kernel other than the one which comes from the Debian repository, so that it is possible to test one kernel as host with a different specified kernel as guest.
- Task 2b: Upstream this code into the production osstest.git.
- Task 3: Determine which combinations of kernel branches should be added to the test schedules, push gates, etc. and write this up in a report for deployment by the infrastructure maintainers.
- Task 4: Assist with deployment and debugging after the new functionality is deployed in production in accordance with the report from Task 3.
|
| References: See xen-devel test reports (via the xen-devel list archives). Code is at http://xenbits.xen.org/gitweb/?p=osstest.git;a=summary
The introduction to Xen automatic test system is at http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/ |
(delete as addressed)
- Ijc 10:18, 11 February 2013 (UTC): I'd be happy to co-advise on the D-I aspects of this. In Task 2 "kernel other than the one which comes from the Debian repository", do you really mean "from the dom0 filesystem"? The D-I kernels do come from the Debian repo. Also is the intention to support testing guests which use pygrub, since that fits naturally with the D-I approach? Is the intention to only do netinst installs or is there scope to do D-I installs from ISO images too?
|
Testing NetBSD
| Date of insert: 2013-01-23; Verified: Not specified, date when created; GSoC: Yes |
| Mentor: Ian Jackson <ian.jackson@eu.citrix.com> |
| Difficulty: Basic to Medium |
| Skills Needed: Knowledge of Perl and NetBSD's installer |
| Description: The testing system "osstest" which is used for the push gate for the xen and related trees should be able to test NetBSD both as host and guest. |
| Outcomes: Code for host and guest installation in live osstest.git; public report on which combinations of tests to deploy for; testing of NetBSD enabled in production. |
Steps:
- Task 1: Understand how best to automate installation of NetBSD. Write code in osstest which is able to automatically and noninteractively install NetBSD on a bare host.
- Task 2: Test and debug osstest's automatic building arrangements so that they can correctly build Xen on NetBSD.
- Task 2b: Upstream this code into the production osstest.git.
- Task 3: Write code in osstest which can automatically install the Xen from task 2 on the system installed by task 1.
- Task 3b: Upstream this code into the production osstest.git.
- Task 4: Debug at least one of the guest installation capabilities in osstest so that it works on the Xen system from task 3.
- Task 5: Rework the code from task 1 so that it can also install a NetBSD guest, ideally either as a guest of a Linux dom0 or of a NetBSD dom0.
- Task 5b: Upstream this code into the production oosstest.git.
- Task 6: Determine which versions of NetBSD and of Linux should be tested in which combinations and write this up in a report for deployment by the infrastructure maintainers.
- Task 7: Assist with deployment and debugging after the new functionality is deployed in production in accordance with the report from Task 6.
|
| References: See xen-devel test reports (via the xen-devel list archives). Code is at http://xenbits.xen.org/gitweb/?p=osstest.git;a=summary
The introduction to Xen automatic test system is at http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/ |
|
Useful Resources
Here is some links to guides, tools, development flows etc.
Information For Mentors
See here
