NUMA node affinity in the Xen hypervisor: Difference between revisions

From Xen
Jump to navigationJump to search
mNo edit summary
mNo edit summary
Line 1: Line 1:
= Setting a Domain’s node affinity =
= Setting a Domain’s node affinity =

By default, on creation Xen domains have the Xen hypervisor feature flag `auto_node_affinity` enabled.

It automatically deduces and updates the domain's NUMA `node_affinity` to the vCPUs its vCPUs are affine to.

The buddy allocator of the Xen hypervisor uses the domains <code>node_affinity</code>
to select the NUMA node(s) it prefers to allocate from, unless:

* Its call was given a specific NUMA node to allocate from
* No page that would be usable to satisfy of the required page order was available when the allocator was called.


== Hypercall: <code>XEN_DOMCTL_setnodeaffinity</code> ==
== Hypercall: <code>XEN_DOMCTL_setnodeaffinity</code> ==

This hypercall disables the automatic affinity feature of Xen.



It can be called from <code>libxenctrl</code> and <code>libxl</code>
It can be called from <code>libxenctrl</code> and <code>libxl</code>
Line 32: Line 45:
On success, it disables the automatic affinity feature of Xen, where Xen tries to deduce the affinity of the domain from the affinty of its vCPUs for this domain and updates the domain’s <code>node_affinity</code> for memory allocations using the buddy allocator.
On success, it disables the automatic affinity feature of Xen, where Xen tries to deduce the affinity of the domain from the affinty of its vCPUs for this domain and updates the domain’s <code>node_affinity</code> for memory allocations using the buddy allocator.


It also calls into the Xen scheduler to notify it of the change. If the Xen scheduler does not have any vCPUs at that moment, the Xen scheduler notification does nothing.
It also calls into the Xen scheduler to notify it of the change.
If the Xen scheduler does not have any vCPUs at that moment, the Xen scheduler notification does nothing.


However, setting an explicit node_affinity for the domain changes the NUMA nodes to allocate from before the vCPU affinity is set, and it could alter the behavior of the scheduler in theory as well.
However, setting an explicit node_affinity for the domain changes the NUMA nodes to allocate from before the vCPU affinity is set, and it could alter the behavior of the scheduler in theory as well.

Revision as of 10:22, 11 February 2025

Setting a Domain’s node affinity

By default, on creation Xen domains have the Xen hypervisor feature flag `auto_node_affinity` enabled.

It automatically deduces and updates the domain's NUMA `node_affinity` to the vCPUs its vCPUs are affine to.

The buddy allocator of the Xen hypervisor uses the domains node_affinity to select the NUMA node(s) it prefers to allocate from, unless:

  • Its call was given a specific NUMA node to allocate from
  • No page that would be usable to satisfy of the required page order was available when the allocator was called.

Hypercall: XEN_DOMCTL_setnodeaffinity

This hypercall disables the automatic affinity feature of Xen.


It can be called from libxenctrl and libxl

libxenctrl: xc_domain_node_setaffinity()

This call requires the domid of the created domain and an xc_nodemap_t:

int xc_domain_node_setaffinity(xc_interface *xch,
                               uint32_t domid,
                               xc_nodemap_t nodemap)

Source: https://github.com/xen-project/xen/blob/master/tools/libs/ctrl/xc_domain.c#L122

Xen Hypervisor implementation

For setnodeaffinity, do_domctl() calls domain_set_node_affinity()

The entry point for al DOMCTL phypercalls is the function do_domctl():

When asked for XEN_DOMCTL_setnodeaffinity, it calls xenctl_bitmap_to_nodemask() to convert &op->u.nodeaffinity.nodemap to nodemask_t and then calls domain_set_node_affinity(d, &new_affinity); with the nodemask: https://github.com/xen-project/xen/blob/master/xen/common/domctl.c#L516

domain_set_node_affinity()

Starts at https://github.com/xen-project/xen/blob/master/xen/common/domain.c#L943

If the new nodemap does not intersect the node_online_map, it returns -EINVAL.

On success, it disables the automatic affinity feature of Xen, where Xen tries to deduce the affinity of the domain from the affinty of its vCPUs for this domain and updates the domain’s node_affinity for memory allocations using the buddy allocator.

It also calls into the Xen scheduler to notify it of the change. If the Xen scheduler does not have any vCPUs at that moment, the Xen scheduler notification does nothing.

However, setting an explicit node_affinity for the domain changes the NUMA nodes to allocate from before the vCPU affinity is set, and it could alter the behavior of the scheduler in theory as well.

While this call cannot influence the past, whereas domain_create() already created the domain and allocated the internal Xen data structures, the setting node_affinity early can change the course of memory allocations for the domain.

Used by