NUMA node affinity in the Xen hypervisor: Difference between revisions

From Xen
Jump to navigationJump to search
mNo edit summary
mNo edit summary
 
Line 6: Line 6:


The buddy allocator of the Xen hypervisor uses the domains <code>node_affinity</code>
The buddy allocator of the Xen hypervisor uses the domains <code>node_affinity</code>
to select the NUMA node(s) it prefers to allocate from, unless:
to select the NUMA node(s) it prefers to allocate from, unless it was passed a specific NUMA node to allocate from.


For more details, see:
* Its call was given a specific NUMA node to allocate from
* No page that would be usable to satisfy of the required page order was available when the allocator was called.


https://xapi-project.github.io/new-docs/lib/xenctrl/xc_domain_node_setaffinity/
== Hypercall: <code>XEN_DOMCTL_setnodeaffinity</code> ==

This hypercall disables the automatic affinity feature of Xen.


It can be called from <code>libxenctrl</code> and <code>libxl</code>

=== <code>libxenctrl</code>: <code>xc_domain_node_setaffinity()</code> ===

This call requires the <code>domid</code> of the created domain and an <code>xc_nodemap_t</code>:

<syntaxhighlight lang="c">
int xc_domain_node_setaffinity(xc_interface *xch,
uint32_t domid,
xc_nodemap_t nodemap)
</syntaxhighlight>
Source: https://github.com/xen-project/xen/blob/master/tools/libs/ctrl/xc_domain.c#L122

=== Xen Hypervisor implementation ===

==== For <code>setnodeaffinity</code>, <code>do_domctl()</code> calls <code>domain_set_node_affinity()</code> ====

The entry point for al <code>DOMCTL</code> phypercalls is the function <code>do_domctl()</code>:

When asked for XEN_DOMCTL_setnodeaffinity, it calls <code>xenctl_bitmap_to_nodemask()</code> to convert <code>&amp;op-&gt;u.nodeaffinity.nodemap</code> to <code>nodemask_t</code> and then calls <code>domain_set_node_affinity(d, &amp;new_affinity);</code> with the nodemask: https://github.com/xen-project/xen/blob/master/xen/common/domctl.c#L516

==== <code>domain_set_node_affinity()</code> ====

Starts at https://github.com/xen-project/xen/blob/master/xen/common/domain.c#L943

If the new nodemap does not intersect the <code>node_online_map</code>, it returns <code>-EINVAL</code>.

On success, it disables the automatic affinity feature of Xen, where Xen tries to deduce the affinity of the domain from the affinty of its vCPUs for this domain and updates the domain’s <code>node_affinity</code> for memory allocations using the buddy allocator.

It also calls into the Xen scheduler to notify it of the change.
If the Xen scheduler does not have any vCPUs at that moment, the Xen scheduler notification does nothing.

However, setting an explicit node_affinity for the domain changes the NUMA nodes to allocate from before the vCPU affinity is set, and it could alter the behavior of the scheduler in theory as well.

While this call cannot influence the past, whereas domain_create() already created the domain and allocated the internal Xen data structures, the setting node_affinity early can change the course of memory allocations for the domain.

= Used by =


== See also ==
* [[Improved NUMA memory allocation]]
* [[Improved NUMA memory allocation]]
** [[NUMA node-specifc memory allocation]]
** [[NUMA node-specifc memory allocation]]

Latest revision as of 02:09, 14 February 2025

Setting a Domain’s node affinity

By default, on creation Xen domains have the Xen hypervisor feature flag `auto_node_affinity` enabled.

It automatically deduces and updates the domain's NUMA `node_affinity` to the vCPUs its vCPUs are affine to.

The buddy allocator of the Xen hypervisor uses the domains node_affinity to select the NUMA node(s) it prefers to allocate from, unless it was passed a specific NUMA node to allocate from.

For more details, see:

https://xapi-project.github.io/new-docs/lib/xenctrl/xc_domain_node_setaffinity/

See also