NUMA node affinity in the Xen hypervisor: Difference between revisions
From Xen
Jump to navigationJump to search
mNo edit summary |
mNo edit summary |
||
(One intermediate revision by the same user not shown) | |||
Line 1: | Line 1: | ||
= Setting a Domain’s node affinity = |
= Setting a Domain’s node affinity = |
||
By default, on creation Xen domains have the Xen hypervisor feature flag `auto_node_affinity` enabled. |
|||
== Hypercall: <code>XEN_DOMCTL_setnodeaffinity</code> == |
|||
It automatically deduces and updates the domain's NUMA `node_affinity` to the vCPUs its vCPUs are affine to. |
|||
It can be called from <code>libxenctrl</code> and <code>libxl</code> |
|||
The buddy allocator of the Xen hypervisor uses the domains <code>node_affinity</code> |
|||
=== <code>libxenctrl</code>: <code>xc_domain_node_setaffinity()</code> === |
|||
to select the NUMA node(s) it prefers to allocate from, unless it was passed a specific NUMA node to allocate from. |
|||
For more details, see: |
|||
This call requires the <code>domid</code> of the created domain and an <code>xc_nodemap_t</code>: |
|||
https://xapi-project.github.io/new-docs/lib/xenctrl/xc_domain_node_setaffinity/ |
|||
<syntaxhighlight lang="c"> |
|||
int xc_domain_node_setaffinity(xc_interface *xch, |
|||
uint32_t domid, |
|||
xc_nodemap_t nodemap) |
|||
</syntaxhighlight> |
|||
Source: https://github.com/xen-project/xen/blob/master/tools/libs/ctrl/xc_domain.c#L122 |
|||
=== Xen Hypervisor implementation === |
|||
==== For <code>setnodeaffinity</code>, <code>do_domctl()</code> calls <code>domain_set_node_affinity()</code> ==== |
|||
The entry point for al <code>DOMCTL</code> phypercalls is the function <code>do_domctl()</code>: |
|||
When asked for XEN_DOMCTL_setnodeaffinity, it calls <code>xenctl_bitmap_to_nodemask()</code> to convert <code>&op->u.nodeaffinity.nodemap</code> to <code>nodemask_t</code> and then calls <code>domain_set_node_affinity(d, &new_affinity);</code> with the nodemask: https://github.com/xen-project/xen/blob/master/xen/common/domctl.c#L516 |
|||
==== <code>domain_set_node_affinity()</code> ==== |
|||
Starts at https://github.com/xen-project/xen/blob/master/xen/common/domain.c#L943 |
|||
If the new nodemap does not intersect the <code>node_online_map</code>, it returns <code>-EINVAL</code>. |
|||
On success, it disables the automatic affinity feature of Xen, where Xen tries to deduce the affinity of the domain from the affinty of its vCPUs for this domain and updates the domain’s <code>node_affinity</code> for memory allocations using the buddy allocator. |
|||
It also calls into the Xen scheduler to notify it of the change. If the Xen scheduler does not have any vCPUs at that moment, the Xen scheduler notification does nothing. |
|||
However, setting an explicit node_affinity for the domain changes the NUMA nodes to allocate from before the vCPU affinity is set, and it could alter the behavior of the scheduler in theory as well. |
|||
While this call cannot influence the past, whereas domain_create() already created the domain and allocated the internal Xen data structures, the setting node_affinity early can change the course of memory allocations for the domain. |
|||
= Used by = |
|||
== See also == |
|||
* [[Improved NUMA memory allocation]] |
* [[Improved NUMA memory allocation]] |
||
** [[NUMA node-specifc memory allocation]] |
** [[NUMA node-specifc memory allocation]] |
Latest revision as of 02:09, 14 February 2025
Setting a Domain’s node affinity
By default, on creation Xen domains have the Xen hypervisor feature flag `auto_node_affinity` enabled.
It automatically deduces and updates the domain's NUMA `node_affinity` to the vCPUs its vCPUs are affine to.
The buddy allocator of the Xen hypervisor uses the domains node_affinity
to select the NUMA node(s) it prefers to allocate from, unless it was passed a specific NUMA node to allocate from.
For more details, see:
https://xapi-project.github.io/new-docs/lib/xenctrl/xc_domain_node_setaffinity/