Improved NUMA memory allocation: Difference between revisions

From Xen
Jump to navigationJump to search
m (Add link to doc describing meminit_hvm() and vmemranges for NUMA)
m (NODE affinity)
 
Line 9: Line 9:
| content1 = __TOC__
| content1 = __TOC__
}}
}}

NUMA affine memory allocation (NAMA) aims to be similar in spirite to
= Areas of interest =
[https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/virtualization_tuning_and_optimization_guide/sect-virtualization_tuning_optimization_guide-numa-allocation_policy#sect-Virtualization_Tuning_Optimization_Guide-NUMA-Allocation_Policy| this NUMA allocation policy in RHEL]:

<pre>
== NUMA node affinity of Xen domains ==
<numatune>

<memory mode='preferred' nodeset='0'>
Xen has the DOMCTL hypercall XEN_DOMCTL_setnodeaffinity to configure the NUMA node affinity of Xen domains for allocating memory.
</numatune>

</pre>
Main article: [[NUMA node affinity in the Xen hypervisor]]

= Issues =


While Xen is in has some NUMA-awareness, there are potential issues where memory allocation might not be ideal for NUMA.
While Xen is in has some NUMA-awareness, there are potential issues where memory allocation might not be ideal for NUMA.
Line 32: Line 35:


The current plan of [[User:Bernhard.Kaindl|Bernhard Kaindl]] is to investigate further on what would be needed use the vmemranges described in the page about meminit_hvm() code described below https://github.com/xenserver-next/xen/blob/xenguest-numa/docs/designs/numa/memory_alloc.md to allocate memory from (a) specific NUMA node when XAPI creates a domain VM using xenguest.
The current plan of [[User:Bernhard.Kaindl|Bernhard Kaindl]] is to investigate further on what would be needed use the vmemranges described in the page about meminit_hvm() code described below https://github.com/xenserver-next/xen/blob/xenguest-numa/docs/designs/numa/memory_alloc.md to allocate memory from (a) specific NUMA node when XAPI creates a domain VM using xenguest.

== See also ==
- The [https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/virtualization_tuning_and_optimization_guide/sect-virtualization_tuning_optimization_guide-numa-allocation_policy#sect-Virtualization_Tuning_Optimization_Guide-NUMA-Allocation_Policy|NUMA allocation policy of RHEL].


[[Category:NUMA]]
[[Category:NUMA]]

Latest revision as of 10:06, 11 February 2025

Areas of interest

NUMA node affinity of Xen domains

Xen has the DOMCTL hypercall XEN_DOMCTL_setnodeaffinity to configure the NUMA node affinity of Xen domains for allocating memory.

Main article: NUMA node affinity in the Xen hypervisor

Issues

While Xen is in has some NUMA-awareness, there are potential issues where memory allocation might not be ideal for NUMA.

Parallel boot issues

Booting multiple VMs in parallel will result in potentially allocating both on the same NUMA node:

This race condition can happen when multiple VMs are booted in parallel, because the XAPI toolstack does not wait until Xen has constructed each domain before constructing the next VM. That would serialize domain construction, which is currently parallel.

The current understanding of this parallel boot race is that, while creating the p2m (this is before the domain starts running) you could run into a problem without reservations, where during the parallel contruction boot of two domains, the toolstack may think they have individually enough free memory before the VMs are actually fully booted during which the VMs start to populate their p2m, but collectively when theil p2m is fully populated, they don't fit onto the same NUMA node and Xen has to populate the p2m of part of these VMs from other NUMA nodes.

Documentation on now how `meminit_hvm()` allocates memory on domain creation is here: https://github.com/xenserver-next/xen/blob/xenguest-numa/docs/designs/numa/x86/meminit_hvm.md

The current plan of Bernhard Kaindl is to investigate further on what would be needed use the vmemranges described in the page about meminit_hvm() code described below https://github.com/xenserver-next/xen/blob/xenguest-numa/docs/designs/numa/memory_alloc.md to allocate memory from (a) specific NUMA node when XAPI creates a domain VM using xenguest.

See also

- The allocation policy of RHEL.