NUMA node-specifc memory allocation
NUMA node-specific memory allocation
Entry point: xenguest --mode hvm_build
xenguest --mode hvm_build
:
It calls
do_hvm_build()
,
which calls
stub_xc_hvm_build()
.
stub_xc_hvm_build()
It starts the HVM/PVH domain creation by filling out the fields of struct flags
and struct xc_dom_image
and calls hvm_build_setup_mem()
.
hvm_build_setup_mem()
- Gets
struct xc_dom_image *dom
,max_mem_mib
, andmax_start_mib
. - Calculates start and size of most parts of the domain’s memory maps
- taking memory holes for I/O into account, e.g.
mmio_size
andmmio_start
.
- taking memory holes for I/O into account, e.g.
- It then uses those to calculate
lowmem_end
andhighmem_end
. - Finally, calls
xc_dom_boot_mem_init()
.
xc_dom_boot_mem_init()
In all cases, xc_dom_boot_mem_init()
is called.
It calls the architecture-specific meminit
hook for the domain type:
rc = dom->arch_hooks->meminit(dom);
meminit_hvm()
, the x86 HVM meminit() to allocate HVM domain memory.
It is the meminit
hook for x86 HVM domains:
https://github.com/xen-project/xen/blob/master/tools/libs/guest/xg_dom_x86.c#L1348
Dynamic memory / Populate on Demand
When dom->target_pages
is smaller than dom->total_pages
,
the X86 meminit_hvm()
function enables XENMEMF_populate_on_demand
:
https://github.com/xen-project/xen/blob/master/tools/libs/guest/xg_dom_x86.c#L1368
In this case, meminit_hvm()
function calls xc_domain_set_pod_target()
to set the populate on demand target to dom->target_pages
:
https://github.com/xen-project/xen/blob/master/tools/libs/guest/xg_dom_x86.c#L1454
meminit_hvm
: vmemranges
The allocation of domain boot memory is defined using vmemranges
.
meminit_hvm()
creates a default vmemranges
array based on the amount of
lowmem
(below 4G, until dom->lowmem_end
) and
- highmem (above 4G, until
dom->highmem_end
).
The default vmemranges
do not carry information about from which NUMA node the memory shall be allocated.
meminit_hvm()
uses two vmemranges by default:
0
to dom->lowmem_end
4G
to dom->highmem_end
The NUMA node IDs (nid
) are set to 0:
- A dummy
vnode_to_pnode[]
for nid
maps 0
to XC_NUMA_NO_NODE
.
Code: https://github.com/xen-project/xen/blob/master/tools/libs/guest/xg_dom_x86.c#L1371
Alternatively, callers of the meminit
hook have the possibility to pass
an array of memory ranges in dom->vmemranges
,
which can contain a specific NUMA node for each vmemrange
andthe "exact
" flag:
While vmemranges
have been added for vNUMA, meminit_hvm()
does not differentiate on that.
It is only concerned about memory init.
Hence, its code always acts on vmemranges
to avoid code duplications
and creates default non-NUMA vmemranges
.
If the caller does not pass specific dom->vmemranges
for NUMA cases, the default vmemranges
are used.
Caveat: When passing dom->vmemranges
, Populate-On-Demand (target_pages
< total_pages
) cannot be used:
With custom vmemranges
as Populate-On-Demand does not support NUMA.
As a result, memory overcommitment with ballooning is currently impossible for NUMA node-specific memory allocations.
Memory allocation of the Domain’s memory
meminit_hvm()
attempts to allocate 1GB pages if possible, falls back on 2MB pages if 1GB allocation fails,
and 4KB pages will be used eventually if both fail:
https://github.com/xen-project/xen/blob/master/tools/libs/guest/xg_dom_x86.c#L1475
For each vmemrange
, new_memflags
are composed, based on the base memflags
:
unsigned int new_memflags = memflags;
unsigned int vnode = vmemranges[vmemid].nid;
// With custom vmemranges, vnode_to_pnode is custom too:
unsigned int pnode = vnode_to_pnode[vnode]
if ( pnode != XC_NUMA_NO_NODE ) // vnode maps to a physical NUMA node:
// XENMEMF_exact_node: (XENMEMF_node(n) | XENMEMF_exact_node_request)
new_memflags |= XENMEMF_exact_node(pnode);
Thus, to allocate memory in a specific NUMA node, this is needed:
- Populate-On-Demand (POD) must not be used (
target_pages == total_pages
)
- The sum of the
vmemranges
equals dom->total_pages
dom->vmemranges
must be passed with a nid
dom->vnode_to_pnode
must be passed, the nid
must map to the pNUMA node to allocate on.
dom->nr_vmemranges
and dom->nr_vnodes
must be set accordingly
For the allocation of each vmemrange
(extent), a call to xc_domain_populate_physmap()
is used.
It calls the XENMEM_populate_physmap
hypercall for each group of extents to allocate.
See populate_physmap.md for its implementation.
Other callers of the meminit()
calls
libxl
calls xc_dom_boot_mem_init()
using libxl__build_dom()
from init-xenstore-domain.c/build().