NUMA node-specifc memory allocation

From Xen
Revision as of 15:30, 30 January 2025 by Bernhard.Kaindl (talk | contribs)
Jump to navigationJump to search

NUMA node-specific memory allocation

Entry point: xenguest --mode hvm_build

xenguest --mode hvm_build: It calls do_hvm_build(), which calls stub_xc_hvm_build().

stub_xc_hvm_build()

It starts the HVM/PVH domain creation by filling out the fields of struct flags and struct xc_dom_image and calls hvm_build_setup_mem().

hvm_build_setup_mem()

  • Gets struct xc_dom_image *dom, max_mem_mib, and max_start_mib.
  • Calculates start and size of most parts of the domain’s memory maps
    • taking memory holes for I/O into account, e.g. mmio_size and mmio_start.
  • It then uses those to calculate lowmem_end and highmem_end.
  • Finally, calls xc_dom_boot_mem_init().

xc_dom_boot_mem_init()

In all cases, xc_dom_boot_mem_init() is called.

It calls the architecture-specific meminit hook for the domain type:

rc = dom->arch_hooks->meminit(dom);

meminit_hvm(), the x86 HVM meminit() to allocate HVM domain memory.

It is the meminit hook for x86 HVM domains: https://github.com/xen-project/xen/blob/master/tools/libs/guest/xg_dom_x86.c#L1348

Dynamic memory / Populate on Demand

When dom->target_pages is smaller than dom->total_pages, the X86 meminit_hvm() function enables XENMEMF_populate_on_demand: https://github.com/xen-project/xen/blob/master/tools/libs/guest/xg_dom_x86.c#L1368

In this case, meminit_hvm() function calls xc_domain_set_pod_target() to set the populate on demand target to dom->target_pages: https://github.com/xen-project/xen/blob/master/tools/libs/guest/xg_dom_x86.c#L1454

meminit_hvm: vmemranges

The allocation of domain boot memory is defined using vmemranges.

meminit_hvm() creates a default vmemranges array based on the amount of

  • lowmem (below 4G, until dom->lowmem_end) and
  • highmem (above 4G, until dom->highmem_end).

The default vmemranges do not carry information about from which NUMA node the memory shall be allocated.

meminit_hvm() uses two vmemranges by default:

  • 0 to dom->lowmem_end
  • 4G to dom->highmem_end

The NUMA node IDs (nid) are set to 0:

  • A dummy vnode_to_pnode[] for nid maps 0 to XC_NUMA_NO_NODE.

Code: https://github.com/xen-project/xen/blob/master/tools/libs/guest/xg_dom_x86.c#L1371

Alternatively, callers of the meminit hook have the possibility to pass an array of memory ranges in dom->vmemranges, which can contain a specific NUMA node for each vmemrange andthe "exact" flag:

While vmemranges have been added for vNUMA, meminit_hvm() does not differentiate on that. It is only concerned about memory init. Hence, its code always acts on vmemranges to avoid code duplications and creates default non-NUMA vmemranges. If the caller does not pass specific dom->vmemranges for NUMA cases, the default vmemranges are used.

Caveat: When passing dom->vmemranges, Populate-On-Demand (target_pages < total_pages) cannot be used: With custom vmemranges as Populate-On-Demand does not support NUMA. As a result, memory overcommitment with ballooning is currently impossible for NUMA node-specific memory allocations.

Memory allocation of the Domain’s memory

meminit_hvm() attempts to allocate 1GB pages if possible, falls back on 2MB pages if 1GB allocation fails, and 4KB pages will be used eventually if both fail:

https://github.com/xen-project/xen/blob/master/tools/libs/guest/xg_dom_x86.c#L1475

For each vmemrange, new_memflags are composed, based on the base memflags:

unsigned int new_memflags = memflags;
unsigned int vnode = vmemranges[vmemid].nid;
// With custom vmemranges, vnode_to_pnode is custom too:
unsigned int pnode = vnode_to_pnode[vnode]

if ( pnode != XC_NUMA_NO_NODE )  // vnode maps to a physical NUMA node:
    // XENMEMF_exact_node: (XENMEMF_node(n) | XENMEMF_exact_node_request)
    new_memflags |= XENMEMF_exact_node(pnode);

Thus, to allocate memory in a specific NUMA node, this is needed:

  • Populate-On-Demand (POD) must not be used (target_pages == total_pages)
  • The sum of the vmemranges equals dom->total_pages
  • dom->vmemranges must be passed with a nid
  • dom->vnode_to_pnode must be passed, the nid must map to the pNUMA node to allocate on.
  • dom->nr_vmemranges and dom->nr_vnodes must be set accordingly

For the allocation of each vmemrange (extent), a call to xc_domain_populate_physmap() is used.

It calls the XENMEM_populate_physmap hypercall for each group of extents to allocate.

See populate_physmap.md for its implementation.

Other callers of the meminit() calls

libxl calls xc_dom_boot_mem_init() using libxl__build_dom() from init-xenstore-domain.c/build().