TMEM

Enabling Transcendent Memory (tops Linux v5.2)

Update: TMEM has been removed. The latest upstream version containing it seems to be 5.2, namely revision 5.2.21.

Tested with Xen version 4.11.0 and Linux dom0 & domU 4.18.16 & 5.2.14.

TMEM enables dynamic memory allocation for the guests, and the shrinking is quite fast. In less that 3 minutes, it goes from 5GB to approx 900MB. And after quite a while, I even goes down to less than 700MB on idling guests. And this takes into account guests' buff/cache that is still allowed to live, which saves lots of disk I/O hence increases performance, a lot.

Among others Linux kernel settings for the domU, provided by default with x86_64_defconfig and xenconfig, you need to enabled those,

CONFIG_XEN_SELFBALLOONING=y
CONFIG_XEN_TMEM=m

Since TMEM refuses to be built-in, but only as a module, you will have to deliver /lib/modules/kernel-ver onto the PV guests anyways. And make sure depmod -a gets executed at some point with the newly running kernel.

Enabling a swap space onto the guest is not mandatory. It works without swap. Also the additional tmem kernel argument at boot time is not required. You just need to load the module once the guest is up and running, either manually,

modprobe tmem

or by enabling it at startup e.g. on Debian/Ubuntu,

echo tmem >> /etc/modules

You should see this from the system logs,

[   37.471390] xen:tmem: frontswap enabled, RAM provided by Xen Transcendent Memory
[   37.471410] xen:tmem: cleancache enabled, RAM provided by Xen Transcendent Memory
[   37.471423] xen_selfballoon: Initializing Xen selfballooning driver
[   37.471425] xen_selfballoon: Initializing frontswap selfshrinking driver

And magic happens, its xentop MAXMEM(k) starts decreasing, and free -m within the guest also reports decreasing memory.

Note. It seems that for TMEM, only the memory= guest setting matters and that the maxmem= setting actually disables it: it shows static values on xentop output.

This feature has been provided by Dan Magenheimer at Oracle.