Test NUMA Memory Policies Safely with memhog

When an app runs slower on one NUMA node than another, memhog lets you reproduce the effect on purpose with a small, throwaway allocation. It comes from the numactl package, version 2.0.18-1ubuntu0.24.04.1 on Ubuntu 24.04, and it works by mapping a region with mmap, applying whatever policy you ask for, then writing across it with memset.

Allow about ten minutes. You need a shell, the numactl package and enough free memory for the test size. None of this needs elevated privileges on a normal box. Start small: a large allocation can put pressure on the whole machine, and a binding policy can fail outright rather than quietly falling back to another node.

1. Check the installed command

Confirm the binary, its package version and the local help text before you copy any example below:

$ command -v memhog
/usr/bin/memhog
$ dpkg-query -W -f='${Package} ${Version}\n' numactl
numactl 2.0.18-1ubuntu0.24.04.1
$ memhog --help
memhog [-fFILE] [-rNUM] [-H] size[kmg] [policy [nodeset]]
 -f mmap is backed by FILE
 -rNUM repeat memset NUM times
 -H disable transparent hugepages
 Policies: preferred-many local interleave membind preferred default

The syntax is positional once you're past the options: a size, then an optional policy and node set. Size suffixes are case-insensitive, so K, M and G all mean kilobytes, megabytes and gigabytes. The manpage agrees: size, then policy, then nodeset, with default as the local-allocation policy.

2. Inspect the available NUMA nodes

Ask numactl what this host actually offers:

$ numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 31879 MB
node 0 free: 2505 MB

Your numbers will differ. Note the node numbers from the available line and use only those in what follows. This test box has a single node, so an interleave test here can't show distribution across multiple nodes. That's a limit of this host, not proof that interleaving does nothing on a multi-node system.

Do not treat the memory totals as a safe allocation size. They describe the host, not a request memhog can safely make.

3. Run the smallest default-policy test

Start with a one-megabyte region and a single pass. The allocation and write happen while the command runs, then the process exits:

$ memhog -r1 1M --default
.
$ printf 'exit status: %s\n' "$?"
exit status: 0

The dot is progress from the write pass. Status 0 means this run finished cleanly. --default asks for local allocation, the same policy the manpage's own example uses when you leave the policy argument out entirely:

$ memhog -r1 1M
.

Keep the first run small. Only scale up once you have a real reason and have checked free memory first.

4. Compare the policy forms

With node 0 available, these four commands exercise the main policy spellings without leaving any persistent configuration change behind:

$ memhog -r1 1M --interleave 0
.
$ memhog -r1 1M --membind 0
.
$ memhog -r1 1M --preferred 0
.
$ memhog -r1 1M --local 0
.

• interleave spreads the allocation across the listed nodes and can fall back if the current node is short on memory.
• membind is stricter: allocation is restricted to the listed nodes, so running out of space there is an expected failure, not a reason to keep bumping the size.
• preferred names one node it would rather use but may fall back elsewhere.
• local and preferred-many also appear in the installed help; use their exact spelling and check behaviour against your own version's help text.

Node sets can hold multiple nodes. The manpage's own example uses 0-3 for nodes 0 through 3, but only use a range if those nodes actually exist on your hardware.

5. Repeat the memory writes deliberately

-rNUM repeats the memset pass. Reach for it when you want a longer, repeatable write workload rather than a bigger allocation:

$ memhog -r4 1M --interleave 0
....
$ printf 'exit status: %s\n' "$?"
exit status: 0

More repetitions cost more time, not more mapped memory. They still burn CPU and memory bandwidth, so keep the number modest on a shared host. The dots are not a placement report: if you actually need to know where pages landed, pair this with a separate tool such as numastat and make sure it catches the short-lived memhog process while it's still running.

6. Use a backing file only when you need one

-fFILE backs the mapping with a real file. That touches the filesystem and can overwrite an existing file, so point it at a fresh path in a writable test directory:

$ test_file="/tmp/memhog-test.$USER"
$ memhog -r1 1M --default -f"$test_file"
.
$ printf 'exit status: %s\n' "$?"
exit status: 0

Do not aim this at a valuable file or a path another test is using. When you're done, remove only the exact temporary file you created, for example rm -- "$test_file". If the file already existed, restore it from backup rather than guessing at its old contents. For an ordinary policy test, skip -f entirely and avoid this whole class of risk.

7. Disable transparent hugepages only for a controlled comparison

-H turns off transparent hugepages for this one allocation. It's a test switch, not a system-wide hugepage setting:

$ memhog -H -r1 1M --default
.

Compare this run against an otherwise identical one and note down both command lines. Do not read a performance result into the dot count. If hugepage behaviour is actually the question you're chasing, use a real measurement tool and a longer, controlled workload.

Common traps and recovery

• Wrong node number. Run numactl --hardware again. A node set copied from another machine is not portable.
• Binding failure. Reduce the size, check free memory on the requested nodes, and don't switch to interleave unless a fallback is actually acceptable for this experiment.
• Run taking too long. Stop it with Ctrl-C if the allocation is putting pressure on the host. A terminated process releases its private mapping; remove any explicitly created backing file separately.
• Confusing output. Dots show write passes, not NUMA placement. Capture the exit status before running anything else.