Measure NUMA Page Migration Speed with migspeed

migspeed shows how fast the kernel really moves memory between NUMA nodes, and a single-node box will lie to you about it. You will run it against chosen nodes, read the time and throughput for a sample of pages, and learn to spot a same-node smoke test before you mistake it for a benchmark. The installed command is from Ubuntu's numactl package version 2.0.18-1ubuntu0.24.04.1. Allow about ten minutes, including checking the node layout.

This is a live memory test. It allocates pages, dirties them and asks the kernel to migrate them. It does not touch files or services, but it consumes memory and can load a busy host, so run it in a maintenance window or start with a small page count. No elevated privilege was needed for the one-page test shown here; do not add sudo unless the system reports a permission failure.

1. Check the available NUMA nodes

Inspect the topology first. The source and destination arguments must name nodes that really exist on this machine.

$ numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3
node 1 cpus: 4 5 6 7

The CPU and memory lines vary by host. What matters is the available line. If it says 1 nodes (0), there is no cross-node path to measure: you can still run migspeed as an invocation check with node 0 as both arguments, but the reported rate tells you nothing about NUMA performance.

Checkpoint: write down one source node and a different destination node, for example 0 and 1. If the host has only node 0, stop here or use the same-node check in step 3.

2. Check the installed command syntax

The installed binary takes a source node then a destination node, plus -p to set the page count and -v for progress detail.

$ migspeed 2>&1
usage migspeed [-p pages] [-h] [-v] from-nodes to-nodes
      from and to nodes may specified in form N or N-N
      -p pages  number of pages to try (defaults to 1000)
      -v        verbose
      -h        usage

• Missing nodes. The command exits with status 1 when the required node arguments are absent.
• Node ranges. The usage text documents ranges such as N-N, but start with a single node so the test has an obvious source and destination.
• Trust the installed usage, not the manpage alone. The local migspeed(8) page describes -p and the default sample of 1000 pages, but not -v. Treat the binary's own usage output as the version-specific evidence, and do not assume a flag from another numactl build exists here.

3. Run a small, visible test

Start with one page. Replace the two numbers with the source and destination you recorded in step 1:

$ migspeed -p 1 FROM_NODE TO_NODE
0.0 Mbyte migrated in 0.00 secs. 47.2 Mbytes/second

FROM_NODE and TO_NODE are placeholders, not literal arguments: a real two-node run might use migspeed -p 1 0 1. The rate and elapsed time depend on the host, kernel, memory pressure and page size, so do not treat the numbers above as a benchmark result.

Check the exit status straight after the run:

$ printf '%s\n' "$?"
0

A zero status means the test completed. It does not prove the host has a useful cross-node topology, and it does not say every page in a real workload can be migrated as cleanly.

4. Repeat with a useful sample

Once the one-page invocation works, pick a sample large enough to smooth out timer noise. The manpage default is 1000 pages, each allocated by the test and dirtied before migration.

$ migspeed -p 1000 0 1
0.0 Mbyte migrated in 0.01 secs. 120.0 Mbytes/second

Those figures are illustrative output shape only. Your run should show the amount migrated, elapsed seconds and a rate in Mbytes/second. Keep the page count, node direction and host load constant for repeatable comparisons, and test both directions if you suspect an asymmetric NUMA link:

$ migspeed -p 1000 0 1
$ migspeed -p 1000 1 0

Warning: do not start with a large page count on a production host. Check free memory first, and only raise the count once you have agreed the temporary allocation and migration load are acceptable. This is a measurement tool, not a way to relocate an application's working set.

5. Add verbose output when diagnosing a result

Use -v when you need to see the selected nodes and the allocation stage:

$ migspeed -p 1 -v 0 0
From: 0
To: 0
Allocating 1 pages of 4096 bytes of memory
Dirtying memory....
Starting test
... kernelpagesize_kB=4
0.0 Mbyte migrated in 0.00 secs. 160.7 Mbytes/second

Addresses and rates change on every run. What is worth checking is that the intended source and destination are printed, the requested page count is allocated, and the command ends with a result line. A same-node run like this is a smoke test only, not evidence of inter-node migration performance.

6. Handle failures without changing system state

• Node rejected. Go back to numactl --hardware and check the numbering. Do not invent a destination node or copy a topology from another server.
• No NUMA support in the kernel. If the kernel lacks NUMA policy or page-migration support, the manpage's requirement is not met. Changing kernel boot settings or installing a different kernel is an administrative decision outside this test.
• Memory tight. Stop the test and let the process exit before retrying with fewer pages.

If a shell script needs to record the result, capture the status immediately and keep the output alongside it:

if output=$(migspeed -p 1000 0 1 2>&1); then
    printf '%s\n' "$output"
else
    status=$?
    printf 'migspeed failed with status %s\n%s\n' "$status" "$output" >&2
    exit "$status"
fi

This wrapper changes no persistent configuration. There is nothing to roll back, because migspeed does not create a configuration file or service: just stop running it. Keep any measurements alongside the kernel and package versions that produced them.

Done means

• Node numbers confirmed. numactl --hardware matched the numbers you used.
• Exit status checked. You started with a one-page test and read its result.
• Sample size recorded. A larger run used an explicit, noted page count.
• Same-node versus cross-node. You know which one you ran.
• No system state changed. Memory and workload impact stayed acceptable throughout.