Recover a Missing trace.dat with trace-cmd restore

A crashed trace-cmd record does not have to mean a wasted capture. trace-cmd restore can rebuild a readable trace.dat from whatever per-CPU data files survived, and this guide covers both the local recovery path and the less obvious case where the recording machine is no longer available.

Allow about 10 minutes for a straightforward recovery, plus time to locate and copy the files. The commands below use trace-cmd 3.2.0 from the installed Ubuntu package trace-cmd 3.2-1ubuntu2. You need the package installed and read access to the CPU files. Only the partial-metadata step normally needs elevated access to tracing files such as /proc/kallsyms or the tracing directory.

1. Identify the surviving files

Change to the directory containing the failed recording and list the per-CPU files.

cd /path/to/failed-record
ls -l trace.dat.cpu*

A normal recording uses names such as trace.dat.cpu0 and trace.dat.cpu1. If the original recording used trace-cmd record -o /path/to/session.dat, look for files based on that output name instead. The final trace.dat is missing because the record did not finish; do not treat that absence as proof that the CPU files are unusable.

Checkpoint: check that the input files are present before writing anything.

test -r trace.dat.cpu0 && test -r trace.dat.cpu1 && echo "CPU files are readable"

Replace the names with every CPU file from the recording. The command exits successfully only when both example files can be read. A missing CPU file means the recovered trace may be incomplete, so stop and account for it before continuing.

2. Rebuild the trace on the same machine

Run trace-cmd restore with the CPU files, choosing an output name that does not overwrite evidence.

trace-cmd restore -o recovered-trace.dat \
  trace.dat.cpu0 trace.dat.cpu1

The command appends the supplied per-CPU data to create a usable trace file. The default output is trace.dat, but naming the result explicitly makes the operation easier to review. Add every surviving CPU file, not just the two shown here.

Checkpoint: verify that the output exists and ask trace-cmd to read its report data.

test -s recovered-trace.dat && echo "Recovered trace exists"
trace-cmd report recovered-trace.dat | sed -n '1,40p'

The first command checks for a non-empty file. The second should print trace metadata or events. Its exact lines depend on the recording, kernel and enabled events. If it reports a malformed or incomplete file, recheck that you supplied all of the CPU files and that they came from one recording.

3. Recover a recording from another machine

A trace contains system-specific metadata. If the host that crashed is different from the host holding the CPU files, do not silently reconstruct it from the current host's kernel information. First create partial metadata on the recording machine, then transfer that file with the CPU data.

On the recording machine, create a partial file while the relevant tracing data is still available.

sudo trace-cmd restore -c -o box-partial.dat

With -c, trace-cmd reads the machine's tracing information and writes partial metadata instead of a complete reportable trace. When -o is omitted, the default is trace-partial.dat. The command does not complete the final trace by itself. Copy box-partial.dat to the machine where the CPU files are stored, using your normal secure transfer process.

On the destination machine, provide the partial file with -i while restoring the CPU files.

trace-cmd restore -i box-partial.dat -o recovered-box.dat \
  trace.dat.cpu0 trace.dat.cpu1

The -i option tells restore to use metadata from the other machine rather than reading the current system. Verify the result in the same way:

test -s recovered-box.dat && trace-cmd report recovered-box.dat | sed -n '1,40p'

4. Handle tracing files that live in a copied directory

The partial operation normally reads events from the current debugfs tracing directory and symbols from /proc/kallsyms. For a recording from another host, you can supply copies instead. This avoids accidentally mixing the old trace with the destination host's metadata.

Pass the copied tracing directory and symbol file to the partial operation.

sudo trace-cmd restore -c \
  -t /srv/old-host/debugfs/tracing \
  -k /srv/old-host/kallsyms \
  -o old-host-partial.dat

Use -t for the directory containing the copied tracing data and -k for the copied kallsyms file. These options only apply with -c. Keep the copied files together and record their origin; substituting a similarly named directory from another kernel can produce misleading results.

Common traps and recovery

No data files found: you probably ran restore without CPU-file arguments or from the wrong directory. The installed command exits with an error and status 255 in that case. Use absolute paths when the shell's current directory is uncertain.
Only some CPUs are listed: restore cannot invent missing per-CPU data. Find the original output prefix, check mounts and permissions, then include every file belonging to that recording.
The output name already exists: stop before replacing it. Choose a new -o path or move the old result aside only after preserving a checksum. There is no undo operation for an overwritten trace file.
Permission denied: ordinary restoration needs read access to the CPU files and write access to the destination directory. Use sudo only for the partial step when the kernel tracing or symbol files require it, and check ownership before copying sensitive trace data.

Done means

All surviving per-CPU files from one recording were identified.
A new, non-empty output file was created with trace-cmd restore.
trace-cmd report can read the recovered file.
Cross-machine recovery used -c and -i, with copied tracing and symbol data supplied where needed.