Split Large trace.dat Files with trace-cmd split

trace-cmd split breaks a large trace.dat into numbered chunks by duration, event count, page count or CPU, and leaves the source file untouched. A big trace.dat is awkward to inspect and easy to hand to the wrong tool, so splitting it first saves everyone's time. The examples use trace-cmd 3.2.0 from the installed trace-cmd package.

Allow about fifteen minutes, plus the time needed to copy the resulting files. You need a readable trace file and enough free space for the output. This is normally an unprivileged operation.

1. Check the installed command and make a work area

Confirm the executable and version before relying on option details:

$ command -v trace-cmd
/usr/bin/trace-cmd
$ trace-cmd --version
trace-cmd version 3.2.0 (not-a-git-repo)

Make a directory for the new files and copy the input there if it is an archive you must preserve exactly. Copying is optional, but it gives you a clean recovery path if a later command chooses an unsuitable output name.

$ mkdir -p ~/trace-split-work
$ cd ~/trace-split-work
$ cp --preserve=all /path/to/trace.dat ./trace.dat
$ test -r trace.dat && test -s trace.dat && echo 'input is readable and non-empty'
input is readable and non-empty

The copy can be large, so check available space with df -h . before making several output files. Do not use sudo just because the trace came from a privileged recording process: use it only if the file or destination genuinely needs elevated access, and keep the output ownership in mind.

2. Choose an output base and one split limit

The installed command's usage expects an output base with -o. A base of chunks/events produces chunks/events.1, chunks/events.2, and so on. Create the destination directory first:

$ mkdir -p chunks
$ trace-cmd split -o chunks/events -e 10000
$ printf 'exit status: %s\n' "$?"
exit status: 0

Here -e 10000 caps each output at 10,000 events, subject to the trace format and the final remainder. The command reads the default input name, trace.dat, and only ever writes new files: the input is never edited.

$ find chunks -maxdepth 1 -type f -name 'events.*' -printf '%f %s bytes\n' | sort -V
$ stat -c '%n %s bytes' trace.dat

3. Split a named input file

When the input is not called trace.dat, pass it with -i. Keep -o pointed at a different base so the source cannot be mistaken for an output:

$ trace-cmd split -i /path/to/incident-trace.dat \
    -o chunks/incident -s 2.5
$ printf 'exit status: %s\n' "$?"
exit status: 0

The manpage documents decimal timestamps as optional positional arguments. A start time begins the extracted range, and an end time stops creation after an event beyond that time. If you only need an end time, use 0.0 as the start:

$ trace-cmd split -i /path/to/incident-trace.dat \
    -o chunks/first-minute 0.0 60.0
$ printf 'exit status: %s\n' "$?"
exit status: 0

Use timestamps copied from trace-cmd report rather than inventing a time scale. The range is based on event timestamps, not the file's modification time.

4. Repeat the split across the whole trace

Without -r, a limit creates one output covering the selected range. Add -r when you want numbered files repeatedly until the end time or the input ends:

$ trace-cmd split -i /path/to/incident-trace.dat \
    -o chunks/by-events -r -e 10000
$ find chunks -maxdepth 1 -type f -name 'by-events.*' -printf '%f\n' | sort -V
by-events.1
by-events.2

The exact number of files depends on how many events the trace contains. A non-zero exit status means the command did not complete normally, so inspect its diagnostic before using the outputs in an analysis.

For a per-CPU split, add -c. For example, -c -p 10 asks for 10 pages per CPU rather than 10 pages for the whole trace. This is useful when CPU-local activity matters, but it can create more output than a global split.

$ trace-cmd split -i /path/to/incident-trace.dat \
    -o chunks/per-cpu -c -p 10
$ printf 'exit status: %s\n' "$?"
exit status: 0

5. Extract one CPU when that is the real question

Use -C CPU to extract only one CPU's events. It is not the same option as -c: -c makes the chosen limit apply per CPU, while -C filters the trace down to one CPU.

$ trace-cmd split -i /path/to/incident-trace.dat \
    -o chunks/cpu-1 -C 1
$ printf 'exit status: %s\n' "$?"
exit status: 0

Replace 1 with a CPU present in the trace. The command does not discover a meaningful CPU number for you, so if the result is unexpectedly empty or fails, check the report first to see which CPUs actually contributed events.

6. Handle failures and clean up safely

A missing default input produces an error like this, with exit status 2:

$ trace-cmd split -o chunks/events -e 10000
trace-cmd: No such file or directory
  error reading trace.dat
$ printf 'exit status: %s\n' "$?"
2

• Fix the working directory or supply -i. That is the usual cause of a missing-input error.
• Do not overwrite an existing output. If a name already exists, stop and inspect it before rerunning. The simplest recovery is a new base such as chunks/events-rerun, which preserves both the old output and your ability to compare runs.
• Clean up narrowly. Remove only the numbered outputs you created, using an explicit path and pattern. Keep the source trace as your recovery copy until the split files have been reported or otherwise checked.