Split Large trace.dat Files Without Losing the Original

A large trace.dat is awkward to inspect and easy to hand to the wrong tool. You will split it into numbered trace files by duration, event count, page count or CPU, while leaving the source file untouched. The examples use trace-cmd 3.2.0 from the installed trace-cmd package.

Allow about fifteen minutes, plus the time needed to copy the resulting files. You need a readable trace file and enough free space for the output. This is normally an unprivileged operation. Do not run the split in a directory where numbered output names could overwrite unrelated files.

1. Check the installed command and make a work area

Confirm the executable and version before relying on option details:

$ command -v trace-cmd
/usr/bin/trace-cmd
$ trace-cmd --version
trace-cmd version 3.2.0 (not-a-git-repo)

Make a directory for the new files and copy the input there if it is an archive you must preserve exactly. Copying is optional, but it gives you a clean recovery path if a later command chooses an unsuitable output name.

$ mkdir -p ~/trace-split-work
$ cd ~/trace-split-work
$ cp --preserve=all /path/to/trace.dat ./trace.dat
$ test -r trace.dat && test -s trace.dat && echo 'input is readable and non-empty'
input is readable and non-empty

The copy can be large. Check available space with df -h . before making several output files. Do not use sudo just because the trace came from a privileged recording process. Use it only if the file or destination genuinely requires elevated access, and keep the output ownership in mind.

2. Choose an output base and one split limit

The installed command's usage expects an output base with -o. A base of chunks/events produces chunks/events.1, chunks/events.2, and so on. Create the destination directory first:

$ mkdir -p chunks
$ trace-cmd split -o chunks/events -e 10000
$ printf 'exit status: %s\n' "$?"
exit status: 0

Here -e 10000 makes each output contain at most 10,000 events, subject to the trace format and the final remainder. The command reads the default input name, trace.dat. The output names are new files; the input is not edited.

Only one of -s, -m, -u, -e and -p may be used for one split. They mean seconds, milliseconds, microseconds, events and pages respectively. Pick the unit that matches the question you are asking, rather than combining several limits and guessing which one stopped the file.

Checkpoint: list the result and compare the source size. The individual sizes will differ, so do not expect each output to have the same number of bytes.

$ find chunks -maxdepth 1 -type f -name 'events.*' -printf '%f %s bytes\n' | sort -V
$ stat -c '%n %s bytes' trace.dat

3. Split a named input file

When the input is not called trace.dat, pass it with -i. Keep -o pointed at a different base so the source cannot be mistaken for an output:

$ trace-cmd split -i /path/to/incident-trace.dat \
    -o chunks/incident -s 2.5
$ printf 'exit status: %s\n' "$?"
exit status: 0

The manpage documents decimal timestamps as optional positional arguments. A start time begins the extracted range, and an end time stops creation after an event beyond that time. If you need only an end time, use 0.0 as the start:

$ trace-cmd split -i /path/to/incident-trace.dat \
    -o chunks/first-minute 0.0 60.0
$ printf 'exit status: %s\n' "$?"
exit status: 0

Use timestamps copied from trace-cmd report, rather than inventing a time scale. The range is based on event timestamps, not the file's modification time.

4. Repeat the split across the whole trace

Without -r, a limit creates one output covering the selected range. Add -r when you want numbered files repeatedly until the end time or the input ends:

$ trace-cmd split -i /path/to/incident-trace.dat \
    -o chunks/by-events -r -e 10000
$ find chunks -maxdepth 1 -type f -name 'by-events.*' -printf '%f\n' | sort -V
by-events.1
by-events.2

The exact number of files depends on how many events the trace contains. A single non-zero status means the command did not complete normally, so inspect its diagnostic before using the outputs in an analysis.

For a per-CPU split, add -c. For example, -c -p 10 asks for 10 pages per CPU rather than 10 pages for the whole trace. This is useful when CPU-local activity matters, but it can create more output than a global split.

$ trace-cmd split -i /path/to/incident-trace.dat \
    -o chunks/per-cpu -c -p 10
$ printf 'exit status: %s\n' "$?"
exit status: 0

5. Extract one CPU when that is the real question

Use -C CPU to extract only one CPU's events. The option is different from -c: -c makes the chosen limit apply per CPU, while -C filters the trace to one CPU.

$ trace-cmd split -i /path/to/incident-trace.dat \
    -o chunks/cpu-1 -C 1
$ printf 'exit status: %s\n' "$?"
exit status: 0

Replace 1 with a CPU present in the trace. The command does not discover a meaningful CPU number for you. If the result is unexpectedly empty or fails, inspect the report first and check which CPUs actually contributed events.

6. Handle failures and clean up safely

A missing default input produces an error like this and status 2:

$ trace-cmd split -o chunks/events -e 10000
trace-cmd: No such file or directory
  error reading trace.dat
$ printf 'exit status: %s\n' "$?"
2

Fix the working directory or supply -i. If an output name already exists, stop and inspect it before rerunning. The simplest recovery is to choose a new base such as chunks/events-rerun. That preserves both the old output and your ability to compare runs.

When you are satisfied with the new files, remove only the numbered outputs you created, using an explicit path and pattern. The source trace is the recovery copy, so keep it until the split files have been reported or otherwise checked.