Split Files by Lines, Bytes or Records with GNU split

A log file too big to email, a CSV too big for one import job: GNU split divides it into pieces you can name, sort and reassemble on purpose. These examples use coreutils 9.4, installed here as package version 9.4-3ubuntu6.3.

• Time: about fifteen minutes.
• Needs: a shell, a readable input file, and enough free space for the pieces.
• Privileges: none. The normal examples write only in your current directory, so pick that directory on purpose.

1. Check the installed command

Confirm the implementation and skim the local option list before this goes anywhere near a script:

$ split --version
split (GNU coreutils) 9.4
$ dpkg-query -W -f='${Package} ${Version}\n' coreutils
coreutils 9.4-3ubuntu6.3
$ split --help

GNU split reads a named file, or standard input when you omit the file or write -. Left to its defaults it writes 1,000 lines per piece, prefix x, two-letter suffixes like xaa and xab.

$ command -v split
/usr/bin/split

2. Split a text file by lines

Make a small test input first so the result is easy to eyeball. This only touches files in the current directory:

$ mkdir -p split-work
$ cd split-work
$ printf 'alpha\nbeta\ngamma\ndelta\nepsilon\nzeta\n' > records.txt
$ split --lines=2 records.txt part-
$ ls -1 part-*
part-aa
part-ab
part-ac

• --lines=2 puts two newline-terminated records in each output file. The final piece can hold fewer.
• part- is the output prefix, the last argument. Skip it and you get xaa, xab, and so on instead.

Inspect the pieces and count their lines before trusting them:

$ for file in part-*; do printf '%s: ' "$file"; wc -l < "$file"; done
part-aa: 2
part-ab: 2
part-ac: 2
$ cat part-aa part-ab part-ac
alpha
beta
gamma
delta
epsilon
zeta

Shell glob order matters here. The command above works because these alphabetic suffixes happen to sort in creation order. Keep the original input around until the pieces have been checked.

3. Prevent accidental overwrites

split opens its output names for writing, which means a matching name from an earlier run gets replaced without a fuss. Use a new prefix, or check for existing files first:

$ if compgen -G 'batch-*' >/dev/null; then
>     printf 'refusing to overwrite existing batch-* files\n' >&2
>     exit 1
> fi
$ split --lines=2 records.txt batch-
$ ls -1 batch-*
batch-aa
batch-ab
batch-ac

$ rm -- batch-aa batch-ab batch-ac part-aa part-ab part-ac records.txt
$ cd ..
$ rmdir split-work

That deletion is permanent unless another copy exists. For anything valuable, move the pieces to a quarantine directory instead, or keep them until whoever consumes them has verified them.

4. Use numeric suffixes for scripts

Alphabetic suffixes run from aa through zz and then need more suffix space. Numeric names are usually easier for another program to sort and parse:

$ printf 'one\ntwo\nthree\nfour\nfive\n' > input.txt
$ split --lines=2 --numeric-suffixes=1 --suffix-length=3 input.txt chunk-
$ ls -1 chunk-*
chunk-001
chunk-002
chunk-003

• --numeric-suffixes=1 switches to numeric suffixes and starts counting at 1.
• --suffix-length=3 fixes the width so it is predictable.
• -d is the short form, starting at 0. Use the long forms in scheduled jobs where clarity matters more than keystrokes.

5. Split by bytes without confusing units

Use --bytes when a piece needs an approximate or exact byte limit rather than a line limit:

$ split --bytes=1K --numeric-suffixes=1 --suffix-length=2 \
    large.bin binary-
$ wc -c binary-*
1024 binary-01
1024 binary-02

In GNU split, K means 1,024 bytes. Decimal units like KB mean 1,000 bytes; binary spellings like KiB also work. The final file comes out shorter when the input size is not an exact multiple. Do not use byte splitting on a format that must stay record-aligned.

For text or another record-based format, --line-bytes=SIZE keeps at most that many bytes of whole records per output file, never cutting a record in half. A single record longer than the limit can still push a piece over it. Check this against your format's actual record boundaries rather than trusting the file sizes alone.

6. Keep records together or spread them out

--number divides input into a chosen number of outputs, and its mode decides whether records survive intact and how they get assigned:

$ split --number=l/3 --numeric-suffixes=1 --suffix-length=2 \
    records.txt group-
$ for file in group-*; do printf '%s: ' "$file"; wc -l < "$file"; done
group-01: 2
group-02: 2
group-03: 1

• l/3 makes three pieces without splitting any line or record.
• Plain 3 divides purely by input size, ignoring record boundaries.
• r/3 distributes records round-robin instead of sequentially.
• --elide-empty-files suppresses empty outputs a numbered mode would otherwise create.

Treat the mode as part of the data contract. A consumer expecting sequential batches will not get the same order out of round-robin output.

7. Read standard input and verify a reassembly

Omit the input file, or use -, to read from a pipe. Give the output prefix explicitly so the names are obvious:

$ printf 'red\ngreen\nblue\nyellow\n' | split --lines=2 - stdin-
$ cat stdin-aa stdin-ab
red
green
blue
yellow

For a byte-for-byte check, take a digest before splitting, concatenate the pieces back in order, then compare digests. This leaves the original input untouched:

$ sha256sum input.txt
<hash>  input.txt
$ cat chunk-001 chunk-002 chunk-003 > reassembled.txt
$ sha256sum reassembled.txt
<hash>  reassembled.txt
$ cmp --silent input.txt reassembled.txt && echo 'reassembly matches'
reassembly matches

The <hash> above is a stand-in for the real value your machine prints. If cmp reports a difference, stop before deleting anything. Check the suffix order, the split mode you chose, and whether the input ended with a newline.

8. Diagnose common failures

An error saying the input cannot be opened usually means the path, permissions or current directory are wrong. Check without touching the file:

$ pwd
$ ls -l /path/to/input.txt
$ test -r /path/to/input.txt && echo readable

An error about an invalid size, line count or suffix length means the argument itself was rejected. Check the spelling and value against split --help. If an output file already exists, change the prefix, or move the old output aside once you have confirmed it is safe to do so.

split does not encrypt, compress, checksum or delete the source. Those are separate operations with separate failure and recovery paths, so for a service or scheduled job, write to a dedicated directory with a temporary prefix until verification is complete.