Go's bufio.Scanner Stops at 64 KiB: Reading Long Lines Safely

A line-oriented Go program can work perfectly for months, then reject one unusually enthusiastic JSON record with:

bufio.Scanner: token too long

The file is not corrupt. The machine is not short of memory. The default bufio.Scanner simply refuses to buffer a token larger than 64 KiB.

That limit is easy to raise. The more interesting question is what the limit should become, because replacing 64 KiB with "however much the sender fancies" turns a tidy parsing fix into a memory-exhaustion bug.

Where the limit comes from

A scanner does not inherently read lines. It reads tokens according to a split function. The default split function is bufio.ScanLines, so each line is one token and must fit in the scanner's token buffer.

The standard library defines bufio.MaxScanTokenSize as 64 * 1024. The documentation adds a small but relevant wrinkle: the actual maximum token can be slightly smaller because the buffer may also need to contain a newline. Treat 64 KiB as the buffering boundary, not a contractual maximum line length.

A common loop conceals the distinction between EOF and failure:

scanner := bufio.NewScanner(input)
for scanner.Scan() {
	process(scanner.Text())
}

Scan returns false for both. If the program does not inspect Err, a long line looks like an ordinary end of file and the input is silently truncated. Always finish the loop like this:

if err := scanner.Err(); err != nil {
	return fmt.Errorf("scan input: %w", err)
}

The straightforward fix: Scanner.Buffer

If the format has a sensible upper bound, keep the scanner and configure it before the first call to Scan:

const maxRecordSize = 1024 * 1024 // 1 MiB, including parsing headroom

scanner := bufio.NewScanner(input)
scanner.Buffer(make([]byte, 64*1024), maxRecordSize)

for scanner.Scan() {
	if err := process(scanner.Bytes()); err != nil {
		return err
	}
}
if err := scanner.Err(); err != nil {
	return fmt.Errorf("scan input: %w", err)
}

The first argument supplies the initial buffer. The second sets the maximum buffer size that the scanner may allocate. Here, normal records fit in 64 KiB while exceptional records can grow towards 1 MiB.

Calling Buffer after scanning has begun panics, which is a particularly direct way for the API to say that buffer policy belongs in setup code.

There is also a lifetime detail worth remembering. scanner.Bytes() avoids a string allocation, but its backing storage may be overwritten by the next scan. Processing it synchronously is fine. Sending it to another goroutine or storing it requires a copy:

record := bytes.Clone(scanner.Bytes())
queue <- record

Do not merely choose an enormous number

A scanner must hold the whole token before returning it. Setting the maximum to 1 GiB means one malicious or malformed line can make one connection consume roughly 1 GiB. With several concurrent uploads, the arithmetic becomes unpleasant rather quickly.

Choose the limit from the input format, not from available RAM. A 1 MiB cap might be reasonable for newline-delimited JSON events if producers have a documented record limit. It would be arbitrary for genomic data, generated SQL or machine-learning datasets, where very long physical lines may be normal.

Actually, this bit is interesting: a scanner's limit is both a correctness decision and a resource policy. If the source is untrusted, it belongs alongside request-body limits, connection limits and timeouts. Raising one without considering the others only moves the bottleneck.

Use bufio.Reader when records can be genuinely large

The scanner documentation recommends bufio.Reader when programs need large tokens or tighter error control. ReadString('\n') and ReadBytes('\n') are convenient, but both can accumulate an unbounded line. They are not, by themselves, a defence against hostile input.

ReadSlice is a better primitive for an explicitly bounded reader. It returns a fragment and bufio.ErrBufferFull whenever the reader's internal buffer fills before finding the delimiter. The caller can assemble fragments while enforcing a hard maximum:

package records

import (
	"bufio"
	"bytes"
	"errors"
	"fmt"
	"io"
)

var ErrLineTooLong = errors.New("line exceeds configured limit")

// ReadLine returns one line without its trailing LF or CRLF.
// maxRecordBytes counts the complete physical record, including its newline.
func ReadLine(r *bufio.Reader, maxRecordBytes int) ([]byte, error) {
	if maxRecordBytes < 1 {
		return nil, fmt.Errorf("max record bytes must be positive")
	}

	capacity := maxRecordBytes
	if capacity > 4096 {
		capacity = 4096
	}
	line := make([]byte, 0, capacity)

	for {
		fragment, err := r.ReadSlice('\n')
		if len(fragment) > maxRecordBytes-len(line) {
			return nil, ErrLineTooLong
		}
		line = append(line, fragment...)

		switch {
		case err == nil:
			line = bytes.TrimSuffix(line, []byte{'\n'})
			line = bytes.TrimSuffix(line, []byte{'\r'})
			return line, nil
		case errors.Is(err, bufio.ErrBufferFull):
			continue
		case errors.Is(err, io.EOF):
			if len(line) == 0 {
				return nil, io.EOF
			}
			return line, nil
		default:
			return nil, err
		}
	}
}

The subtraction in the length check is deliberate. Checking len(line)+len(fragment) can overflow an int in sufficiently absurd circumstances. The function also gives the limit an exact meaning: it counts bytes on the wire, including the line ending. Explicit semantics save off-by-one arguments later.

On an overlong record, this function returns immediately and leaves the reader somewhere inside that record. That is appropriate when the caller will reject the file or close the connection. A parser that intends to recover must drain fragments until it reaches a newline, without retaining them, before attempting another record.

Scanner or Reader?

Use Scanner when records have a known, moderate maximum and token-by-token iteration is what the program needs.
Call Scanner.Buffer before scanning, set a defensible cap, and always check Scanner.Err.
Use bufio.Reader when lines may be large, exact size enforcement matters, or recovery behaviour must be under your control.
For formats where a "line" may be hundreds of megabytes, reconsider whether the program needs to materialise it at all. A streaming decoder may be the honest solution.

The 64 KiB default is not a mysterious Go failure. It is the scanner declining to make an unlimited allocation on your behalf. Once the input format supplies a real boundary, the correct fix becomes pleasantly ordinary.

Phone:

Email:

[email protected]

Category:

Tags: