Articles / Go

1057 words

  • go
  • tcp
  • networking
  • protocol-design
  • framing
  • security

Why Your Go TCP Service Needs a Length Prefix, Not a Newline

The first TCP protocol most of us write is "one message per line", read with bufio.Scanner or ReadString('\n'). It works in the demo. It stops working the first time a payload contains a newline, which is the first time anyone sends JSON with a string in it, or a file, or an attacker's idea of a string.

The fix is old and boring: say how long the message is, then send it. This post builds that in about thirty lines of Go and covers the two limits that stop it becoming a new hole.

TCP has no messages, only bytes

A TCP connection is a stream. One Write on the sender does not become one Read on the receiver. Two writes can arrive as one read, and one write can arrive as three.

So every protocol on top needs a rule for where a message ends. That rule is called framing, and there are really only two families:

  • Delimiter: read until a marker such as \n.
  • Length prefix: read a fixed-size header that says how many bytes follow.

HTTP/1 and SMTP use delimiters for their headers, and have a long history of bugs to show for it. Most binary protocols use lengths.

Where the newline version breaks

A delimiter only works if the payload can never contain it. That is a promise you have to keep forever, on every code path, including ones written by someone else next year.

  • Binary data (a file, a protobuf, an encrypted blob) contains 0x0A about one byte in 256, so it will break within a few hundred bytes.
  • Text that happens to contain a newline splits into two messages, and the second half is parsed as a fresh command.
  • Escaping fixes it until someone forgets to escape in one place.

That third point is the security angle. A user-supplied name containing \nDELETE everything becomes a second command. It is the same family of bug as header injection in HTTP, and it exists because data and control share one channel.

The receiving side also has to find the end

With a delimiter you do not know how much memory you need until you have already read it. You buffer, and keep buffering, until the marker shows up. A peer who never sends one makes you grow without bound.

Go's bufio.Scanner at least stops at 64 KiB by default and returns ErrTooLong, which is a reasonable safety net. Raise that limit to "fix" it and you have just raised the size of the hole.

The length-prefixed version

The header is a 4-byte big-endian unsigned integer. Then comes exactly that many bytes of payload, whatever they contain. Nothing inside the payload can be mistaken for framing, because the reader never looks inside it.

package frame

import (
	"encoding/binary"
	"errors"
	"fmt"
	"io"
)

// MaxFrame bounds memory per message. Pick it from your real traffic.
const MaxFrame = 1 << 20

func Write(w io.Writer, payload []byte) error {
	if len(payload) > MaxFrame {
		return fmt.Errorf("frame too large: %d bytes", len(payload))
	}
	buf := make([]byte, 4+len(payload))
	binary.BigEndian.PutUint32(buf, uint32(len(payload)))
	copy(buf[4:], payload)
	_, err := w.Write(buf)
	return err
}

func Read(r io.Reader) ([]byte, error) {
	var hdr [4]byte
	if _, err := io.ReadFull(r, hdr[:]); err != nil {
		return nil, err // io.EOF here means a clean close between frames
	}
	n := binary.BigEndian.Uint32(hdr[:])
	if n > MaxFrame {
		return nil, fmt.Errorf("frame too large: %d bytes", n)
	}
	buf := make([]byte, n)
	if _, err := io.ReadFull(r, buf); err != nil {
		if errors.Is(err, io.EOF) {
			err = io.ErrUnexpectedEOF
		}
		return nil, err
	}
	return buf, nil
}

Both ends use the same package, so the format lives in one place. A server loop becomes for { msg, err := frame.Read(conn); ... }, and an empty message is just a header of zero.

The check that matters: cap the length before allocating

The line if n > MaxFrame is the whole point of the design. The header is attacker-controlled, so without it a peer sends 0xFFFFFFFF and you call make([]byte, 4294967295). That is 4 GiB from a four-byte message.

Length prefixes move the denial-of-service risk from "unbounded buffering" to "trusting a number". The cap turns it back into a bounded cost. Check it before allocating, never after.

Pick the cap from what your protocol really sends, then add some headroom. If one message type legitimately needs to be large, give it its own limit once you know the type, rather than raising the global one.

Why io.ReadFull and not Read

A single conn.Read may return fewer bytes than you asked for, with a nil error. That is normal for streams, and it is the classic bug in hand-rolled readers: they work on loopback, where everything arrives at once, and fail on a real network.

io.ReadFull loops until the buffer is full or the connection fails. Its errors are worth knowing:

  • io.EOF: zero bytes were read, so the peer closed cleanly.
  • io.ErrUnexpectedEOF: some bytes arrived, then the stream ended. A truncated message.

That is why the body read above remaps EOF. Closing between frames is fine; closing in the middle of one is an error you want to see in your logs.

Quick detour: why one Write for the header and body

The writer builds one buffer rather than writing the header and then the payload. Two small writes can become two packets. Go enables TCP_NODELAY on TCP connections by default, so the second write is not held back waiting for an ACK, and you really would send a four-byte packet followed by the rest.

One buffer means one Write, and net.Conn.Write keeps going until everything is sent or it errors. The copy costs a little; for large frames you can use net.Buffers instead, which does a vectored write.

Things the length prefix does not fix

  • Slow peers. A peer can send the header, then one byte a minute. Set a deadline per frame with conn.SetReadDeadline, and refresh it for each frame rather than once at accept time.
  • Connection count. A cap per frame says nothing about the number of connections. That needs its own limit.
  • Versioning. A bare length gives you no room to evolve. Add a type or version byte inside the payload from day one.
  • Byte order. Pick big-endian (network order) and write it in the spec. Mixed endianness between a Go server and a C client is a very quiet way to lose an afternoon.

When a newline is still fine

Line protocols are not wrong. For a human typing into nc, a debug console or a log stream with a strict, validated alphabet, newlines are friendly and easy to inspect. The line to draw is whether the payload is arbitrary. If it can be anything, frame it by length; if you can guarantee it never holds the delimiter, a delimiter is simpler to debug with a plain terminal.

One last tip: when you write the frame spec, include the maximum size and the byte order in the first paragraph. Those are the two things the next implementer will otherwise guess.

← All articles