Articles / Security

821 words

  • parsers
  • security
  • protocol-design
  • input-validation
  • go

Make Your Parser Refuse Ambiguous Input

Ambiguous input is a parser bug even when every interpretation looks reasonable. If one component reads 01 as a number, another treats it as a string, and a third quietly normalises it, you do not have a flexible system. You have several systems making different decisions about the same bytes.

This is how request smuggling, signature mismatches, cache poisoning and configuration surprises begin. The fix is not merely better documentation. Make invalid or ambiguous representations impossible to accept at the boundary.

Start by deciding what the bytes mean

A parser should answer one small question: does this exact input have one meaning under this grammar? If the answer is "probably", reject it or define the missing rule before writing code.

Suppose a service accepts a tiny header format:

version=1
mode=fast
limit=50

That looks harmless until someone asks what these mean:

  • mode=fast=really
  • limit=050
  • mode=fast followed by another mode=slow
  • version=1 followed by a blank line and then another document
  • limit=50 with one trailing space

Every one of those cases needs a deliberate answer. A permissive parser will still answer, but its answer may differ from the validator, logger, cache or downstream service.

Do not silently repair the input

Normalisation is a decision, not a neutral convenience. Trimming whitespace, folding case, decoding escapes or accepting alternate number forms can be sensible, but only when the grammar says so and every consumer applies the same rule.

There is a useful distinction between canonical input and normalised input. A canonical parser accepts one spelling, such as 50, and rejects 050. A normalising parser might accept both and convert them to the same value. Neither choice is automatically correct. The dangerous choice is allowing different parts of the system to make it independently.

Quick detour: Unicode makes this especially entertaining. Two strings can look identical while containing different code-point sequences. If a name is checked before normalisation but stored after normalisation, an attacker may get two answers from one identity. If names matter to security, define the allowed character set narrowly or specify normalisation at the boundary and test it there.

A small strict parser is often enough

Reject ambiguity before producing a useful value. This Go example accepts a bounded header document, permits only known keys, rejects duplicate keys, and refuses extra separators or trailing whitespace.

package header

import (
	"fmt"
	"strings"
)

type Header struct {
	Version int
	Mode    string
	Limit   int
}

func Parse(input []byte) (Header, error) {
	if len(input) == 0 || len(input) > 4096 {
		return Header{}, fmt.Errorf("invalid header size")
	}
	if strings.ContainsAny(string(input), "\r\x00") {
		return Header{}, fmt.Errorf("invalid control character")
	}
	if strings.HasSuffix(string(input), "\n") {
		return Header{}, fmt.Errorf("trailing newline is not allowed")
	}

	seen := make(map[string]bool)
	var header Header
	for _, line := range strings.Split(string(input), "\n") {
		if line == "" || strings.Count(line, "=") != 1 {
			return Header{}, fmt.Errorf("invalid line %q", line)
		}
		key, value, _ := strings.Cut(line, "=")
		if key == "" || value == "" || strings.TrimSpace(line) != line {
			return Header{}, fmt.Errorf("invalid field %q", line)
		}
		if seen[key] {
			return Header{}, fmt.Errorf("duplicate field %q", key)
		}
		seen[key] = true

		switch key {
		case "version":
			if value != "1" {
				return Header{}, fmt.Errorf("unsupported version %q", value)
			}
			header.Version = 1
		case "mode":
			if value != "fast" && value != "safe" {
				return Header{}, fmt.Errorf("invalid mode %q", value)
			}
			header.Mode = value
		case "limit":
			if value[0] == '0' || value < "1" || value > "9999" {
				return Header{}, fmt.Errorf("invalid limit %q", value)
			}
			var limit int
			for _, digit := range value {
				if digit < '0' || digit > '9' {
					return Header{}, fmt.Errorf("invalid limit %q", value)
				}
				limit = limit*10 + int(digit-'0')
			}
			header.Limit = limit
		default:
			return Header{}, fmt.Errorf("unknown field %q", key)
		}
	}
	if header.Version == 0 || header.Mode == "" || header.Limit == 0 {
		return Header{}, fmt.Errorf("required field missing")
	}
	return header, nil
}

The example is intentionally strict. It does not sort fields, fill in defaults or accept a second spelling for a value. Those can be added later, but each addition should be a grammar change with tests, not an accidental side effect of a library call.

One detail is worth calling out: the parser rejects a final newline. That is not universally the best rule, but it prevents two possible interpretations of concatenated input. If records are separated by newlines, define whether the separator belongs to the record and enforce that choice consistently.

Be suspicious of successful library calls

Parsing one value successfully does not prove that the whole input was consumed. Many libraries are designed to find a value at the start of a stream. Your protocol may require the stream to contain exactly one value.

This matters with JSON, numbers, timestamps and encoded data. A decoder that reads one JSON object may leave another object unread. A numeric parser may accept a prefix while a caller ignores the remaining bytes. A base64 decoder may accept a padded and an unpadded form that later produce different signatures if the original spelling is also authenticated.

After decoding, check the remainder. For a structured format, also decide what happens with duplicate object members, unknown fields, excessive nesting and numbers outside the range your application can represent. If the library does not expose the rule you need, put a stricter wrapper around it or choose another format.

Test the edges as properties, not anecdotes

The useful test cases are the ones that almost pass. Test the grammar's boundary, then assert that rejected inputs stay rejected as the parser evolves.

  • Every valid spelling produces exactly one value.
  • Duplicate fields are rejected, regardless of their order.
  • Adding a byte after a valid document makes it invalid.
  • Whitespace, case and leading zeroes follow the documented rule.
  • Unknown fields do not become accepted merely because a library ignores them.
  • Oversized or deeply nested input fails before expensive work begins.

Fuzzing is particularly good at finding accidental permissiveness. Start with a few valid documents, mutate bytes, and assert either a valid result that satisfies the invariants or a clean error. A panic, partial result or silently ignored suffix is a parser failure.

There is a slightly awkward but valuable design rule here: make the error boring and the accepted language small. Security boundaries benefit from predictable rejection. If a caller wants to accept a new spelling, add it to the grammar, document the interaction with existing spellings, and test both sides. The parser should never have to guess what the author meant.

← All articles