Articles / Go

1109 words

  • go
  • gomemlimit
  • garbage-collection
  • containers
  • kubernetes
  • cgroups

Keeping the OOM Killer Off Your Go Container with GOMEMLIMIT

A Go service in a pod with a 1GiB memory limit gets SIGKILLed by the kernel, restarts, and does it again an hour later. It had been fine for weeks, with boring, flat memory graphs, and nobody touched the deployment.

There was no leak in the traditional sense. Heap profiles taken minutes before the kill show a live set of maybe 150MiB. The garbage collector was doing exactly what it was told to do; what it was told to do just had nothing to do with the number that mattered.

That number is the cgroup memory limit, and until Go 1.19 the runtime had no idea it existed.

GOGC is a ratio, not a ceiling

Go's garbage collector has always been paced by GOGC, which defaults to 100. The pacer's rule is roughly: after a GC finishes and the live heap is measured, allow the heap to grow to double that size before triggering the next collection.

That's a good rule if your only concern is CPU spent collecting versus memory spent holding garbage, because it scales the trigger to how much your program is actually using:

  • a service with a 10MiB live set gets GC'd around 20MiB;
  • a service with a 2GiB live set gets GC'd around 4GiB.

Proportional, cheap, and completely blind to how much memory is actually available.

Why that blindness gets you killed inside a cgroup

The blindness was fine when Go programs ran on machines they could see the whole of. It stops being fine the moment the program is confined to a slice of a machine by a cgroup. Then a burst in allocations can double the live heap well past the container's limit before the next GC cycle even fires. Typical bursts:

  • a large request body;
  • a spike in concurrent connections;
  • a batch job that briefly holds more state than usual.

It happens because the pacer's target was set relative to the previous, smaller heap. And the kernel's OOM killer doesn't negotiate. It doesn't wait for a graceful GC pass. It sends SIGKILL to the process with the highest OOM score in the cgroup and moves on.

Quick detour, because this rhymes with something. Go has the same problem on the CPU side, where GOMAXPROCS defaults to the host's core count rather than the container's CPU quota. Same root cause: the runtime was written for a world of whole machines, and cgroups quietly slice that world up underneath it.

GOMEMLIMIT is a soft cap that sits alongside GOGC

Go 1.19 added a second control: GOMEMLIMIT, a soft cap on the total memory the runtime is willing to use. You set it either as an environment variable (GOMEMLIMIT=900MiB) or programmatically with runtime/debug.SetMemoryLimit. "Soft" is doing real work in that sentence, so it's worth being precise about what it means.

GOMEMLIMIT doesn't replace GOGC's ratio-based trigger; it sits alongside it as an upper bound. Below the limit, GC behaves exactly as before: cheap, ratio-paced, mostly invisible.

As usage climbs towards the limit, the pacer starts running more aggressive and more frequent collections to keep total memory under the ceiling. It will also lean on the runtime's memory scavenger to hand pages back to the OS sooner than it otherwise would. It is genuinely trying to respect the number you gave it.

It lowers the odds of an OOM kill, it doesn't remove them

The runtime can't guarantee the limit. The Go documentation is explicit that in extreme cases, such as an allocation rate the collector simply cannot keep pace with, the runtime will exceed the limit rather than deadlock the program.

A soft limit that the runtime overshoots by design under load is a real constraint on how much protection this buys you. It's the reason GOMEMLIMIT reduces the probability of an OOM kill rather than eliminating it.

There's also a coverage gap. The limit applies to memory the Go runtime tracks and manages directly:

  • heap allocations;
  • goroutine stacks;
  • GC bookkeeping.

Memory allocated outside the Go allocator, most obviously C memory allocated through cgo, isn't counted towards it. It still counts fully against the cgroup limit that actually gets you killed. If your service does any cgo-heavy work, that's memory GOMEMLIMIT can't see coming.

Read the limit from the container, not from a constant

The value you want isn't baked in at build time. It's whatever limit the orchestrator handed the container at deploy time, minus some headroom for the memory GOMEMLIMIT doesn't track. Reading it straight from the cgroup at startup keeps the binary portable across environments with different limits:

func setMemLimitFromCgroup() {
	if os.Getenv("GOMEMLIMIT") != "" {
		return // an operator set it explicitly, don't override
	}

	data, err := os.ReadFile("/sys/fs/cgroup/memory.max")
	if err != nil {
		return // not running under cgroup v2, leave the default
	}

	s := strings.TrimSpace(string(data))
	if s == "max" {
		return // no limit set on this cgroup
	}

	limit, err := strconv.ParseInt(s, 10, 64)
	if err != nil {
		return
	}

	// Leave headroom for goroutine stacks the GC hasn't scavenged yet,
	// cgo allocations, and anything else outside the Go allocator.
	debug.SetMemoryLimit(int64(float64(limit) * 0.9))
}

That's cgroup v2's memory.max; a v1 host exposes the same figure at /sys/fs/cgroup/memory/memory.limit_in_bytes.

If you'd rather not carry that branching logic around, automemlimit does the same detection with the version differences handled for you. That's the sensible choice for anything beyond a quick fix.

The 90% figure isn't a magic constant, it's a starting point. How much headroom you need depends on how much non-Go memory your process carries. A pure-Go HTTP service with small request bodies can probably push closer to 95%. Anything that wants more slack:

  • cgo;
  • large mmap'd buffers;
  • subprocesses spawned inside the same cgroup.

Leave GOGC switched on

It's tempting, once GOMEMLIMIT is doing the safety job, to set GOGC=off and let the memory limit be the only trigger. Resist that.

With GOGC disabled, the collector does nothing at all until usage approaches the GOMEMLIMIT ceiling. Then it has to run to keep the program alive, sits idle again as usage drops, and runs again as it climbs back up.

For a service that legitimately runs close to its memory limit as steady state, that means GC cycles firing back to back near the ceiling. The alternative is a cheap, well-spaced, ratio-paced collection doing most of the work earlier and more gradually.

So keep GOGC at its default, or something close to it, and let GOMEMLIMIT do exactly what it's named for. It's the backstop that only matters when a burst pushes the ratio-paced GC further than it was designed to go.

That combination, an unmodified GOGC plus a GOMEMLIMIT read from the real cgroup limit, is the pairing the Go team recommends. It turns "OOM-killed with a suspiciously small live heap" from a mystery into a graph you can watch.

If you want to see the pacer's decisions rather than infer them, GODEBUG=gctrace=1 prints a line per collection showing the heap goal alongside the live set. It's the fastest way to confirm the limit is actually being respected before you trust it in production.

← All articles