Go's GOMEMLIMIT: Keeping the OOM Killer Away From a Container

A Go service sits in a pod with a 1GiB memory limit. It's been running fine for weeks, memory graphs look boring and flat, and then one afternoon it gets SIGKILLed by the kernel, restarts, and does it again an hour later. Nobody touched the deployment. There was no leak in the traditional sense: heap profiles taken minutes before the kill show a live set of maybe 150MiB. The garbage collector was doing exactly what it was told to do. What it was told to do just had nothing to do with the number that mattered.

That number is the cgroup memory limit, and until Go 1.19 the runtime had no idea it existed.

GOGC is a ratio, not a ceiling

Go's garbage collector has always been paced by GOGC, which defaults to 100. The pacer's rule is roughly: after a GC finishes and the live heap is measured, allow the heap to grow to double that size before triggering the next collection. It's a good rule if your only concern is CPU spent collecting versus memory spent holding garbage, because it scales the trigger to how much your program is actually using. A service with a 10MiB live set gets GC'd around 20MiB; a service with a 2GiB live set gets GC'd around 4GiB. Proportional, cheap, and completely blind to how much memory is actually available.

That blindness was fine when Go programs ran on machines they could see the whole of. It stops being fine the moment the program is confined to a slice of a machine by a cgroup, and a burst in allocations (a large request body, a spike in concurrent connections, a batch job that briefly holds more state than usual) can double the live heap well past the container's limit before the next GC cycle even fires, because the pacer's target was set relative to the previous, smaller heap. The kernel's OOM killer doesn't negotiate. It doesn't wait for a graceful GC pass. It sends SIGKILL to the process with the highest OOM score in the cgroup and moves on.

This is the memory-side equivalent of a problem Go has on the CPU side too, where GOMAXPROCS defaults to the host's core count rather than the container's CPU quota. Same root cause: the runtime was written for a world of whole machines, and cgroups quietly slice that world up underneath it.

What GOMEMLIMIT actually does

Go 1.19 added a second control: GOMEMLIMIT, a soft cap on the total memory the runtime is willing to use, set either as an environment variable (GOMEMLIMIT=900MiB) or programmatically with runtime/debug.SetMemoryLimit. "Soft" is doing real work in that sentence, so it's worth being precise about what it means.

GOMEMLIMIT doesn't replace GOGC's ratio-based trigger, it sits alongside it as an upper bound. Below the limit, GC behaves exactly as before: cheap, ratio-paced, mostly invisible. As usage climbs towards the limit, the pacer starts running more aggressive and more frequent collections to keep total memory under the ceiling, and it will lean on the runtime's memory scavenger to hand pages back to the OS sooner than it otherwise would. It is genuinely trying to respect the number you gave it.

But it can't guarantee it. The Go documentation is explicit that in extreme cases, such as an allocation rate the collector simply cannot keep pace with, the runtime will exceed the limit rather than deadlock the program. A soft limit that the runtime overshoots by design under load is a real constraint on how much protection this actually buys you, and it's the reason GOMEMLIMIT reduces the probability of an OOM kill rather than eliminating it.

There's also a coverage gap worth knowing about: the limit applies to memory the Go runtime tracks and manages directly, heap allocations, goroutine stacks, GC bookkeeping. Memory allocated outside the Go allocator, most obviously C memory allocated through cgo, isn't counted towards it, even though it counts fully against the cgroup limit that actually gets you killed. If your service does any cgo-heavy work, that's memory GOMEMLIMIT can't see coming.

Setting it from the actual container limit

The value you want isn't a constant baked in at build time, it's whatever limit the orchestrator handed the container at deploy time, minus some headroom for the memory GOMEMLIMIT doesn't track. Reading it straight from the cgroup at startup keeps the binary portable across environments with different limits:

func setMemLimitFromCgroup() {
	if os.Getenv("GOMEMLIMIT") != "" {
		return // an operator set it explicitly, don't override
	}

	data, err := os.ReadFile("/sys/fs/cgroup/memory.max")
	if err != nil {
		return // not running under cgroup v2, leave the default
	}

	s := strings.TrimSpace(string(data))
	if s == "max" {
		return // no limit set on this cgroup
	}

	limit, err := strconv.ParseInt(s, 10, 64)
	if err != nil {
		return
	}

	// Leave headroom for goroutine stacks the GC hasn't scavenged yet,
	// cgo allocations, and anything else outside the Go allocator.
	debug.SetMemoryLimit(int64(float64(limit) * 0.9))
}

That's cgroup v2's memory.max; a v1 host exposes the same figure at /sys/fs/cgroup/memory/memory.limit_in_bytes. If you'd rather not carry that branching logic around, automemlimit does the same detection with the version differences handled for you, which is the sensible choice for anything beyond a quick fix.

The 90% figure isn't a magic constant, it's a starting point. How much headroom you actually need depends on how much non-Go memory your process carries: a pure-Go HTTP service with small request bodies can probably push closer to 95%, while anything doing cgo, large mmap'd buffers, or spawning subprocesses inside the same cgroup wants more slack.

Don't turn GOGC off to compensate

It's tempting, once GOMEMLIMIT is doing the safety job, to set GOGC=off and let the memory limit be the only trigger. Resist that. With GOGC disabled, the collector does nothing at all until usage approaches the GOMEMLIMIT ceiling, at which point it has to run to keep the program alive, then sits idle again as usage drops, then runs again as it climbs back up. For a service that legitimately runs close to its memory limit as steady state, that's a GC cycle firing back to back near the ceiling instead of a cheap, well-spaced ratio-paced collection doing most of the work earlier and more gradually.

Keep GOGC at its default, or something close to it, and let GOMEMLIMIT do exactly what it's named for: act as the backstop that only matters when a burst pushes the ratio-paced GC further than it was designed to go. That combination, an unmodified GOGC plus a GOMEMLIMIT read from the real cgroup limit, is the pairing the Go team actually recommends, and it's the one that turns "OOM-killed with a suspiciously small live heap" from a mystery into a graph you can watch.

If you want to see the pacer's decisions rather than infer them, GODEBUG=gctrace=1 prints a line per collection showing the heap goal alongside the live set, and it's the fastest way to confirm the limit is actually being respected before you trust it in production.