Articles / Go

869 words

  • go
  • mutex
  • concurrency
  • starvation
  • scheduling
  • debugging

Go Mutexes Aren't FIFO: What Starvation Mode Really Does

A lot of Go code quietly assumes that goroutines waiting on a sync.Mutex get it in the order they asked. They don't. In its default mode the mutex lets a goroutine that is already running jump ahead of one that has been asleep in the queue, and that can leave a waiter unlucky for a surprisingly long time.

Go does have a safety net, called starvation mode. It is worth knowing exactly what it promises, because it is narrower than "the mutex is fair".

Normal mode lets newcomers barge in

When you call Unlock, the mutex wakes the goroutine at the front of the wait queue. But waking is not owning. The woken goroutine has to compete with any goroutine already on a CPU and calling Lock right now.

The newcomer has a real advantage: it is already running. The woken waiter still has to be scheduled. So it often loses, goes back to the front of the queue, and sleeps again.

This is deliberate. Handing the lock to a sleeping goroutine costs a context switch, and letting a running goroutine carry on is much cheaper. Normal mode trades fairness for throughput, and most of the time that is the right trade.

Watching it happen

Here is a small program. Four goroutines hammer one mutex, each holding it for about 50 microseconds, and each counts how many times it got in over two seconds.

package main

import (
	"fmt"
	"sync"
	"time"
)

func main() {
	var (
		mu     sync.Mutex
		wg     sync.WaitGroup
		counts [4]int
		stop   = time.Now().Add(2 * time.Second)
	)
	for i := range counts {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for time.Now().Before(stop) {
				mu.Lock()
				counts[i]++
				for t := time.Now(); time.Since(t) < 50*time.Microsecond; {
				}
				mu.Unlock()
			}
		}()
	}
	wg.Wait()
	fmt.Println(counts)
}

The numbers vary by machine and run, so I won't quote any. What you should look for is whether the four counts are roughly equal. They are usually close, but not identical, and the spread is the point: nothing here guarantees equal shares, let alone arrival order.

Note that the loop variable is per-iteration since Go 1.22, so capturing i is safe in the code above.

Where starvation mode steps in

The mutex keeps an eye on its waiters. If a waiter fails to get the lock for more than 1 millisecond, the mutex switches to starvation mode. This has been the behaviour since Go 1.9.

In starvation mode the rules flip:

  • Unlock hands ownership directly to the waiter at the front of the queue.
  • New arrivals do not try to grab the lock, and do not spin.
  • New arrivals join the back of the queue instead.

So the unlucky waiter is guaranteed to get its turn. The mutex then drops back to normal mode when a waiter is the last in the queue, or when it acquired the lock in under 1 millisecond.

Quick detour: why is that threshold in time, not count?

Hang on, why 1ms rather than "after N failed attempts"? Because what hurts is latency, not attempts. A waiter that lost twice but waited 5 microseconds is fine. A waiter that lost once to a long critical section is not.

It also explains something odd you may see in profiles: critical sections held for longer than a millisecond push the mutex into starvation mode almost permanently. Every handoff then involves a wake-up and a context switch, and throughput drops. Long critical sections are what turn fairness from free into expensive.

What the guarantee does not cover

Starvation mode bounds how long a waiter can wait, roughly. It is not a promise about order from the caller's point of view. A few things it does not give you:

  • Strict FIFO in normal mode, which is where the mutex spends most of its life.
  • Any ordering between goroutines that call Lock at almost the same instant, since "first" is not well defined there.
  • A way to ask which mode the mutex is in; it is internal state.

So code that uses a mutex as a ticket queue, such as "process events in the order goroutines arrived", has a bug that will not show up in testing and will show up under load.

RWMutex adds its own twist

A sync.RWMutex has a related rule: once a writer is blocked in Lock, new readers are blocked too. That stops readers starving the writer. The documentation notes the consequence: you must not take a read lock recursively, because a writer arriving between the two RLock calls will deadlock you.

If you need order, build it

When arrival order matters, do not lean on the mutex. Use something that has ordering as part of its contract:

  1. A buffered channel as a queue, with a single goroutine consuming it.
  2. A ticket counter taken under the lock, with each waiter blocking on its own channel until its turn.
  3. A semaphore such as golang.org/x/sync/semaphore, whose docs describe FIFO waiting.

The first is almost always the simplest. A single owner goroutine also removes the lock entirely, which is usually a better design anyway.

If the problem is merely that one goroutine is slow to get in, shorten the critical section first. Do the expensive work outside the lock, copy what you need, release, then carry on. That fixes starvation at the cause rather than relying on the 1ms rescue.

The official description of the type is on pkg.go.dev; the long comment in the runtime source explains the two modes in more detail than the public docs do.

← All articles