Articles / Go

704 words

  • go
  • concurrency
  • channels
  • pipelines
  • context
  • debugging

Why Your Go Pipeline Hangs When One Stage Returns Early

A Go pipeline works perfectly until the last stage hits an error and returns. Then the whole thing hangs, or quietly leaks goroutines, and nothing in the logs says why. The cause is simple: a send on a channel nobody is receiving from blocks for ever, and the stages upstream of the one that left have no idea it has gone.

A pipeline that looks fine

Here is a small three-part pipeline: a generator, a squarer, and a consumer that gives up when it sees a value it dislikes. The stages are tracked with a sync.WaitGroup so the function can wait for them before returning.

func run() error {
	var wg sync.WaitGroup
	src := make(chan int)
	sq := make(chan int)

	wg.Add(2)
	go func() {
		defer wg.Done()
		defer close(src)
		for i := 0; i < 1000; i++ {
			src <- i
		}
	}()
	go func() {
		defer wg.Done()
		defer close(sq)
		for v := range src {
			sq <- v * v
		}
	}()

	var err error
	for v := range sq {
		if v > 100 {
			err = fmt.Errorf("value too big: %d", v)
			break
		}
	}
	wg.Wait() // never returns
	return err
}

Run it and the consumer breaks out at 121. Then wg.Wait() sits there for ever.

What is actually stuck

The consumer has stopped reading from sq. The squarer is mid-loop, trying to send the next value on an unbuffered channel. Nobody will ever receive it, so it blocks. That in turn stops it reading from src, so the generator blocks too.

  • The consumer returned, so nobody receives from sq.
  • The squarer is stuck on sq <- v * v, so it never reaches its deferred wg.Done().
  • The generator is stuck on src <- i for the same reason, one stage further back.
  • wg.Wait() waits for two Done calls that cannot happen.

In this toy program the runtime notices that every goroutine is asleep and aborts with fatal error: all goroutines are asleep - deadlock!. In a real server there are always other goroutines running, so the detector stays quiet and you get a hung request or a slow leak instead.

Quick detour: why close() does not help

The instinct is to close(sq) from the consumer. Hang on, that makes things worse. Closing is a signal from the sender, and a send on a closed channel panics. The squarer owns sq, so only the squarer may close it.

The receiver needs a different way to say "stop", and the channel itself has no such operation. That is what a separate cancellation signal is for.

The fix: every send also listens for cancellation

Give the pipeline a context, and make every blocking send a select between the send and ctx.Done(). A small helper keeps that readable.

func send(ctx context.Context, out chan<- int, v int) bool {
	select {
	case out <- v:
		return true
	case <-ctx.Done():
		return false
	}
}

func run(parent context.Context) error {
	ctx, cancel := context.WithCancel(parent)
	defer cancel()

	var wg sync.WaitGroup
	src := make(chan int)
	sq := make(chan int)

	wg.Add(2)
	go func() {
		defer wg.Done()
		defer close(src)
		for i := 0; i < 1000; i++ {
			if !send(ctx, src, i) {
				return
			}
		}
	}()
	go func() {
		defer wg.Done()
		defer close(sq)
		for v := range src {
			if !send(ctx, sq, v*v) {
				return
			}
		}
	}()

	var err error
	for v := range sq {
		if v > 100 {
			err = fmt.Errorf("value too big: %d", v)
			break
		}
	}
	cancel() // before Wait, not just via defer
	wg.Wait()
	return err
}

Now the consumer cancels, both senders see ctx.Done() fire, return, run their deferred closes and call Done. wg.Wait() finishes.

The pitfall that catches people twice

Look at the cancel() before wg.Wait(). The defer cancel() at the top is not enough, because deferred calls run when the function returns, and the function is blocked in Wait. Cancel before you wait, otherwise you have rebuilt the original deadlock with extra steps.

The second trap is the receive side. A stage that does for v := range src is fine here because the upstream closes src on cancellation. But a stage that blocks on something else, such as a network read or a slow database call, will not notice the context at all. Pass the context into that call too, or the stage stays stuck until the call returns by itself.

Draining instead of cancelling

There is a cruder alternative: when the consumer wants to leave, it keeps receiving and throws the values away until the channel closes.

go func() {
	for range sq {
	}
}()

It works, and it needs no context. But the upstream stages do all the remaining work for nothing, and if the generator is a million-row query, that is a million rows computed to be discarded. Cancellation stops the work; draining only unblocks it. Use draining only when the remaining input is tiny and bounded.

Letting a library do the bookkeeping

Hand-rolling the context, the WaitGroup and the first-error capture is where mistakes creep in. golang.org/x/sync/errgroup bundles them: errgroup.WithContext gives you a context that is cancelled the moment any goroutine returns an error, and Wait returns that first error. The send-with-select rule still applies; the library cancels the context, but your stages must be listening for it.

A good habit for reviewing any pipeline: for each channel send, ask "if the receiver vanished right now, what unblocks this?" If the answer is "nothing", that line is your hang.

← All articles