Why Your Go Worker Pool Deadlocks: An Unbuffered Channel Post-Mortem

Here is a worker pool that worked fine in testing, passed code review, and then locked up in production the moment someone fed it more than a handful of jobs:

func process(jobs []Job) []Result {
	jobCh := make(chan Job)
	resultCh := make(chan Result)

	var wg sync.WaitGroup
	for i := 0; i < 4; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for job := range jobCh {
				resultCh <- doWork(job)
			}
		}()
	}

	for _, j := range jobs {
		jobCh <- j
	}
	close(jobCh)

	wg.Wait()
	close(resultCh)

	var results []Result
	for r := range resultCh {
		results = append(results, r)
	}
	return results
}

Run it with three jobs and it hangs forever. The runtime eventually notices and prints:

fatal error: all goroutines are asleep - deadlock!

No panic in doWork, no race, nothing you'd catch with go vet. Just silence, then a crash. It's worth working through exactly why, because the bug is structural rather than a typo, and the same shape reappears in a lot of hand-rolled pool code.

What an unbuffered channel actually promises

An unbuffered channel has no storage. A send on it does not complete until some goroutine is doing a matching receive at that exact moment. It's a rendezvous, not a queue. So resultCh <- doWork(job) is not "hand this result off and carry on"; it's "block here until someone reads it". That's fine in isolation. The problem is who is meant to be doing the reading.

Tracing the wedge

Walk through process with, say, three jobs and four workers:

1. The main goroutine starts sending jobs into jobCh. Workers pick them up as fast as they're sent, since jobCh is also unbuffered and a worker is usually waiting on it.
2. A worker finishes doWork and tries resultCh <- .... Nobody is receiving from resultCh yet, because the main goroutine is still in the loop sending jobs, not in the loop reading results. The worker blocks.
3. If enough workers finish before the main goroutine has sent all jobs, every worker ends up parked on resultCh <- .... None of them can go back to range jobCh to accept the next job.
4. The main goroutine is stuck trying to send the next job into jobCh, because there's no worker free to receive it.

Now everyone is asleep waiting on someone else: the main goroutine wants a worker to receive a job, the workers want the main goroutine to receive a result, and the main goroutine won't get to the result-reading loop until it has finished sending every job. That's the deadlock, and it's a genuine circular wait, not a timing fluke. Whether it triggers depends only on whether a worker manages to produce a result before the job-sending loop finishes, which is why it can pass a quick smoke test with two jobs and fail immediately with twenty.

The fix: never let dispatch and collection block on each other

The rule is simple once you see the trap: dispatching and collecting have to run concurrently, not sequentially, whenever the channel connecting them is unbuffered. The standard shape drains results in a separate goroutine and uses sync.WaitGroup purely to know when it is safe to close the results channel:

func process(jobs []Job) []Result {
	jobCh := make(chan Job)
	resultCh := make(chan Result)

	var wg sync.WaitGroup
	for i := 0; i < 4; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for job := range jobCh {
				resultCh <- doWork(job)
			}
		}()
	}

	go func() {
		for _, j := range jobs {
			jobCh <- j
		}
		close(jobCh)
	}()

	go func() {
		wg.Wait()
		close(resultCh)
	}()

	var results []Result
	for r := range resultCh {
		results = append(results, r)
	}
	return results
}

The main goroutine's only job now is to range over resultCh, which means there is always someone available to receive as soon as a worker sends. Dispatch happens on its own goroutine, so it can block on jobCh <- without stopping anyone reading results. The wg.Wait() that closes resultCh also runs on its own goroutine, because calling it inline in the main goroutine (right after the dispatch loop, as in the original) would once again make result-collection wait for dispatch to finish first.

Three goroutines doing one thing each, none of them able to block the others out of the loop. That's the whole fix.

Why a buffered channel isn't really a fix

A common first instinct is to make resultCh buffered:

resultCh := make(chan Result, 100)

This will genuinely make the deadlock disappear for a workload of 100 jobs or fewer, which is exactly what makes it a dangerous fix. It doesn't remove the circular dependency between dispatch and collection, it just gives the buffer enough slack to absorb it up to a point. Feed the same code 101 jobs, or run it on a slower path where doWork finishes faster relative to dispatch, and it wedges again in exactly the same way, just later. You've turned a reliable, fast-failing bug into an intermittent, load-dependent one, which is worse: it'll pass CI, pass staging, and then show up in production traffic six months from now when volume finally crosses the buffer size. If you do want a buffer for throughput reasons once the structural fix is in place, that's a legitimate tuning decision; using one to paper over a blocking cycle is not.

The deadlock detector only catches the total case

One thing worth knowing about the Go runtime's "all goroutines are asleep" detector: it only fires when literally every goroutine in the process is blocked. If your worker pool is embedded in a larger service with an HTTP server or other goroutines still running, this exact same circular wait will not crash the process at all. The pool goroutines will simply sit there forever, leaked, quietly consuming a few kilobytes each and never returning. You'll see it as a slow memory climb and a request that never completes, with nothing in the logs pointing at channels. go tool pprof on the goroutine profile, or just GODEBUG=schedtrace, is usually how people actually find this kind of thing in the wild, because by the time it's visible in production there's no standalone reproduction to run the deadlock detector against.