Articles / Go

  • go
  • concurrency
  • mutex
  • channels
  • deadlock
  • debugging

A Go Channel Send Inside a Mutex Is a Deadlock Waiting

A channel send can block for as long as the receiver likes. A mutex held across that send is therefore held for as long as the receiver likes. Put those together and one slow consumer can freeze every goroutine that wants the lock, or deadlock the program if the consumer needs that same lock to make progress.

It looks harmless in review, because the send is one line inside a perfectly reasonable critical section.

The innocent-looking broker

Here is a tiny publish/subscribe broker. The mutex protects the map of subscribers, and Publish sends to each one while it iterates.

type Broker struct {
	mu   sync.Mutex
	next int
	subs map[int]chan string
}

func NewBroker() *Broker {
	return &Broker{subs: make(map[int]chan string)}
}

func (b *Broker) Subscribe() (int, <-chan string) {
	b.mu.Lock()
	defer b.mu.Unlock()
	id := b.next
	b.next++
	ch := make(chan string)
	b.subs[id] = ch
	return id, ch
}

func (b *Broker) Unsubscribe(id int) {
	b.mu.Lock()
	defer b.mu.Unlock()
	if ch, ok := b.subs[id]; ok {
		delete(b.subs, id)
		close(ch)
	}
}

func (b *Broker) Publish(msg string) {
	b.mu.Lock()
	defer b.mu.Unlock()
	for _, ch := range b.subs {
		ch <- msg // blocks until the receiver is ready
	}
}

With fast receivers this works for months. The bug only shows up when someone is slow, which is exactly when you least want to be debugging it.

Failure one: everyone queues behind the slow receiver

Suppose one subscriber is busy writing to a slow network client. Publish reaches that subscriber, blocks on the send, and still holds the mutex.

  • A new Subscribe call waits for the lock.
  • Every Unsubscribe waits for the lock.
  • A second Publish waits for the lock.

Nothing is technically deadlocked yet; it is just stalled for as long as that one receiver takes. In a server, that looks like request handlers hanging on an innocent-looking Subscribe call.

Failure two: the receiver needs the lock

Now the proper deadlock. A consumer reads a message, decides it has had enough, and calls Unsubscribe before reading again.

id, ch := b.Subscribe()
go func() {
	for msg := range ch {
		if msg == "quit" {
			b.Unsubscribe(id) // needs b.mu
			return
		}
	}
}()

If a second publish is in flight, Publish holds b.mu and is blocked sending to this consumer. The consumer is blocked waiting for b.mu in Unsubscribe, so it will never receive again. Each side waits for the other, and nobody wins.

Go's runtime only reports "all goroutines are asleep" when every goroutine is stuck. In a real server other goroutines keep running, so you get silence and a slowly growing pile of blocked handlers instead.

Quick detour: why a buffer only hides it

Giving the channels a buffer, say make(chan string, 16), makes the problem disappear in testing. The send no longer blocks until 16 messages are queued.

That is a delay, not a fix. A receiver that stops reading entirely will fill any buffer eventually, and then you are back to blocking under the lock, now with a much harder reproduction.

The fix: decide under the lock, send outside it

The mutex exists to protect the map. It does not need to protect the sends. So copy what you need under the lock, release it, then send.

Moving the send out of the lock creates a new problem, though. Another goroutine can now call Unsubscribe and close(ch) while Publish is about to send, and a send on a closed channel panics. The answer is to stop closing the data channel from the unsubscribing side and use a separate done channel as the signal.

type sub struct {
	ch   chan string
	done chan struct{}
}

type Broker struct {
	mu   sync.Mutex
	next int
	subs map[int]*sub
}

func NewBroker() *Broker {
	return &Broker{subs: make(map[int]*sub)}
}

func (b *Broker) Subscribe() (int, <-chan string) {
	b.mu.Lock()
	defer b.mu.Unlock()
	id := b.next
	b.next++
	s := &sub{ch: make(chan string, 16), done: make(chan struct{})}
	b.subs[id] = s
	return id, s.ch
}

func (b *Broker) Unsubscribe(id int) {
	b.mu.Lock()
	s, ok := b.subs[id]
	delete(b.subs, id)
	b.mu.Unlock()
	if ok {
		close(s.done)
	}
}

func (b *Broker) Publish(msg string) {
	b.mu.Lock()
	snapshot := make([]*sub, 0, len(b.subs))
	for _, s := range b.subs {
		snapshot = append(snapshot, s)
	}
	b.mu.Unlock()

	for _, s := range snapshot {
		select {
		case s.ch <- msg:
		case <-s.done:
		default:
			// receiver is full: drop rather than stall everyone
		}
	}
}

The lock is now held for a map copy and nothing else. Unsubscribe can run at any time, including from inside the consumer, because Publish is never waiting on the lock while blocked on a send.

The default case is a policy decision

Dropping messages for a full receiver is one choice, and it suits things like live status updates. It is wrong for anything where loss matters.

The alternatives are:

  • Drop the oldest message for that subscriber and keep the newest.
  • Disconnect the slow subscriber by calling Unsubscribe on it.
  • Block the publisher, but with a timeout or a context, and never while holding the lock.

Whichever you pick, write it down in the code. The original version made this decision implicitly, by letting the slowest receiver set the pace for the whole program.

Spotting it elsewhere

The pattern is wider than brokers. Any time you hold a lock and do something that can wait on another goroutine, you have given that goroutine a say over your lock. Channel sends are the obvious case, but the same applies to:

  • Calling a user-supplied callback under the lock.
  • Doing network or disk I/O inside the critical section.
  • Calling another method on the same type that also takes the lock.

A useful review habit: for each Lock(), read down to the matching Unlock() and ask whether anything in between can wait on someone else. If it can, move it out. The race detector will not flag any of this, since there is no data race, only a lock held for too long.

The goroutine dump is your friend when it does happen. Send SIGQUIT to the process, or fetch /debug/pprof/goroutine?debug=2, and look for one goroutine in chan send and several in sync.Mutex.Lock. That pair is the signature.