Articles / Go

757 words

  • go
  • channels
  • concurrency
  • panic
  • race-detector
  • debugging

Why Your Go Channel Close Panics Only Under Load

Your tests pass, staging is fine, and then production throws panic: send on closed channel at 2am during a traffic spike. The code looks correct because it checks whether the channel is closed before sending. That check is exactly the bug.

The code that looks safe

Here is a tiny event bus. Publishers check a flag, then send. Shutdown sets the flag, then closes the channel.

type Bus struct {
	mu     sync.Mutex
	closed bool
	ch     chan Event
}

func (b *Bus) Publish(ev Event) {
	b.mu.Lock()
	closed := b.closed
	b.mu.Unlock()
	if closed {
		return
	}
	b.ch <- ev // panics if Close ran since the check
}

func (b *Bus) Close() {
	b.mu.Lock()
	b.closed = true
	b.mu.Unlock()
	close(b.ch)
}

Every access to closed is locked, so it feels thread-safe. But the lock is released before the send. Between "I saw it was open" and "I sent", another goroutine can run the whole of Close.

Why it only bites under load

The window is tiny: a few nanoseconds between the check and the send. With one publisher and a quiet system, nobody lands in it. With hundreds of publishers on several cores, and a send that blocks because the buffer is full, the window gets much wider.

That last point matters most. If b.ch is full, the publisher parks inside the send, possibly for a long time. If Close runs while it is parked, the sender wakes up to a closed channel and panics. A full buffer turns a nanosecond race into a seconds-long one.

The rules the runtime enforces

  • Sending on a closed channel panics, including a send that was already blocked when the close happened.
  • Closing a closed channel panics.
  • Closing a nil channel panics.
  • Receiving from a closed channel never panics: it drains the buffer, then returns zero values forever.

So the asymmetry is the whole story. Receivers can be careless; senders and closers must agree on who goes first. The language gives you no way to ask "is this closed?" safely, because the answer would be stale before you used it.

Quick detour: a select can panic too

Hang on, this one surprised me. A select with a send case on a closed channel does not skip that case as "not ready". The case counts as ready, and if it is chosen, it panics. So wrapping the send in a select with a default does not save you either. The only safe option is to never let a send race a close.

Fix one: do not close the data channel

The lazy fix is often the best. Leave the data channel open for the garbage collector to clean up, and signal shutdown with a separate channel that is only ever closed, never sent on.

type Bus struct {
	ch   chan Event
	done chan struct{}
	once sync.Once
}

func NewBus() *Bus {
	return &Bus{ch: make(chan Event, 64), done: make(chan struct{})}
}

func (b *Bus) Publish(ev Event) bool {
	select {
	case b.ch <- ev:
		return true
	case <-b.done:
		return false
	}
}

func (b *Bus) Close() {
	b.once.Do(func() { close(b.done) })
}

func (b *Bus) Run(handle func(Event)) {
	for {
		select {
		case ev := <-b.ch:
			handle(ev)
		case <-b.done:
			return
		}
	}
}

Nothing can panic here. Blocked publishers wake up when done closes and return false. The sync.Once makes Close safe to call twice, which matters because shutdown paths love to run twice.

One caveat: when both cases are ready, select picks at random, so an event published around shutdown may be accepted into the buffer and never handled. If that loss matters, drain b.ch after Run returns, or decide it is acceptable and document it.

Fix two: one closer that waits for all senders

Sometimes you do need to close the data channel, because consumers use for range. The rule then is that the closer must be something that knows every sender has finished.

func fanIn(sources []func(chan<- int)) <-chan int {
	out := make(chan int)
	var wg sync.WaitGroup
	for _, src := range sources {
		wg.Add(1)
		go func() {
			defer wg.Done()
			src(out)
		}()
	}
	go func() {
		wg.Wait()
		close(out)
	}()
	return out
}

Senders never close. A single goroutine closes after wg.Wait(), so by construction no send can follow the close. This only works when the sender set is fixed up front; wg.Add must happen before wg.Wait can return.

Fix three: if there is one sender, it closes

The old Go proverb is that the sending side closes, never the receiving side. With exactly one sending goroutine, that is simply correct: it finishes its loop, then closes. Receivers that want to stop early cancel a context, which the sender watches. The trouble starts when "the sender" quietly became many.

Catching it before production

The race detector helps here. It treats close as a write to the channel and a send as a read, so when the two overlap in a test it reports a data race instead of waiting for the panic. Run a stress test with many publishers and a tiny buffer:

go test -race -count=50 -run TestBusShutdown ./...

Make the test close the bus while publishers are blocked on a full buffer. That is the exact condition production created for you. If the test never fills the buffer, it will probably never fail, which is how this bug survived review in the first place.

Recovering from the panic with defer recover() in Publish does technically work. I would not: it hides a design where nobody owns the close, and the next person will copy the pattern.

← All articles