Blog / Go

905 words

  • go
  • net-conn
  • timeouts
  • networking
  • tcp
  • debugging

A Read Deadline Set Once Will Kill a Long-Lived Connection

Connections that are chattering away happily get cut off exactly 30 seconds after they opened, with an i/o timeout error, mid-conversation. The TCP handler sets a read deadline of 30 seconds, loops reading lines, and passes every test you are likely to write. Only production notices.

func handle(c net.Conn) {
	defer c.Close()
	c.SetReadDeadline(time.Now().Add(30 * time.Second)) // set once, before the loop

	r := bufio.NewReader(c)
	for {
		line, err := r.ReadString('\n')
		if err != nil {
			log.Printf("read: %v", err) // read tcp 10.0.0.5:9000->10.0.0.9:51234: i/o timeout
			return
		}
		process(line)
	}
}

The name SetReadDeadline does not help here, and neither does the mental model most of us bring from other languages. In C or Python, a "timeout" is usually a duration: wait up to N seconds for this call.

In Go, a deadline is an absolute time.Time. It is a point on the clock, and once the clock passes it, every read on that connection fails, now and in future, until you set a new one.

The docs spell out the idle-timeout pattern

The net.Conn documentation is clearer than the method name. It says three useful things:

  • A deadline "applies to all future and pending I/O, not just the immediately following call to Read or Write".
  • After a deadline has been exceeded, the connection can be refreshed by setting a deadline in the future.
  • "An idle timeout can be implemented by repeatedly extending the deadline after successful Read or Write calls."

So the fix is what the docs describe: move the call inside the loop.

const idle = 30 * time.Second

func handle(c net.Conn) {
	defer c.Close()
	r := bufio.NewReader(c)
	for {
		if err := c.SetReadDeadline(time.Now().Add(idle)); err != nil {
			return
		}
		line, err := r.ReadString('\n')
		if err != nil {
			var ne net.Error
			if errors.As(err, &ne) && ne.Timeout() {
				log.Printf("%s idle for %s, closing", c.RemoteAddr(), idle)
			} else if !errors.Is(err, io.EOF) {
				log.Printf("read: %v", err)
			}
			return
		}
		process(line)
	}
}

Each iteration now says "you have 30 seconds from now to give me a full line". A client that sends something every 29 seconds lives forever; one that goes quiet for 31 gets closed. That is an idle timeout, which is usually what people meant in the first place.

You can see it without a network

net.Pipe implements deadlines too, so the behaviour is easy to pin down in a test with no sockets and no flakiness from the OS:

func TestDeadlineIsAbsolute(t *testing.T) {
	client, server := net.Pipe()
	defer client.Close()
	defer server.Close()

	server.SetReadDeadline(time.Now().Add(50 * time.Millisecond))
	time.Sleep(100 * time.Millisecond) // "healthy" gap, deadline passes anyway

	go client.Write([]byte("hello\n")) // data is available, but too late

	buf := make([]byte, 16)
	_, err := server.Read(buf)
	if !errors.Is(err, os.ErrDeadlineExceeded) {
		t.Fatalf("want deadline exceeded, got %v", err)
	}

	// Refreshing the deadline makes the same connection usable again.
	server.SetReadDeadline(time.Now().Add(time.Second))
	if _, err := server.Read(buf); err != nil {
		t.Fatalf("read after refresh: %v", err)
	}
}

Note the errors.Is against os.ErrDeadlineExceeded. Timeout errors from the net package wrap it, so you have two good options:

  • test for it directly with errors.Is;
  • use the older net.Error interface and its Timeout() method, as in the handler above.

Either is fine; just do not match on the string "i/o timeout".

Where you put the reset decides what you are timing

Actually, this bit is the interesting part. Resetting the deadline "each loop" is not one policy, it is a family of them, depending on where the call sits.

  • Before ReadString, as above: the client gets 30 seconds to deliver a whole line. Because the deadline is absolute, a client dribbling one byte every 10 seconds still gets cut off after 30 seconds total for that line.
  • Inside a wrapper whose Read resets the deadline every call: the client gets 30 seconds per read. A single byte every 29 seconds keeps the connection open indefinitely, and a line that never ends holds a goroutine and a buffer for as long as they like.

The second is the classic slow-drip attack. It is why "reset on every read" wrappers, which look tidy and appear in plenty of snippets, deserve suspicion.

For anything facing the internet:

  • prefer a deadline per message (or per request);
  • consider a hard cap on total session time as well as the idle one;
  • pair it with a size limit on the line, since a per-message deadline does not stop a fast client sending a gigabyte of newline-free data.

Smaller traps that bite later

Reads can return data and an error together

When a deadline fires partway through, Read may return n > 0 with a timeout error. Process the n bytes before you look at the error, or you drop data. The bufio code above sidesteps that, since ReadString hands back the partial line alongside the error; whether you want to use it is your call.

Write deadlines are separate, and sometimes fatal

SetReadDeadline and SetWriteDeadline are independent; SetDeadline sets both. Same rule applies: set the write deadline before each write, or a slow reader on the other end will hold your goroutine indefinitely once the send buffer fills.

The catch is TLS. For *tls.Conn, the documentation says that after a write has timed out the TLS state is corrupt and all future writes return the same error. With plain TCP you can refresh and carry on; with TLS a write timeout means close the connection.

A stale deadline follows the connection around

If you set a deadline and then hand the net.Conn to something else (a protocol upgrade, a different handler, a pool), that deadline is still armed. The next component sees spurious timeouts that look like network trouble. Clear it with the zero value: c.SetDeadline(time.Time{}).

Deadlines can be moved from another goroutine

Quick detour, because this one is genuinely handy. Deadlines apply to pending I/O, so setting one in the past from another goroutine wakes up a blocked Read straight away with a timeout error. That is the standard way to interrupt a read on shutdown without closing the connection first, so the reading goroutine can flush and exit cleanly.

Ask what event your deadline is measuring

If a SetReadDeadline call is not inside the loop that performs the reads, ask what event it is measuring:

  • A deadline set once means "this connection has N seconds to live".
  • A deadline set per message means "each message has N seconds to arrive".

Only the second one is an idle timeout, and only if you actually wrote it that way.

← Back to the blog