Phone:

Hidden from the page source until you click: friction against scrapers, not a guarantee.

Email:

[email protected]

Category:

Go

Tags:
  • go
  • net-dialer
  • happy-eyeballs
  • ipv6
  • networking
  • performance

Go's net.Dialer and Happy Eyeballs: Why Dual-Stack Connects Lag

A service I was looking at had a peculiar latency profile: every fresh outbound connection to one particular host took a suspiciously round 300ms longer than the same connection made with curl -4. Not 250, not 340. Three hundred. When a number is that tidy, somebody hard-coded it, and in this case the somebody was the Go standard library.

What Dial does with a hostname

Give net.Dialer a name like example.com:443 and it resolves the name first, which may give you several addresses in both families. The resolver returns them sorted (roughly RFC 6724 rules, so IPv6 usually comes first if you have a global IPv6 address). The dialer then splits that list into two groups: "primaries", meaning addresses of the same family as the first entry, and "fallbacks", meaning the other family.

Primaries start connecting straight away. If nothing has succeeded after FallbackDelay, the fallbacks start too, in parallel, and whichever TCP handshake completes first wins. The loser is closed. This is the RFC 6555 style of "Happy Eyeballs", and the docs say the default delay is 300ms when the field is zero, with a negative value disabling the whole thing. (Dialer.DualStack is deprecated; fast fallback has been on by default for years.)

The important detail is what triggers the fallback early. If the primary attempt fails outright, say with a connection refused or a "network is unreachable" error, the fallback timer is reset to zero and the IPv4 attempt starts immediately. So a host with a broken but loudly broken IPv6 path costs you almost nothing. The 300ms only bites when IPv6 fails silently: packets are dropped, no RST, no ICMP error, and the SYN just vanishes. That is precisely what a lot of half-configured firewalls, tunnels and container networks do.

Watching it happen

You can see every socket attempt with the dialer's Control hook, which runs once per socket before the connect. This is Unix-flavoured because of the syscall.RawConn signature's usual companions, but the code itself compiles on Windows too.

package main

import (
	"context"
	"fmt"
	"net"
	"os"
	"syscall"
	"time"
)

func main() {
	start := time.Now()
	ms := func() int64 { return time.Since(start).Milliseconds() }

	d := net.Dialer{
		Timeout: 5 * time.Second,
		Control: func(network, address string, _ syscall.RawConn) error {
			fmt.Printf("%5dms  try %s %s\n", ms(), network, address)
			return nil
		},
	}

	conn, err := d.DialContext(context.Background(), "tcp", os.Args[1])
	if err != nil {
		fmt.Printf("%5dms  failed: %v\n", ms(), err)
		return
	}
	defer conn.Close()
	fmt.Printf("%5dms  connected to %s\n", ms(), conn.RemoteAddr())
}

Run that against a name whose AAAA record points at an address that drops traffic (a documentation prefix such as 2001:db8:: in a test zone works, if your network has no route it will fail loudly, so you may need a deliberate drop rule in nftables). You should see something like this:

    0ms  try tcp6 [2001:db8::10]:443
  301ms  try tcp4 192.0.2.10:443
  318ms  connected to 192.0.2.10:443

That 301ms gap is the whole story. The IPv4 handshake itself took 17ms.

Nothing is remembered

Here is the bit that surprised me. Go does not remember that IPv6 to this host was dead a moment ago. Every new dial re-runs the race from scratch, so every new connection pays the full delay. There is no per-destination cache of "v4 won last time".

For an http.Client that is mostly hidden by connection pooling: you pay once, then keep-alive reuses the connection. But anything that dials a lot (a client without keep-alive, a service that talks to many short-lived backends, a health checker, a load test with a fresh connection per request) pays 300ms every single time. If you have ever wondered why your p50 has a floor you cannot explain, check this before you go profiling the TLS stack.

Knobs, and what they cost

The blunt options:

  • Lower FallbackDelay, say to 50ms. The RFC 8305 revision of Happy Eyeballs suggests a connection attempt delay in the low hundreds of milliseconds with a sensible floor, so going tiny means you race more often and open more doomed connections. On a healthy network it wastes a few SYNs. On a slow mobile link you may pick IPv4 when IPv6 would have won in 80ms.
  • Disable it with a negative FallbackDelay. Now Go tries addresses strictly one after another, and a black-holed IPv6 address burns its full share of your timeout before IPv4 gets a turn. That is usually worse, not better.
  • Force a family by dialling "tcp4" instead of "tcp". Fine for a service you know is IPv4-only. Rude if the whole point of your tool is to work on IPv6-only networks, and you will find out about that at the worst time.

Fixing the actual IPv6 path, or removing the bad AAAA record, beats all three. But when the record belongs to someone else, a dialer tweak is what you have.

Timeout is split, not shared

Actually, this next bit is the one that catches people in production. Dialer.Timeout covers the whole dial including DNS, but within an address family the remaining time is divided across the addresses still to try. With a 10 second timeout and three dead IPv6 addresses, the first attempt gets roughly a third of what is left, not all of it. There is a floor: each attempt gets at least 2 seconds where the remaining time allows, and never more than what is left.

The two families run in parallel and share the same overall deadline, so a set of black-holed IPv6 addresses does not eat the IPv4 budget. It does mean the total time to failure is not simply the timeout multiplied by anything; it is the timeout, cut into slices.

Which is also why the default http.DefaultTransport dialer, with its 30 second timeout, is generous while a hand-rolled 2 second one can make a host with four AAAA records fail in ways that look random. If you set Timeout tightly, count the addresses.

Configuring it for an HTTP client

tr := http.DefaultTransport.(*http.Transport).Clone()
tr.DialContext = (&net.Dialer{
	Timeout:       5 * time.Second,
	KeepAlive:     30 * time.Second,
	FallbackDelay: 100 * time.Millisecond,
}).DialContext

client := &http.Client{Transport: tr, Timeout: 15 * time.Second}

Cloning the default transport keeps the proxy settings, HTTP/2 support and idle connection limits that you would otherwise lose by building a bare http.Transport{}. Only the dialer changes.

One last caveat: I have described Go's behaviour from its documented RFC 6555 semantics and from reading the dialer code, not from a promise in the language spec, so the exact numbers (300ms, the 2 second floor) are implementation details that could shift between releases. Check the net.Dialer documentation for the release you ship, and measure with the Control hook rather than trusting me.

← Back to the blog