Articles / Networking

1303 words

  • reverse-proxy
  • http
  • http2
  • connection-pooling
  • backpressure
  • networking

Your Reverse Proxy Can Create More Connections Than Clients

A reverse proxy can have fewer clients than upstream TCP connections. That sounds backwards until you remember that a client connection carries requests, while an upstream connection carries whatever the proxy's protocol and pooling rules allow.

One browser connection can turn into several backend sockets. HTTP/2 makes this especially easy. A browser may send many concurrent streams over one connection to the proxy, while the proxy speaks HTTP/1.1 to the application and needs one upstream connection per active request.

Count requests, not just clients

Suppose 100 clients connect to the proxy. If each client uses HTTP/2 and sends 20 concurrent requests, the proxy may be handling 2,000 requests despite having only 100 downstream connections.

If the upstream connection is HTTP/1.1, those requests cannot all share one socket at the same time. The proxy therefore needs a pool of upstream connections, potentially approaching 2,000 sockets if the application is slow enough and the pool is allowed to grow.

The rough shape is:

  • downstream connections: how many clients are connected;
  • downstream streams: how many requests are active on those connections;
  • upstream connections: how many backend sockets are needed by the proxy protocol;
  • upstream streams: how many requests those backend sockets can carry concurrently.

These are different quantities. Monitoring only the first one can leave you wondering why the proxy has run out of file descriptors while the client connection count looks perfectly respectable.

HTTP/2 changes the shape of the problem

With HTTP/1.1, a client usually needs several connections to achieve parallelism. With HTTP/2, many requests can share one connection as independent streams. That is efficient for the client, but it moves the concurrency pressure into the proxy.

Now add protocol translation. The client speaks HTTP/2 to the edge, the edge speaks HTTP/1.1 to the application, and every simultaneous request may need its own upstream socket. The proxy has become a fan-out device even though the client has not opened more connections.

Quick detour: HTTP/2 upstream does not automatically solve everything. One HTTP/2 connection can carry many streams, but the proxy still has limits for stream counts, flow control, request queues and backend capacity. Fewer sockets can mean more work concentrated on each socket.

Idle pools count as connections too

A proxy does not only hold sockets for requests currently being served. It often keeps idle upstream connections ready for the next request, because creating a TCP connection and performing TLS can be more expensive than reusing one.

Idle does not mean free. An idle connection still consumes a file descriptor, kernel memory, proxy bookkeeping and sometimes a slot in a load balancer or firewall table.

A pool can also be spread across several upstream addresses. If the service has ten backend IP addresses and the proxy keeps 20 idle connections per address, the apparent limit is not 20. It may be 200 connections for that upstream group, before active requests are counted.

Pool settings often have different meanings:

  • maximum total connections to one host;
  • maximum idle connections kept for reuse;
  • maximum concurrent requests per connection;
  • maximum time an idle connection is retained.

Confusing an idle limit with an active limit is a reliable way to get a surprise during a traffic spike. A setting that sounds like "keep 100 connections" may mean 100 idle connections in addition to every connection currently serving a request.

Retries can briefly double the damage

Retries are another way for one client request to create several upstream attempts. A proxy might retry after a connection reset, a refused connection or a selected 5xx response.

If the first attempt has genuinely failed, the extra work may be reasonable. The dangerous case is an uncertain failure. The backend may have received the request and be processing it, while the proxy has already decided that the connection is dead and started a second attempt.

A retry is extra concurrency until the old attempt is definitely gone. During a slow failure, one client request can therefore occupy two or more upstream connections at once.

Retries are particularly awkward for non-idempotent requests. Repeating a payment, account change or message submission can be worse than returning an error. Connection limits and retry policy need to be designed together, not configured independently by different layers.

Health checks and prewarming have their own traffic

Some connection growth is not caused by client requests at all. Proxies may open sockets for health checks, backend discovery, connection prewarming or periodic keep-alive probes.

That traffic is usually small, but it matters when the service has many backend instances. A fleet of proxy workers can multiply a modest health-check interval into a steady stream of connections. During a rollout, every worker may also discover new addresses and build a fresh pool while the old pool is still draining.

Actually, this is one reason a deployment can fail without a traffic increase. The application has the same users, but the proxy temporarily has old connections, new connections, health-check connections and requests waiting for either pool.

Set a budget at every boundary

The useful control is not one enormous global connection limit. It is a set of budgets that match the shape of the system.

  • Limit concurrent streams or requests accepted from each client where practical.
  • Limit active upstream connections per backend or upstream group.
  • Limit idle connections so quiet services do not retain a large pool forever.
  • Limit retries and make retryable methods explicit.
  • Set deadlines so stuck attempts leave the pool.
  • Reserve capacity for health checks and administrative traffic.

For a Go service making upstream requests directly, the standard HTTP transport exposes several of these controls:

transport := &http.Transport{
	MaxConnsPerHost:     200,
	MaxIdleConns:        100,
	MaxIdleConnsPerHost: 20,
	IdleConnTimeout:     90 * time.Second,
	ResponseHeaderTimeout: 5 * time.Second,
}

client := &http.Client{
	Transport: transport,
	Timeout:   15 * time.Second,
}

The exact values are workload-dependent. The important distinction is that MaxConnsPerHost limits active and idle connections managed by the transport, while MaxIdleConnsPerHost controls how many idle connections are retained for reuse. A client timeout is also not a substitute for a deliberate retry policy.

Backpressure is better than an unlimited queue

When the upstream budget is full, the proxy has three broad choices: queue the request, reject it, or let some other timeout eventually reject it. The third option is usually the least intentional and the hardest to operate.

Bound the queue as well as the connections. An unlimited queue converts socket pressure into memory pressure and latency. Under sustained overload, requests wait longer, clients retry, and the extra retries create still more work. That is how a fairly ordinary saturation event turns into a small self-licking ice cream cone.

A bounded queue with a fast, visible failure is often healthier than accepting work that cannot be started within a useful deadline. Return an appropriate overload response, record the rejection, and let the caller decide whether retrying makes sense.

Measure the layers separately

To find the real limit, measure downstream connections, downstream requests, upstream active connections, upstream idle connections, queued requests and retry attempts as separate metrics.

For each backend, also record connection creation rate and connection reuse. A stable connection count can hide a proxy that is constantly closing and recreating sockets. That creates handshake work and can overload a backend even when the number of open connections never looks alarming.

During a load test, deliberately vary concurrency per client. Ten clients with one request each is not equivalent to ten HTTP/2 clients with 100 concurrent streams each. Test slow responses too, because that is where request concurrency becomes connection pressure.

The practical rule

A reverse proxy is a protocol translator, a connection pool and often a retry engine. Treating it as a transparent pipe leads to misleading capacity calculations.

Start with the maximum concurrent requests the proxy can accept. Work out how those requests map to upstream streams and sockets, then add idle pools, health checks and retry overlap. If that total is not an explicit budget, the kernel will eventually choose one for you.

← All articles