Articles / Rust

704 words

  • rust
  • tokio
  • networking
  • emfile
  • file-descriptors
  • debugging

Why a Tokio accept Loop Exits on 'Too many open files'

The accept loop in most Tokio tutorials has a quiet flaw: one transient error and the whole server stops accepting connections. The culprit is the innocent-looking ? after accept().await, and the error that triggers it is usually EMFILE, "Too many open files" (os error 24).

The loop everyone writes

use tokio::net::TcpListener;

#[tokio::main]
async fn main() -> std::io::Result<()> {
    let listener = TcpListener::bind("0.0.0.0:8080").await?;
    loop {
        let (sock, _addr) = listener.accept().await?;
        tokio::spawn(async move {
            // handle sock
            drop(sock);
        });
    }
}

Every accepted connection costs one file descriptor. Run the process with a limit of 1024 (a common default) and open enough connections, and accept(2) fails with EMFILE. The ? returns from main, the runtime shuts down, and every in-flight connection is dropped with it.

You can reproduce it in a minute: run ulimit -n 64 in a shell, start the server from that shell, then open a hundred connections from another terminal. The server exits with Too many open files (os error 24).

It is worse if the loop is in a spawned task

If the accept loop lives inside tokio::spawn rather than main, the failure is quieter. The task returns an error nobody reads, the TcpListener is dropped, and the port closes. The process keeps running, health checks that only look at the PID stay green, and nothing is listening.

That version is the one that gets people paged at 3am. Either way the fix is the same.

Not every accept error means the same thing

accept can fail for quite different reasons, and they want different responses:

  • Per-connection errors such as ECONNABORTED: the client gave up between the handshake and your accept. Ignore it and accept the next one straight away.
  • Resource exhaustion such as EMFILE, ENFILE, ENOBUFS, ENOMEM: the process or the machine is out of something. This is temporary; back off and retry.
  • Anything else such as EBADF or EINVAL: the listener itself is broken. Retrying is pointless, so return the error.

Rust's io::ErrorKind does not give you a stable variant for EMFILE, so you check the raw OS error. The libc crate provides the constants.

use std::{io, time::Duration};
use tokio::{net::TcpListener, time::sleep};

fn is_connection_error(e: &io::Error) -> bool {
    matches!(
        e.kind(),
        io::ErrorKind::ConnectionAborted
            | io::ErrorKind::ConnectionReset
            | io::ErrorKind::ConnectionRefused
    )
}

fn is_resource_exhaustion(e: &io::Error) -> bool {
    matches!(
        e.raw_os_error(),
        Some(libc::EMFILE | libc::ENFILE | libc::ENOBUFS | libc::ENOMEM)
    )
}

async fn serve(listener: TcpListener) -> io::Result<()> {
    loop {
        match listener.accept().await {
            Ok((sock, _addr)) => {
                tokio::spawn(async move {
                    let mut sock = sock;
                    let (mut r, mut w) = sock.split();
                    let _ = tokio::io::copy(&mut r, &mut w).await;
                });
            }
            Err(e) if is_connection_error(&e) => continue,
            Err(e) if is_resource_exhaustion(&e) => {
                eprintln!("accept failed, backing off: {e}");
                sleep(Duration::from_secs(1)).await;
            }
            Err(e) => return Err(e),
        }
    }
}

The libc crate goes in Cargo.toml as a normal dependency. The echo handler is only there to make the example runnable.

Why the sleep matters

Quick detour, because this bit is interesting. When accept hits EMFILE, the kernel does not drop the pending connection. It stays in the listen queue, fully handshaken, and the client thinks it has connected.

So the very next accept call will most likely fail the same way, instantly. Without a sleep, your loop spins at 100% CPU while doing nothing useful. With it, you retry once a second, and as soon as some connection closes and frees a descriptor, the queue drains. As far as I know, hyper-based servers such as axum do much the same thing; check the version you use rather than trusting my memory.

Better: do not reach the limit at all

Backing off keeps the server alive, but clients in the queue still hang while you are at the limit. A cleaner approach is to cap concurrent connections well below the descriptor limit, and wait for a permit before calling accept.

use std::sync::Arc;
use tokio::sync::Semaphore;

async fn serve_capped(listener: TcpListener) -> io::Result<()> {
    let permits = Arc::new(Semaphore::new(512));
    loop {
        let permit = permits.clone().acquire_owned().await.expect("semaphore closed");
        match listener.accept().await {
            Ok((mut sock, _addr)) => {
                tokio::spawn(async move {
                    let (mut r, mut w) = sock.split();
                    let _ = tokio::io::copy(&mut r, &mut w).await;
                    drop(permit); // freed when the connection ends
                });
            }
            Err(e) if is_connection_error(&e) => continue,
            Err(e) if is_resource_exhaustion(&e) => {
                eprintln!("accept failed, backing off: {e}");
                sleep(Duration::from_secs(1)).await;
            }
            Err(e) => return Err(e),
        }
    }
}

Because the permit is taken before accept, excess clients wait in the kernel backlog instead of consuming descriptors. If a client abandons the wait, nothing leaks. Keep the retry arms anyway: the cap does not cover descriptors used by files, logs, or outbound connections to a database.

Choosing the numbers

  • Check the limit the process really has: cat /proc/<pid>/limits, and look at the "Max open files" line.
  • Count what is in use: ls /proc/<pid>/fd | wc -l.
  • Under systemd, raise it with LimitNOFILE= in the unit rather than ulimit in a wrapper script.
  • Set the connection cap so that cap, plus upstream connections per request, plus a margin for files, stays under the limit.

The semaphore value is a policy choice, not a magic figure. If every request opens one upstream connection, a cap of 512 can mean 1024 descriptors, so size it from your own workload.

One last check: after the fix, rerun the ulimit -n 64 experiment. The server should log a few backoff lines, stay up, and carry on serving as connections close.

← All articles