Phone:

Hidden from the page source until you click: friction against scrapers, not a guarantee.

Email:

[email protected]

Category:

Systems Programming

Published:

Tags:
  • linux
  • io_uring
  • networking
  • performance
  • security
  • epoll

Why io_uring Doesn't Automatically Make Your Linux Network Service Faster

Every few months someone posts a benchmark showing io_uring beating epoll by some large multiple, and someone else concludes their network service should be rewritten to use it. Most of the time this ends with a service that is exactly as fast as before, considerably more complex, and running with a larger kernel attack surface than it needs. The benchmarks aren't wrong. They're usually measuring something your network service doesn't actually do much of.

What io_uring actually replaces

epoll is a readiness API. You call epoll_wait, the kernel tells you which file descriptors are readable or writable, and you then make a separate syscall (read, write, accept4) to actually do the I/O. Two syscalls per operation, minimum, plus whatever epoll_ctl bookkeeping you did to register interest in the first place. io_uring replaces this with a pair of ring buffers shared between userspace and the kernel: a submission queue (SQ) and a completion queue (CQ). You write a submission queue entry describing an operation (read this fd into this buffer, accept on this listener, write this buffer to that fd), bump a tail pointer, and the kernel processes it asynchronously and posts a completion queue entry when it's done. Crucially, you can queue up dozens of operations and enter the kernel once with a single io_uring_enter call, rather than once per operation.

/* conceptual shape, not exact liburing API */
sqe = io_uring_get_sqe(&ring);
io_uring_prep_recv(sqe, fd, buf, len, 0);
sqe->user_data = conn_id;

/* ... queue up N more operations against other fds ...*/

io_uring_submit(&ring);          /* one syscall for all of them */

io_uring_wait_cqe(&ring, &cqe);  /* completions arrive asynchronously */

That amortisation of syscall overhead is real and it's where essentially all of the headline numbers come from. The question that matters for your service is: how many operations are you actually batching per io_uring_enter call, and was syscall entry/exit your bottleneck to begin with?

The bottleneck it doesn't touch

For a typical network service, the syscall overhead per request was never the dominant cost. epoll_wait already returns a batch of ready descriptors in one call; the kernel isn't making you pay for readiness notification one fd at a time. What actually costs CPU in a request/response service is: copying bytes between kernel and user buffers, TLS handshake and record encryption, JSON or protobuf marshalling, allocation and GC pressure (if you're in Go), and whatever your application logic does with the request. None of that moves when you switch the I/O multiplexing mechanism underneath it. If you profile a Go HTTP service under load, the epoll-related syscalls are rarely more than a few percent of total CPU time; TLS and allocation dwarf them. io_uring can still batch the syscalls in a request/response loop, but if each connection is doing one read and one write per round trip, and requests arrive independently rather than in bursts, there's often nothing to batch. You end up submitting one SQE at a time anyway, which gets you the completion-based model's added bookkeeping without the syscall-amortisation benefit that justified it. A single io_uring_enter per operation is not obviously cheaper than a read or write syscall, and on some kernel/hardware combinations it measures slightly worse once you count the ring housekeeping.

Where the batching genuinely pays for itself

The workloads where io_uring reliably wins share a shape: many independent operations available to submit at once, or operations that would otherwise block a whole thread. That's mostly storage I/O, not networking. A database doing thousands of concurrent random reads against NVMe, a backup tool fanning out reads across many files, a proxy doing bulk file serving alongside socket I/O: these have genuine batches to submit, and storage syscalls (unlike socket syscalls) don't have an equivalent async, non-blocking mode without io_uring or a thread pool. That asymmetry is the real reason io_uring's storage benchmarks look so good: the previous baseline usually involved a thread pool blocking on synchronous file I/O, which is a much lower bar than "a socket epoll loop that was already non-blocking." For network servers specifically, the cases that benefit are things like reverse proxies or load balancers handling tens of thousands of connections with a genuinely high aggregate syscall rate, where accept/recv/send volume is large enough that syscall entry cost shows up in a flame graph. Multishot operations help here: IORING_ACCEPT_MULTISHOT and multishot recv (landed roughly in the 5.19 and 6.0 kernels respectively) let you post one SQE that keeps generating completions for new connections or new data without you re-submitting each time, which is a genuinely different cost profile from epoll's per-event re-arming. If you're not on a kernel that supports these, you're comparing io_uring's older, single-shot socket support against epoll, and that comparison is far less favourable than the multishot numbers people usually quote.

Buffer ownership is not a small detail

epoll's readiness model means you own your buffers throughout: the kernel tells you data is ready, you call read into a buffer you control, and the syscall is synchronous from the caller's point of view. io_uring is a completion model: once you submit a read into a buffer, the kernel owns that memory until the completion is posted. You cannot safely reuse, move, or free that buffer in the meantime. This is a mismatch with async runtimes built around cancellable futures. In Tokio, dropping a future is supposed to stop the work; that's the whole basis of select!'s cancellation. But if the future underneath was an in-flight io_uring read, the kernel may still hold a pointer into your buffer when you drop it. tokio-uring deals with this by requiring you to move buffer ownership into the operation and hand it back on completion, rather than borrowing it, and by issuing an actual IORING_OP_ASYNC_CANCEL and waiting for the cancellation to complete before the buffer is considered free again. It works, but it means io_uring-based I/O in Rust can't just slot into code written against borrowed &mut [u8] buffers; the ownership-passing API is a deliberate, structural difference, not an implementation detail you can paper over. Go has no standard library story for this at all. The runtime's netpoller is epoll-based (kqueue on BSD, IOCP on Windows), and there's no equivalent goroutine-friendly abstraction over io_uring's completion model in the standard toolchain. Using it means a cgo binding or a third-party ring wrapper, managing buffer lifetimes yourself outside the garbage collector's usual guarantees, and losing the portability the netpoller gives you for free.

SQPOLL isn't a free lunch either

io_uring has a mode, IORING_SETUP_SQPOLL, where a dedicated kernel thread polls the submission queue so your application never has to call io_uring_enter at all for submission. This is the configuration used in the most eye-catching benchmarks, because it removes submission-side syscalls entirely. It also means that kernel thread is spinning, consuming a full CPU core, whether or not you're actually submitting work. On a service where you have cores to spare and a genuinely high operation rate, that trade is worth it. On a modest VM with two or four vCPUs, dedicating one to a polling thread for a workload that wasn't syscall-bound in the first place is a pure loss.

The security cost is not hypothetical

io_uring's completion model means many operations happen without going through the traditional per-syscall path that tools like seccomp filter on; a sandbox that blocks specific syscalls by name can be blind to the underlying operation once it's submitted through io_uring_enter. Combined with the ring buffers being fairly complex shared kernel/userspace state, this has made io_uring an unusually productive target for kernel exploitation. Google's kCTF vulnerability reward programme reported that around 60% of the working exploits submitted under it targeted io_uring bugs, with roughly a million dollars paid out for io_uring-related exploits (Phoronix). Off the back of that, Google disabled io_uring by default in ChromeOS, restricted it on Android via a seccomp-bpf filter (with SELinux confinement planned for later releases), and disabled it on Google's own production servers and in GKE Autopilot (openSUSE Forums summary of the Google announcement). None of that means io_uring is unsafe to use anywhere. It means that adopting it for a network service is not a neutral performance tweak: it's opting a process into a kernel interface with a worse security track record than epoll's, for a benefit that, per the above, frequently isn't there for network I/O specifically. That trade might be worth making for a storage-heavy service processing untrusted-but-not-adversarial input. It's a much harder sell for an internet-facing socket handler where the attacker controls the input and the whole point of the sandboxing was to contain exactly that.

Before reaching for io_uring, the useful question isn't "is io_uring faster than epoll" (sometimes, for some things) but "what fraction of my request latency is spent in syscall entry and exit." strace -c or a perf profile with syscall attribution will answer that in about five minutes, and for most network services the answer is a low single-digit percentage sitting underneath TLS, marshalling, and application logic that io_uring cannot touch at all.