I/O Models: Blocking, epoll & io_uring

A network server scales or collapses on one decision: what a thread does while it waits for bytes that have not arrived yet. Under blocking I/O the kernel takes the calling thread off the CPU entirely — marks its task TASK_INTERRUPTIBLE, runs the scheduler, and only wakes it when a NIC interrupt delivers data — so serving N connections costs N threads. Non-blocking I/O plus a kernel readiness notifier (epoll) or a completion queue (io_uring) lets a single thread multiplex tens of thousands of connections, because it never parks on any one of them. This page traces how each model actually moves bytes across the syscall boundary, why select hit a wall at ~10,000 connections, and how epoll and io_uring got past it.

Blocking vs non-blocking: what read() does at the syscall boundary

Every socket read is a syscall into the kernel, and the socket's receive buffer is either empty or not. That single fact drives everything.

Blocking (the default). read(fd, buf, n) on an empty socket puts the calling thread to sleep in the kernel. The scheduler picks another runnable task; your thread consumes no CPU but holds its full kernel and user stack. When a packet arrives, the NIC raises an interrupt, the network softirq copies the segment into the socket buffer, marks the waiting task runnable, and — after a context switch — read() returns the bytes. Simple to reason about: one connection, one thread, straight-line code. The cost is that waiting is expensive — it burns a whole thread.

Non-blocking (fcntl(fd, F_SETFL, O_NONBLOCK)). Now read() on an empty buffer returns -1 immediately with errno == EAGAIN (a.k.a. EWOULDBLOCK). The thread is never parked, so it can go serve other sockets. But this only helps if something tells you when a socket becomes readable — otherwise you spin, calling read() in a hot loop and burning 100% CPU on EAGAIN. That "something" is the readiness notifier.

The C10k problem: why thread-per-connection stops scaling

Dan Kegel named it in 1999: how do you serve 10,000 concurrent connections on one box? The blocking, thread-per-connection model has three costs that all grow with connection count, not with actual work:

• Memory. Each thread reserves a stack — commonly 1–8 MB of virtual address space (resident is smaller, but non-trivial). 10,000 threads × even ~1 MB touched ≈ gigabytes, most of it holding idle keep-alive connections.
• Scheduler & context switches. With thousands of runnable threads the scheduler thrashes; each context switch flushes CPU caches and TLB entries. On old kernels the scheduler itself was O(n) in runnable tasks.
• The notifier is O(N). If you go non-blocking with select/poll, every event-loop iteration re-scans all N descriptors — even though at any instant only a handful have data.

The killer detail: on a busy web server most connections are idle at any moment (HTTP keep-alive, slow clients, long-poll). You are paying O(N) for O(active). The fix is to make the kernel remember your interest set and report only what changed.

select/poll → epoll: from O(N) rescan to O(ready)

select takes three fd_set bitmasks (read/write/error). Every call you rebuild the masks, copy them into the kernel, the kernel scans all bits, marks the ready ones, copies everything back, and you scan again to find them. Cost is O(N) per call in both directions, and FD_SETSIZE caps it at 1024 fds. poll replaces the bitmask with a struct pollfd[] array — no 1024 limit — but it is still O(N): you pass the whole array every call and the kernel walks all of it.

epoll (Linux 2.6) breaks the pattern by splitting registration from waiting:

1. epoll_create1() makes a kernel epoll instance.
2. epoll_ctl(ep, EPOLL_CTL_ADD, fd, &ev) registers each fd once. The kernel stores it in a red-black tree (the "interest set") and hooks a callback (ep_poll_callback) onto that socket.
3. When a packet arrives, the network softirq fires that callback, which appends the fd to a ready list inside the epoll instance.
4. epoll_wait() just returns the ready list. Its cost is proportional to the number of ready fds, not the total registered. Nothing is copied per-call for the idle millions.

Trace it: 10,000 keep-alive connections, 3 with pending data. poll hands the kernel 10,000 pollfds and the kernel touches all 10,000. epoll_wait returns an array of length 3. Same workload, 3,000× less bookkeeping per loop.

Make it wall-clock: 10,000 pollfds × 8 bytes = 80 KB copied into the kernel and scanned on every loop iteration. At tens of thousands of iterations per second — say 50k/s under load — that is 80 KB × 50k/s = 4 GB/s of pure copy-and-scan, a large fraction of a core burned on bookkeeping, while epoll's cost stays pinned to the 3 ready events. That, not any per-connection limit, is where the C10k wall physically sat: the cost scales with N × loop rate, not with N alone — which is why a poll server dies fastest exactly when traffic (and thus wakeup rate) peaks.

Level-triggered vs edge-triggered — and the drain pitfall

epoll has two notification modes. Level-triggered (default): epoll_wait keeps reporting a fd as long as it has readable data — forgiving, but a fd you haven't fully read stays "hot." Edge-triggered (EPOLLET): you're notified only on the transition from not-ready to ready — one wakeup per arrival. Edge-triggered is faster (fewer wakeups) but has a mandatory contract: on each notification you must read until EAGAIN, or leftover bytes sit in the buffer with no further wakeup and the connection stalls forever. Level-triggered is the safe default — use it unless you have measured that edge-triggered's extra wakeup savings matter, and even then, wrap every read in a drain loop.

int ep = epoll_create1(0);
struct epoll_event ev = { .events = EPOLLIN | EPOLLET, .data.fd = conn };
epoll_ctl(ep, EPOLL_CTL_ADD, conn, &ev);

struct epoll_event events[64];
for (;;) {
    int n = epoll_wait(ep, events, 64, -1);   /* returns ONLY ready fds */
    for (int i = 0; i < n; i++) {
        int fd = events[i].data.fd;
        for (;;) {                            /* edge-triggered: must drain */
            ssize_t r = read(fd, buf, sizeof buf);
            if (r > 0)                     { handle(buf, r); continue; }
            if (r == -1 && errno == EAGAIN) break;       /* buffer drained */
            if (r == -1 && errno == EINTR)  continue;     /* retry */
            close(fd); break;                            /* r==0 peer closed, or error */
        }
    }
}

The inner for(;;) loop is the whole point of edge-triggered epoll: a single read() is a bug.

io_uring: from readiness to completion, and batched syscalls

epoll still has two structural taxes. First, it is readiness-based: it tells you a fd is ready, then you make a separate read() syscall to actually move the data — two boundary crossings per operation. Second, epoll never worked for regular files — epoll_ctl rejects them with EPERM, since a disk file has no not-ready state to wait on — so buffered file I/O stayed synchronous. (select/poll accept regular files but report them permanently "ready," which is just as useless.) io_uring (Linux 5.1, 2019, by Jens Axboe) fixes both by being completion-based over two shared ring buffers.

The app and kernel share two lock-free single-producer/single-consumer rings via mmap, so no data is copied to move a request or result across the boundary:

• Submission Queue (SQ). The app fills a Submission Queue Entry (SQE) — "read fd=7 into buf, offset 0", "accept on fd=3", "write fd=9" — and advances the SQ tail. Any op that has a syscall has an SQE opcode: read, write, accept, connect, recv, send, openat, fsync, even timeouts and linked op chains.
• Completion Queue (CQ). When the kernel finishes an op it posts a Completion Queue Entry (CQE) holding the result (bytes transferred or -errno) and your user_data tag, and advances the CQ tail.

io_uring_enter() submits a batch: 128 queued reads go across the boundary in one syscall instead of 128. With IORING_SETUP_SQPOLL, a dedicated kernel thread continuously polls the SQ ring for new entries, so a saturated server can submit and reap I/O with zero syscalls in steady state — the app just writes SQEs into shared memory and the kernel thread picks them up (at the cost of that thread spinning and burning a CPU core, and it must be re-armed if it idles out). io_uring_register() can pre-register fds and fixed buffers so the kernel skips the per-call fd-table lookup and page-pinning that a plain read() repeats every time. If the app falls behind draining completions, the CQ can fill up; io_uring reports this as CQE overflow, and with IORING_FEATURE_NODROP the kernel backpressures the ring rather than silently dropping completions. Because it's completion-based, there's no second read after readiness, and it does true async for disk files that epoll never could.

The pre-io_uring async-I/O landscape

Before io_uring, Linux offered two completion-based paths: POSIX AIO (the aio_* family in glibc, often implemented with a thread pool under the hood) and Linux AIO (the kernel io_submit/io_getevents family). They never displaced epoll because they were limited. Linux AIO performed well only with O_DIRECT files, did not support sockets or accept(), required manual event handling, and still cost a syscall per batch. POSIX AIO in glibc frequently dispatched operations to worker threads, which removed predictable latency wins. io_uring was designed to subsume both: one ring interface for sockets, files, accepts, and timeouts, with batched submission and completion. If you maintain code on a pre-5.1 kernel, Linux AIO is the completion-based option you may encounter; on any modern kernel, io_uring is the path forward.

io_uring vs epoll: when to reach for which

They are not strict upgrades of each other — pick based on what you're actually bottlenecked on.

• Syscall count. epoll+read/write still pays one syscall per readiness check plus one per actual read/write. io_uring collapses N operations into one io_uring_enter(), or zero with SQPOLL. Under high fan-out (many small ops/sec) this is the single biggest win.
• Regular files. epoll cannot watch disk files at all (epoll_ctl returns EPERM); io_uring can do real async disk I/O. If your workload is proxying sockets only, this advantage is moot.
• Maturity and portability. epoll is 20+ years old, present on every Linux since 2.6, and battle-tested in every major event loop. io_uring needs a modern kernel (meaningful features arrived 5.6–5.11+), so containers/cloud fleets pinned to older LTS kernels, or anything targeting non-Linux, can't rely on it.
• Security surface. io_uring's large, fast-evolving in-kernel surface has had a disproportionate share of Linux kernel CVEs (privilege escalation bugs, e.g. those found by Google's kCTF/kernel fuzzing efforts circa 2021–2023). Several cloud providers and hardened distros (Google's ChromeOS, some managed Kubernetes/container platforms) disable io_uring by default or restrict it via seccomp, which can silently break an io_uring-based app in production.
• Complexity. Raw io_uring is a ring-buffer protocol, not a friendly API; almost everyone uses liburing to wrap it, and even then, correctly reasoning about SQE lifetime, buffer registration, and CQE overflow is harder to get right than an epoll readiness loop.

Rule of thumb: reach for epoll first — it's simpler, universally available, and sufficient for most socket-only servers. Reach for io_uring when profiling shows syscall overhead dominating (very high connection/op churn) or when you need real async file I/O, and you control the kernel version and can tolerate the newer, occasionally-patched security surface.

The reactor pattern (and its proactor cousin)

These primitives are wired together by the reactor pattern, the backbone of Nginx, Node.js (libuv), Netty, and Redis. Its parts: a synchronous event demultiplexer (epoll), a set of non-blocking event handlers registered per fd, and a single-threaded dispatcher loop. The loop calls epoll_wait, gets the ready fds, and for each one invokes its handler, which does a non-blocking read/write and returns immediately. One thread, one loop, thousands of connections — the handler must never block or it stalls every other connection on that loop.

io_uring instead fits the proactor pattern (the same shape as Windows IOCP): you initiate an operation and the OS completes it, handing you the finished result. The dispatcher reaps completions rather than readiness. Reactor asks "which fds can I act on?"; proactor says "here are the ops that are done." That shift is why completion-based I/O removes the extra syscall per op that reactor-style epoll+read/write always pays — the price is a fundamentally different, ring-buffer-shaped programming model instead of a simple readiness callback.

A minimal liburing read-batch

Using liburing is still ring-buffer programming, but it hides the tail/head arithmetic. The core loop is: get an SQE, fill it, submit, then reap CQEs. Here is a batch read of BATCH files.

#define BATCH 32

struct io_uring ring;
io_uring_queue_init(BATCH, &ring, 0);

int fds[BATCH];
char bufs[BATCH][4096];

for (int i = 0; i < BATCH; i++) {
    struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
    io_uring_prep_read(sqe, fds[i], bufs[i], sizeof(bufs[i]), 0);
    io_uring_sqe_set_data(sqe, (void *)(uintptr_t)i); // user tag
}

io_uring_submit(&ring);                    // one syscall submits BATCH reads

struct io_uring_cqe *cqe;
for (int i = 0; i < BATCH; i++) {
    io_uring_wait_cqe(&ring, &cqe);        // wait for next completion
    int idx = (int)(uintptr_t)cqe->user_data;
    if (cqe->res < 0) {
        fprintf(stderr, "read %d failed: %s\n", idx, strerror(-cqe->res));
    } else {
        printf("read %d: %d bytes\n", idx, cqe->res);
    }
    io_uring_cqe_seen(&ring, cqe);
}

io_uring_queue_exit(&ring);

The win is in the submission: 32 reads cross the user/kernel boundary in one io_uring_submit syscall, not 32 separate read() traps. In SQPOLL mode even that one syscall disappears while the kernel poll thread is awake.

🪜 Drill ladder: Level-triggered vs edge-triggered epoll

Use these questions to test whether you own the notification contract, not just the API.

1. When will LT report a ready fd? On every epoll_wait as long as the socket buffer has data. You can read one chunk and the next epoll_wait will still report it.
2. When will ET report a ready fd? Only on the transition from not-ready to ready. If you read only 1 KB from a 4 KB buffer, the remaining 3 KB will not wake you up again.
3. What is the mandatory ET handler shape? A drain loop with four exits: keep calling read(). r > 0 → consume and continue; r == 0 → peer closed, close(fd) and stop; r == -1 && errno == EINTR → retry; r == -1 && errno == EAGAIN → buffer drained, stop (the next edge re-arms you); any other errno → close(fd). Compressing this to "loop until EAGAIN" is a bug: it spins forever on EOF or a persistent error, because neither sets EAGAIN.
4. Which mode is safer as a default? LT. ET is a measured optimization, not the default, because one missed drain stalls the connection forever.
5. Can you use epoll for regular files? You can't even register one: epoll_ctl(EPOLL_CTL_ADD) on a regular file fails with EPERM, because regular files have no readiness state to wait on. (select/poll do take them — but report them permanently "ready," which is equally useless.) Either way, buffered disk reads block the caller; real async disk I/O needs io_uring or a thread pool.

io_uring caveats and kernel-version reality

io_uring is powerful, but the power comes with sharp edges that matter in production.

• Kernel version. The basic interface needs Linux 5.1+. Modern helpers such as multishot accept, direct descriptors, registered buffers, and IORING_OP_SEND_ZC need 5.19–6.x. If your container host or cloud fleet runs an older LTS kernel, features you develop locally will silently fail or fall back.
• CVE surface. The fast-growing in-kernel io_uring code has been the source of several privilege-escalation vulnerabilities. Some managed Kubernetes platforms and hardened distros disable io_uring via seccomp by default, so an app that works on your laptop may get -EPERM in production.
• Buffer lifetime. With non-registered buffers, the kernel may access the buffer asynchronously. You must keep the buffer alive until the CQE returns. Reusing or freeing it too early is a use-after-free that is not possible with synchronous read().
• SQPOLL cost. Zero syscalls in steady state sounds ideal, but the kernel poll thread burns a CPU core and must be re-armed if it idles out. It is a throughput optimization, not a free lunch.
• Ordering. By default, SQEs submitted in one batch may complete out of order. If you need ordering, use linked SQEs or explicit sequencing — do not assume FIFO.

Rule of thumb: start with epoll for socket-only servers; switch to io_uring when you have measured syscall overhead as a bottleneck and you control the kernel version.

🪜 Drill ladder: I/O Models — Blocking, epoll & io_uring

1. Why does thread-per-connection hit a wall? Memory stacks + scheduler overhead + O(N) readiness scans; most connections are idle keep-alives.
2. What is the epoll complexity win? Interest set is registered once; epoll_wait costs O(ready), not O(total).
3. What must an ET handler do that an LT handler can skip? Drain the fd until EAGAIN; otherwise leftover bytes never trigger another wakeup.
4. How does io_uring reduce crossings? Requests and completions move through shared rings; one io_uring_submit can carry many operations.
5. Why can't you rely on io_uring in every environment? Older kernels, disabled-by-policy seccomp, and CVE history mean availability is not universal.