× Install ThecoreGrid App
Tap below and select "Add to Home Screen" for full-screen experience.
B2B Engineering Insights & Architectural Teardowns

io_uring offloads CPU through asymmetric I/O

Asymmetric io_uring redistributes I/O from application cores to dedicated cores. This changes the balance of CPU, latency, and throughput in high-load systems.

Modern multi-core systems are constrained not only by network or disk but also by competition for CPU. In Seastar, each core runs its own shard with a reactor and performs both computations and low-level I/O operations. In the symmetric model of io_uring, each shard maintains its own ring and sometimes bypasses it through the “fast path” with synchronous syscalls. This saves time on non-blocking sockets, but ultimately mixes compute and I/O on the same cores. As the load increases, this leads to worse cache locality and competition between request processing and system calls.

The idea behind the solution is simple: if part of the I/O is already being handled by dedicated networking cores (via IRQ and SoftIRQ), it makes sense to offload more work there. The asymmetric io_uring backend uses SQ polling and binds kernel worker threads to these cores. Application shards stop performing synchronous syscalls. They only place requests in the submission queue. The entire I/O execution chain is offloaded to the networking cores. This frees up the CPU for computations and reduces interference with hot caches. The trade-off is clear: there is a risk of overloading a limited number of networking cores and creating a new bottleneck.

The implementation relies on Seastar’s shared-nothing model. To avoid introducing locks and inter-shard synchronization, each shard maintains its own io_uring instance. Meanwhile, the worker pool is shared at the core level to avoid proliferating kernel threads and overloading the CPU. Shards are grouped, and each group is serviced by a dedicated networking core. Importantly, the fast path with speculative syscalls is completely removed. This is crucial: any bypass of io_uring destroys the asymmetry and shifts the load back to the application cores. Topology also matters: NUMA and SMT are taken into account via hwloc to minimize inter-node accesses and reduce latency in communication through ring buffers.

Buffer rings were considered separately. They allow for pre-registering buffers and reducing the overhead of memory allocation. Options with fixed buffers, breaking large operations into chains (IOSQE_IO_LINK), and multiple rings of different sizes were tested. However, integration conflicts with the current memory management model in Seastar. This would require deeper architectural changes, so the idea was abandoned at this stage.

Results show that behavior heavily depends on the load profile. In a test with a single shard, asymmetric io_uring provides about a 15% increase in throughput and improves p50 latency. This is expected: the freed CPU processes user code faster. But as the number of shards increases, the picture changes. One networking core becomes a bottleneck. Profiling with perf shows that up to 40–50% of the time is spent on memory copying within the kernel. Previously, this load was distributed among application cores during syscalls. Now it is concentrated on one core and hits its throughput limit.

This is an important observation. Asymmetric io_uring excels where there is a significant share of computations or waits. In pure I/O scenarios (network-bound throughput), offloading the load may worsen scaling. But in real systems, such as databases, requests rarely consist solely of byte transfers. There is parsing, business logic, coordination. In such conditions, freeing application cores provides more predictable latency and a better system balance.

In conclusion, this is a pragmatic compromise. Asymmetric io_uring is not universally faster. It changes the distribution of work: less noise on application cores, more pressure on networking cores. Architecturally, this aligns with the trend toward resource isolation and explicit CPU management. But it requires careful tuning: the number of worker cores, their binding, and understanding the load profile. Without this, it is easy to replace one point of degradation with another.

Reference source

×

🚀 Deploy the Blocks

Controls: ← → to move, ↑ to rotate, ↓ to drop.
Mobile: use buttons below.