uringscope: Portable, Low-Overhead Observability for io_uring

This paper presents uringscope, a portable, low-overhead eBPF-based observability tool that reconstructs io_uring request lifecycles and diagnoses performance issues by overcoming the limitations of unstable kernel tracepoints and the invisibility of shared-memory rings.

The Gist

Imagine your computer's operating system (the kernel) and your applications are like a busy restaurant kitchen. For decades, if you wanted to know what the chef was doing, you'd stand outside the kitchen door and listen to every time they shouted an order or walked out to grab a plate. This is how tools like strace work: they watch the "door" (system calls).

But then, the kitchen got a massive upgrade called io_uring. Instead of shouting orders, the chef and the waitstaff now use a shared whiteboard (shared memory rings) right in the middle of the room. The chef writes down thousands of orders on the whiteboard, and the waitstaff picks them up and brings them back. The chef never has to shout, and the waitstaff never has to walk to the door.

The Problem:
Because everything happens on this invisible whiteboard, the old tools standing outside the door (strace) are now blind. They can see the chef walking to the whiteboard once, but they can't see the thousands of orders being written or the plates being returned. They see nothing but "Chef went to the whiteboard."

Furthermore, the layout of this whiteboard changes every time the kitchen manager updates the rules (kernel updates). A tool built for last month's whiteboard might not work today because the "Order" section moved to a different spot, or the "Status" column was renamed.

The Solution: uringscope
The authors built a new tool called uringscope. Think of it as a smart, invisible camera that sits directly above the whiteboard. It doesn't just watch the door; it watches the actual orders being written and the plates being returned.

Here is how it works, using simple analogies:

1. The "Magic Glasses" (CO-RE and eBPF)

The kitchen rules change constantly. If you built a camera that only worked on the "2023 Whiteboard," it would break when the "2024 Whiteboard" arrived.

• The Trick: uringscope uses "Magic Glasses" (a technology called CO-RE). Instead of memorizing exactly where the "Order" column is, the camera asks the kitchen manager, "Hey, where is the Order column right now?"
• The Result: The camera can adapt instantly. If the kitchen manager moves a column or renames a section, the camera finds the new spot automatically. This allows it to work on almost any version of the kitchen, from old ones to brand new ones.

2. The Two Modes: "The Summary" vs. "The Movie"

The tool offers two ways to watch the kitchen, depending on how much detail you need:

• Aggregate Mode (The Daily Report): This mode doesn't record every single plate. Instead, it keeps a running tally in the kitchen. "We served 1,000 pizzas, 500 burgers, and 3 burnt orders." It's very light on the chef's energy, so you can leave it running all day without slowing down the kitchen.
• Trace Mode (The Movie): This mode records every single action in high definition, creating a movie of the kitchen's activity. This is great for debugging a specific problem, but it uses more energy, so you only turn it on when you need to investigate a specific incident.

3. The "Kitchen Doctor"

Sometimes, the kitchen runs slowly. Is it because the chef is too slow? Is the fridge broken? Or is the waitstaff stuck in the hallway?

• uringscope has a built-in Doctor. Instead of just showing you a graph of "slow times," the Doctor looks at the data and says: "I see that 100% of your 'Sync' orders are getting stuck in the hallway (the worker pool). That's why your customers are waiting."
• It doesn't just show the problem; it points to the specific order and suggests a fix.

4. Safety Checks (The "Correctness" Mode)

In a busy kitchen, you don't want two waiters grabbing the same tray at the same time, or a waiter trying to carry a tray that was already thrown away.

• uringscope has a safety mode that acts like a safety inspector. It watches the whiteboard to make sure:
  • No two orders are trying to use the same tray at the same time (which would ruin the food).
  • No waiter is trying to carry a tray that was already taken off the board.
  • No orders are left sitting on the board forever, forgotten.
• If it finds a mistake, it flags it immediately so the kitchen manager can fix it before the food gets ruined.

Why This Matters

The paper tested this tool in a real kitchen (a high-performance computer server).

• Speed: When the kitchen is busy with heavy lifting (like moving big crates of food), uringscope's "Daily Report" mode only slows things down by a tiny bit (less than 10%). Other tools that try to watch the kitchen slow it down by 40% or even 90%.
• Accuracy: It can see 100% of the orders, even when the kitchen is moving at lightning speed.
• Adaptability: It survived a surprise kitchen renovation (a new kernel update) that happened while the authors were testing it, proving it can handle changes without breaking.

In short: uringscope is a portable, low-cost, and adaptable tool that lets you finally see what's happening inside the "invisible" part of modern computer systems, helping engineers fix slow performance and dangerous errors without slowing down the system itself.

Technical Summary

Technical Summary: uringscope

Problem Statement

The emergence of io_uring has fundamentally altered the contract between user-space applications and the kernel regarding I/O observability. Unlike traditional synchronous syscalls, io_uring moves I/O submission and completion into shared-memory rings. While this design yields high performance, it renders the request flow "invisible" to standard tracing tools.

• The Visibility Gap: Tools like strace only observe the io_uring_enter syscall boundary, missing the millions of individual requests contained within a single call.
• Instability of Interfaces: The kernel exposes tracepoints for the io_uring lifecycle, but these are explicitly not part of the stable ABI. They undergo frequent renaming, re-prototyping, and structural changes across kernel versions (e.g., 5.15 to 6.x), causing tools built on them to break rapidly.
• Existing Limitations: Current solutions are fragmented. strace distorts tail latency by orders of magnitude. perf and manual bpftrace scripts provide raw events but lack request-level correlation and require heavy offline processing. The only packaged tool, uring-trace, is limited to a narrow kernel range (6.1–6.7) and lacks portability.

Methodology

The paper presents uringscope, a single-binary, language-agnostic observability tool built on CO-RE (Compile Once, Run Everywhere) eBPF. The methodology addresses two distinct challenges: semantic reconstruction and mechanical portability.

1. Semantic Reconstruction of Request Lifecycle

The authors construct a precise model of the io_uring request lifecycle to map kernel events back to application-meaningful flows. This model accounts for:

• Three Execution Paths:
  • Inline: Immediate completion (fast path).
  • Poll-armed: Parked and re-issued if not ready.
  • Punt: Offloaded to the io-wq asynchronous worker pool (a hidden source of tail latency).
• Complex Semantics: Handling multishot operations (multiple CQEs per SQE), linked SQEs (ordered chains), SQPOLL (kernel-thread submission), and DEFER_TASKRUN (batched completion delivery).
• Correlation: Using the io_kiocb pointer as a stable key to correlate submissions with completions, while handling legacy kernels where user_data reuse requires weaker correlation logic.

2. Portable Attachment to Unstable Tracepoints

To survive the "dependency surface" of changing kernel interfaces, uringscope employs a layered technique:

• CO-RE Flavors: Using bpf_core_field_exists() to handle struct field layout drift.
• Struct-Centric Reads: Trusting the io_kiocb pointer rather than argument positions, which change across versions.
• BTF-Probed Variants: The tool compiles multiple program variants for renamed or re-prototyped tracepoints. At load time, it probes the kernel's BTF (BPF Type Format) to detect available symbols and autoloads the correct variant, degrading features gracefully rather than failing to load.
• Continuous Verification: A kernel-matrix CI tests portability across a range of kernels (5.15 to mainline).

3. Operational Modes

uringscope offers two modes to trade fidelity for overhead:

• Aggregate Mode: Keeps all per-event data (histograms, counters) inside kernel BPF maps. It exports aggregated statistics at exit or via Prometheus. This mode is designed for production use with minimal overhead.
• Trace Mode: Streams per-request lifecycle records to a Perfetto-compatible timeline for detailed diagnosis. Fidelity is bounded by ring-buffer capacity, with drops explicitly counted.

4. Correctness Checking ("The Doctor")

A lightweight correctness layer reuses the lifecycle reconstruction to detect submission-boundary hazards:

• Overlapping in-flight operations on the same buffer.
• Registered-buffer lifetime violations (unregistering while in use).
• Dropped requests that are never reaped.
• Use-after-free/munmap of in-flight buffers (specifically the munmap variant detectable via kernel tracepoints).
  The "Doctor" layer converts these measurements into named pathologies with evidence and suggested fixes.

Key Contributions

1. Request Lifecycle Model: A formalized model of the io_uring request flow that handles multishot, linked, and deferred operations, enabling per-request flow reconstruction.
2. Portable Attachment Technique: A method for attaching to unstable tracepoint surfaces using BTF-probed variants and CO-RE flavors, validated across kernel versions 5.15 through 6.17.
3. uringscope Tool: An open-source, single-binary tool that provides both aggregate production monitoring and detailed trace modes.
4. Correctness Mode: A "valgrind-like" checker for the submission boundary that detects specific buffer ownership hazards without requiring application instrumentation.

Evaluation Results

The evaluation was conducted on a 36-cell grid covering six workloads (including NVMe, null_blk, and network) and six observers (uringscope, perf, bpftrace, strace, baseline).

• Overhead:
  • Device-Bound Workloads: On NVMe storage, uringscope's aggregate mode incurs 0.7% to 9.9% throughput loss, significantly cheaper than perf (up to 13.3%) and strace (up to 97%).
  • CPU-Bound Workloads: At saturation (810k IOPS on null_blk), aggregate mode costs 31.9% throughput (583 ns/request). While not free, it is more efficient than perf (40.2%) and strace (93%+).
  • Tail Latency: uringscope-aggregate perturbs p99.9 latency by only 1.02x–1.03x on device-bound workloads, whereas strace distorts it by 33x.
• Fidelity: uringscope reconstructed 100% of completed requests in both modes, with zero untracked completions or ring-buffer drops in the tested grid.
• Portability: The tool successfully adapted to a kernel (6.17) released after its development, which changed the io_uring_complete prototype. The BTF-probing mechanism automatically selected the correct variant without manual intervention.
• Pathology Detection: All 11 injected pathologies (including punt storms, CQ overflow, and buffer overlaps) were detected with zero false positives.

Significance and Claims

The paper argues that the observability gap for io_uring is closable. The significance of uringscope lies in its ability to:

• Surface Hidden Latency: It identifies the "io-wq punt" as a classic hidden source of tail latency, turning vague complaints of "slow writes" into specific diagnoses (e.g., "fsync is punting 100% to the worker pool").
• Bridge the ABI Gap: It demonstrates that stable, portable tools can be built on top of unstable kernel interfaces through disciplined use of CO-RE and BTF probing.
• Provide Actionable Insights: Unlike raw event streams, the "Doctor" layer provides named pathologies with evidence, aiding operators debugging tail-latency incidents.

The authors position uringscope not as a replacement for kernel source reading, but as the necessary tool for engineers debugging io_uring-native systems (databases, storage engines, runtimes) where strace is useless and raw tracepoints are too brittle to manage manually. The tool is explicitly limited to the kernel-side view; it cannot measure the gap between application preparation and submission if the application inlines library calls, a limitation the authors acknowledge and document.