KernelDiag: Agent-Based Root Cause Diagnosis for Kernel Crashes

Technical Summary: KernelDiag

Problem Statement
The Linux kernel, a highly complex system with over 40 million lines of code, relies on automated fuzzing (e.g., Syzkaller) to expose thousands of crashes annually. However, diagnosing the root cause of these crashes remains a manual, labor-intensive bottleneck. Existing Root Cause Analysis (RCA) techniques, particularly those leveraging Large Language Models (LLMs), are effective for distributed systems but fail to generalize to kernel debugging. This failure stems from two primary gaps:

1. Sparse and Heterogeneous Artifacts: Unlike cloud environments rich in end-to-end traces, kernel debugging relies on low-level, fragmented evidence: triggering syscalls, unstructured runtime logs, and crash reports. These artifacts are loosely coupled, temporally misaligned, and exist at different abstraction levels.
2. Complex Fault Propagation: Kernel failures often involve non-linear, indirect execution paths (e.g., macro expansions, interrupt-driven callbacks, configuration-dependent behaviors). Existing approaches often provide coarse-grained outputs (e.g., faulty modules) insufficient for the fine-grained, method-level localization required for actionable kernel patching.

Methodology: KernelDiag
KernelDiag is an agent-based framework designed to perform structured causal reasoning over heterogeneous kernel artifacts. Its architecture consists of three main phases:

1. Log-to-Code Mapping (Artifact Alignment):
   To bridge the gap between unstructured logs and source code, KernelDiag employs a static-analysis-based approach. It identifies "atomic logging sinks" (functions directly invoking primitives like printk) and performs a backward reachability analysis over the logging dependency flow (including macro expansions). This maps log entries to precise source-level locations, creating a common semantic space for cross-artifact correlation.

2. Agent-Based Causal Reasoning:
   KernelDiag decomposes the reasoning task into role-specialized agents, each analyzing a specific artifact type to construct a schema-validated Evidence Graph:

   • Syscall Agent: Analyzes user-space invocations to infer the execution skeleton and kernel objects under test.
   • Log Agent: Treats logs as observable state transitions, performing backward reasoning to identify the earliest deviation from expected behavior.
   • Report Agent: Analyzes crash signals (stack traces, register states) to identify violated invariants and trace corrupted states.

   These agents utilize two kernel-aware mechanisms to deepen their analysis:

   • Semantic Function Introspection: Agents retrieve full function bodies, macro definitions, and struct declarations to inspect branch conditions, error handling, and state-update logic, resolving pointer-based accesses to their concrete owners.
   • Environment-Aware Semantic Pruning: Agents inspect the kernel .config file to eliminate execution paths that are infeasible under the specific build configuration, filtering out spurious branches.

   The agents incrementally update the Evidence Graph, where nodes represent kernel entities (operations, data carriers, constraints) and edges represent causal dependencies (data/control flow, lifecycle transitions).

3. Root-Cause Diagnosis:

   • Localization: A two-stage process first identifies faulty files (Tier-1 vs. Tier-2) based on causal origin rather than crash proximity, followed by fine-grained method-level localization within prioritized files.
   • Explanation: An Explanation Agent synthesizes the Evidence Graph and localization results to generate a causal narrative tracing the failure from the root method to the observed crash.

Key Contributions

1. Framework Proposal: The introduction of KernelDiag, a novel agent-based framework that leverages structured reasoning over heterogeneous artifacts (syscalls, logs, crash reports) to model fault propagation in the Linux kernel.
2. Performance Gains: Demonstration of superior root-cause localization effectiveness at both file and method levels, particularly in challenging "No-Hint" scenarios where explicit root-cause signals are absent.
3. Diagnostic Quality: Generation of high-quality, evidence-grounded diagnostic explanations that support developers in understanding and fixing kernel failures.
4. Reproducibility: Public release of a replication package to facilitate future research.

Experimental Results
The authors evaluated KernelDiag on the KGYM benchmark, comprising 279 real-world kernel crashes.

• Localization: KernelDiag consistently outperformed state-of-the-art baselines (LinuxFL+ and Agentless).
  • At the file level, it achieved 65.95% Top@1 accuracy (vs. 51.61% for LinuxFL+), a 27.78% improvement. In the challenging No-Hint setting, it achieved 31.43% Top@1 accuracy compared to 0.00% for baselines.
  • At the method level, it achieved 33.33% Top@1 accuracy (vs. 26.52% for LinuxFL+). In the No-Hint setting, it reached 34.78% Top@10 accuracy, nearly doubling the performance of existing approaches.
• Explanation Quality: Both LLM-assisted (using GPT-5.2) and human evaluations (by kernel testing experts) confirmed that KernelDiag generates more consistent, useful, and clear explanations than baselines.
• Robustness: Evaluations on a post-release dataset (crashes fixed after the LLM's pre-training cutoff) showed KernelDiag maintained significant advantages, suggesting its performance stems from reasoning capabilities rather than data memorization.
• Ablation Studies: Removing any core component (e.g., specific artifacts, log-to-code mapping, or the Evidence Graph structure) resulted in consistent performance degradation, validating the necessity of the full framework.

Significance and Claims
The paper claims that KernelDiag lays the foundation for automated kernel root-cause diagnosis by effectively bridging low-level diagnostic evidence with source-level causal reasoning. By moving beyond surface-level signal matching to structured, multi-agent causal inference, the framework addresses the specific complexities of the Linux kernel—such as sparse artifacts and non-linear fault propagation—that have hindered previous RCA approaches. The authors posit that this approach enables more accurate fault localization and actionable debugging guidance, potentially reducing the mean time to repair (MTTR) for critical kernel issues. The work emphasizes that while it incurs higher computational costs due to deep semantic exploration, this trade-off is justified by the critical need for accuracy in kernel systems where manual diagnosis is currently the bottleneck.