Taming System Complexity: Demystifying Software Engineering Agents in Diagnosing Linux Kernel Faults

Technical Summary: Taming System Complexity with LINUXFL+

Problem Statement
Fault localization (FL) is a critical component of software quality assurance, aiming to identify buggy code elements (files or functions) given a bug report and a codebase. While recent Large Language Model (LLM) agents have demonstrated promising accuracy in FL for general software systems (e.g., achieving ~70% accuracy on the SWE-bench benchmark), their performance in complex, large-scale systems like the Linux kernel remains unexplored and likely insufficient. The Linux kernel presents unique challenges that hinder existing agent-based approaches:

1. Large-scale Codebase: The kernel contains over 69,000 files and 28 million lines of code, significantly larger than typical projects in existing benchmarks.
2. Limited Observability: Due to the kernel's real-time nature and privileged mode, runtime instrumentation and logging are restricted. User-reported bug descriptions often lack detailed runtime data, creating a gap between symptoms and root causes.
3. Diverse Impact Factors: Kernel bugs are influenced by hardware configurations and runtime variability, leading to an exponentially large reasoning space for diagnosis.

Existing agents struggle to navigate this complexity, often failing to capture related files or cover the complete spectrum of potential root causes.

Methodology
The paper introduces a two-pronged approach: the construction of a new benchmark and the development of an enhancement framework.

1. LINUXFLBENCH (Benchmark Construction):
   The authors constructed a new benchmark comprising 250 real-world fault localization tasks derived from the Linux kernel. The construction process involved:

   • Collection: Gathering bug reports from Kernel.org Bugzilla (up to Dec 2024) linked to specific kernel versions.
   • Ground Truth Identification: Identifying buggy locations based on developer-committed patches, filtering for cases where exactly one file was modified to ensure unambiguous ground truth.
   • Manual Inspection: Human annotators verified that reports contained actual bugs, sufficient information (logs, descriptions), and did not explicitly reveal the solution.
   • Characteristics: The benchmark spans 120 kernel versions and 66 components, featuring bug reports that are significantly longer and more complex than those in SWE-bench, with codebases 10–30 times larger.

2. LINUXFL+ (Enhancement Framework):
   To address the limitations of existing agents, the authors propose LINUXFL+, a framework designed to refine agent predictions through two expansion strategies:

   • Directory-Aware Expansion: Recognizing that agents often identify the correct module but fail to pinpoint the specific file within a directory, this strategy expands the search scope to include all files within the directories of the agent's initial predictions. The LLM then re-selects the most relevant files from this expanded set.
   • Potential Cause Expansion: To overcome the tendency of agents to focus on a narrow set of probable causes, this strategy employs two hypothesizing methods:
     • Direct Hypothesis: Leveraging the LLM's intrinsic pre-trained knowledge to generate plausible causes and associated fix files.
     • Mail-Augmented Hypothesis: Utilizing a Retrieval-Augmented Generation (RAG) approach with historical data from the Linux Kernel Mailing List (LKML). The system retrieves relevant emails discussing similar bugs and patches to inform the generation of diverse potential causes.
   • Candidate Integration: Files identified through the initial agent prediction, directory expansion, and cause expansion are merged. A scoring mechanism aggregates ranks from all sources, followed by a final LLM-based re-ranking step to refine the list based on semantic correspondence with the bug report.

Key Results
The paper presents an empirical study evaluating state-of-the-art agents (SWE-Agent, AutoCodeRover, Agentless) on LINUXFLBENCH, followed by the application of LINUXFL+.

• Baseline Performance: Existing agents struggle significantly in the Linux kernel environment. The best-performing agent (SWE-Agent) achieved a top-1 file-level accuracy of only 41.6%, a substantial drop (16.7%–31.9%) compared to their performance on general software benchmarks like SWE-bench. Failure analysis revealed agents often confuse related files within the same directory or fail to explore non-obvious root causes.
• LINUXFL+ Effectiveness: The framework substantially improved the performance of all studied agents with minimal cost.
  • Accuracy Gains: Using GPT-4o, LINUXFL+ increased top-1 accuracy by 7.2% to 11.2% across different agents (e.g., SWE-Agent improved from 41.6% to 52.4%).
  • Robustness: The framework proved effective across different model sizes (including Qwen3-32B) and was particularly beneficial for "Hard" reports lacking explicit file hints, narrowing the performance gap between easy and difficult cases.
  • Method-Level FL: While method-level localization remains challenging (top-1 recall < 0.1 for baselines), LINUXFL+ consistently improved these metrics as well.
  • Cost-Efficiency: The framework incurred an estimated cost of approximately $0.04 per task, roughly one-tenth of the cost of the baseline agent-only approaches, primarily due to efficient email retrieval and targeted expansion.

Significance and Claims
The paper claims that LINUXFLBENCH provides a necessary, more rigorous benchmark for evaluating FL techniques in large-scale, complex systems, highlighting the limitations of current agents when applied to the Linux kernel. The significance of the work lies in demonstrating that while existing agents struggle with the kernel's scale and opacity, targeted expansion strategies leveraging repository structure and historical domain knowledge (LKML) can effectively bridge the gap. The authors assert that LINUXFL+ offers a cost-effective pathway to enhance software maintenance tasks in the Linux kernel, moving beyond the limitations of general-purpose agent evaluation. The work underscores the need for specialized strategies that account for the unique constraints and complexity of operating system kernels.