How and Why Agents Can Identify Bug-Introducing Commits

1. Problem Definition

The paper addresses the challenge of identifying Bug-Introducing Commits (BICs) given a Fix Commit. This is a critical task in software engineering for determining which software versions are affected by security vulnerabilities and for understanding root causes.

• Context: The problem has been studied since the seminal SZZ algorithm (Śliwerski, Zimmermann, and Zeller, 2005), which traces deleted lines in a fix commit back to the commits that last modified them.
• Limitations of Existing Approaches: Despite two decades of research and refinements (e.g., AG-SZZ, LLM4SZZ), performance gains have been marginal. On the popular Linux kernel dataset, the State-of-the-Art (SOTA) method from 2025 (LLM4SZZ) achieved an F1-score of only 0.64, a mere 10 percentage point improvement over the original 2005 SZZ (0.54).
• Core Issue: Traditional methods struggle with cases where fixes add code rather than delete it, or where the relationship between the fix and the bug is semantic rather than syntactic.

2. Methodology

The authors propose a new paradigm using LLM-based Agents equipped with developer tools (file readers, grep, glob, etc.) to navigate repository histories. The study is divided into two parts:

Part 1: SZZ-Agent (The Agentic Workflow)

The authors first designed SZZ-Agent, a two-stage workflow intended to overcome SZZ limitations:

• Stage 1 (SZZ-based Identification): Uses standard SZZ to generate candidate BICs based on deleted lines. An LLM agent then analyzes the fix commit message, diff, and candidate diffs to select the correct BIC.
• Stage 2 (Binary Search): If Stage 1 fails or finds no candidates, the agent performs a binary search over the file history of modified files. It analyzes source code at midpoint commits to determine if the bug exists, narrowing the search space until a small set of candidates remains for direct selection.

Part 2: Simple-SZZ-Agent (The Discovery)

Through ablation studies, the authors discovered that the complex binary search and SZZ filtering were unnecessary. They proposed Simple-SZZ-Agent:

• Mechanism: It collects the full commit history of all files modified by the fix commit into a single candidate set.
• Process: It provides the agent with the fix commit message, diff, and the entire candidate set. The agent uses developer tools (specifically grep) to search the candidates directly.
• Key Insight: The agent does not read every commit sequentially. Instead, it distills the fix commit message and diff into short, greppable string patterns (e.g., function names, specific code snippets) and uses these patterns to locate the bug-introducing commit in the candidate set.

3. Key Contributions

1. SZZ-Agent: An agentic workflow that outperforms all previous approaches by >12 percentage points in F1-score. The authors conducted five ablation studies to prove this gain stems from the agentic workflow, not data leakage or better LLM backbones.
2. Simple-SZZ-Agent: A drastically simplified agent that removes both the SZZ filtering stage and the binary search. It achieves higher performance than SZZ-Agent with 43% lower token costs.
3. Mechanism Discovery: The paper uncovers why agents succeed: they compress bug knowledge into short, searchable patterns (median length 21 characters) derived from the fix, allowing them to efficiently "needle-in-haystack" search without linearly scaling resource consumption with candidate set size.
4. Generalization: Demonstrated that the approach works across different LLM backbones (Claude, OpenAI, Minimax, GLM) and agent frameworks (Claude Code, OpenHands), including low-cost configurations.

4. Experimental Results

The evaluation was conducted on three datasets: DS_LINUX (Linux kernel), DS_GITHUB-c (C/C++), and DS_GITHUB-j (Java).

• Cost Efficiency: Simple-SZZ-Agent achieves an F1-score of 0.81 at $0.33 per fix commit, whereas the binary search approach (SZZ-Agent) cost $0.67.
• Scalability: Performance and cost remain largely independent of the number of candidate commits (up to ~1,000 candidates). The agent uses targeted grep searches rather than sequential reading.
• Robustness: Even the cheapest configuration (OpenHands + minimax-m2.5) achieved an F1-score of 0.79, outperforming all non-agentic SOTA methods.

5. Significance and Future Directions

• Breaking the Stagnation: The paper breaks the decade-long trend of marginal improvements in BIC identification, demonstrating that agentic workflows can solve problems that static algorithms and simple LLM prompting cannot.
• New Paradigm: It shifts the focus from "refining candidate generation" (filtering) to "agentic search and reasoning." The ability of agents to generate and use greppable patterns is a novel mechanism for navigating large codebases.
• Broader Applications: The authors suggest this "pattern compression" capability can be applied to:
  • Root Cause Understanding: Generating explanations of how bugs were introduced.
  • Bug Detection: Automatically collecting context for bug detection using targeted searches.
  • Program Repair: Using identified bug-introducing changes as context to generate fixes.
• Limitations: The approach currently relies on the bug-introducing commit being in the history of files modified by the fix. If the bug was introduced in a file not touched by the fix, the agent cannot find it. Future work aims to expand the candidate search space beyond touched files.

In conclusion, the paper establishes that LLM-based agents, equipped with simple tools like grep, represent a significant leap forward in software engineering automation, capable of identifying bug-introducing commits with high accuracy and low cost by effectively compressing and searching semantic information.