Technical Summary: ReProAgent

Problem Statement

Issue reports in open-source software repositories (e.g., GitHub) rarely include executable tests that reproduce the reported bugs. This absence hinders developers from confirming issues before fixing them and validating patches afterward, significantly increasing debugging and maintenance costs. While recent Large Language Model (LLM)-based approaches have attempted to generate these "reproduction tests" (tests that fail on the buggy code but pass on the fixed code), existing methods largely rely on fixed prompt-based pipelines. These pipelines typically retrieve textual context and directly generate tests, suffering from two fundamental limitations:

1. Limited Context Understanding: They struggle to understand issue behavior across repository-scale codebases, particularly cross-file behavioral dependencies, as they rely primarily on text-based retrieval.
2. Lack of Structured Planning: They treat test generation as a direct mapping from context to code, failing to explicitly model the multi-step reasoning process (bug localization, root cause analysis, test setup construction, and verification) that human developers use.

Methodology: ReProAgent

The authors propose ReProAgent, a tool-augmented, multi-stage agent framework designed to decompose the complex task of reproduction test generation into four distinct agentic stages. Inspired by the manual workflow of developers, the framework integrates three categories of tools to support autonomous exploration and decision-making.

1. Agent Toolset

ReProAgent equips the LLM with a dedicated toolset to facilitate reasoning and interaction:

• Task Decomposition and Reflection Tools: Utilizes Sequential-Thinking to break down complex objectives, maintain reasoning trajectories, and dynamically revise decisions based on intermediate feedback.
• Hybrid Context Retrieval Tools: Combines command-line keyword-based retrieval (e.g., grep, glob) with Knowledge Graph (KG)-based dependency retrieval. The KG is constructed using Tree-sitter to extract program entities (classes, functions) and their semantic relationships, enabling the agent to trace cross-file call chains and dependencies.
• Runtime Interaction Tools: Includes a Docker-based sandbox for executing commands and interactive Python scripts, allowing the agent to observe dynamic execution behavior and validate hypotheses.

2. Four-Stage Agentic Workflow

The generation process is organized into four sequential stages:

• Stage 1: Bug Localization via Hierarchical Analysis
  The agent constructs an initial query from the issue report (symptoms, stack traces) and uses hybrid retrieval to identify suspicious files. It then performs a coarse-to-fine analysis, ranking functions and classes by semantic relevance and refining the search down to specific code lines. The agent provides natural-language rationales and confidence scores for each suspicious region.

• Stage 2: Root Cause Analysis via Execution Path
  Using the suspicious locations and issue description, the agent traces the execution path from the entry point to the failure point. It reconstructs the call chain to identify the underlying logical defect (e.g., incorrect condition checks, boundary handling). The output is a structured report detailing the root cause summary, error category, execution path, trigger conditions, and a confidence score.

• Stage 3: Assertion-Aware Test Planning
  Before generating code, the agent derives a structured test plan. This involves analyzing trigger conditions, constructing the reproduction setup (environment constraints, dependencies), and deriving the necessary assertions. The plan specifies the minimal inputs required to expose the bug and the expected failure signals (e.g., specific exception types or incorrect return values).

• Stage 4: Test Generation via Triadic Review
  The agent generates a candidate test based on the plan and executes it in the sandbox. A Triadic Review process evaluates the test against three criteria:

  1. Semantic alignment with the issue report.
  2. Consistency of the observed failure with the test logic (ruling out syntax or environment errors).
  3. Match between the execution result and the reported issue (error message, exception type).
     If the test fails the review or passes when it should fail, the agent iteratively refines the test using the feedback until a valid reproduction is achieved or the iteration budget is exhausted.

Key Contributions

1. Multi-Stage Agent Framework: The proposal of ReProAgent, which decomposes reproduction test generation into bug localization, root cause analysis, test planning, and test generation, moving beyond fixed prompting pipelines.
2. Task-Specific Toolset: The design of a specialized toolset including sequential thinking for reflection, hybrid retrieval (text + knowledge graph) for context, and runtime interaction for validation.
3. Extensive Evaluation: Comprehensive experiments demonstrating that ReProAgent outperforms state-of-the-art baselines, generalizes across multiple LLM backbones, and improves downstream automated repair tasks.

Experimental Results

The framework was evaluated on SWT-bench-lite and SWT-bench-verified, two benchmarks specifically designed for issue reproduction test generation.

• Performance: ReProAgent achieved a Fail-to-Pass (FP) Rate of 58.43% on SWT-bench-lite and 70.30% on SWT-bench-verified.
• Comparison: It significantly outperformed all baselines. For instance, on SWT-bench-lite, it improved upon the strong baseline AssertFlip by 53.76% and OpenHands (using the same GPT-5-mini backbone) by 20.43 percentage points.
• Generalizability: The framework demonstrated robust performance across different backbone LLMs (GPT-5-mini, Qwen3-Coder, DeepSeek-V3.2, GLM-4.6), with average FP rates ranging from 50.43% to 64.37%, indicating the effectiveness is driven by the framework design rather than a single model.
• Downstream Impact: When integrated with existing repair frameworks (SWE-agent and Agentless), ReProAgent's generated tests improved the number of successfully resolved issues (e.g., increasing SWE-agent's success rate on SWT-bench-lite from 143 to 153).
• Cost: The average cost per instance was $0.14 (using GPT-5-mini), which is competitive with other baselines and affordable for practical application.

Significance

The paper claims that ReProAgent addresses the critical gap in automated software engineering where issue reports lack executable specifications. By mimicking the structured, multi-stage reasoning of human developers and integrating repository-scale context with runtime verification, ReProAgent enables the reliable generation of reproduction tests. This capability not only validates the correctness of patches but also serves as a powerful feedback signal for iterative automated repair workflows, thereby reducing the cost of software debugging and maintenance. The authors emphasize that the framework's success lies in its ability to handle complex, cross-file dependencies and its rigorous triadic review process, which ensures that generated tests faithfully reproduce the reported issues rather than merely generating syntactically correct code.