Echo: Graph-Enhanced Retrieval and Execution Feedback for Issue Reproduction Test Generation

Echo is an advanced agent that leverages code graphs, automatic query refinement, and execution feedback to generate single, high-quality issue reproduction test cases, achieving a new state-of-the-art success rate of 66.28% on the SWT-Bench Verified dataset.

The Gist

Imagine you are a detective trying to solve a crime, but the only clue you have is a blurry, handwritten note from a witness that says, "Something weird happened with the red car, but I don't know exactly what."

In the world of software, this "red car" is a bug, and the "weird note" is a bug report. Developers often get these vague reports from users. To fix the bug, the developer needs to recreate the exact situation that caused the problem. This is called reproducing the issue.

Usually, a human developer has to read the code, guess what went wrong, and write a specific test script to prove the bug exists. This is slow, boring, and hard.

Enter Echo. Think of Echo as a super-smart, tireless detective robot that can read the messy note, investigate the crime scene (the code), and write the perfect test script to prove the bug exists—all by itself.

Here is how Echo works, broken down into simple analogies:

1. The "Code Map" (Better than Google Search)

Most AI tools try to find clues by just searching for keywords, like typing "red car" into Google. If the code is huge, they often get lost in irrelevant files or miss the real clue.

Echo's Superpower: Echo builds a 3D map of the entire code city. It doesn't just look for the word "car"; it understands that the "engine" is connected to the "wheels," and the "wheels" are connected to the "road."

• The Analogy: Instead of searching a library by book title, Echo has a magical map that shows exactly which book, which page, and which paragraph contains the clue you need. It also has a "smart assistant" that rewrites your search query if the first one wasn't good enough, ensuring it finds the perfect evidence.

2. The "Trial Run" (The Robot's Gym)

Once Echo finds the clues, it tries to write a test script. But here's the catch: writing the script is easy; making sure it actually runs on the specific computer where the bug lives is hard. Different projects have different rules, like different gym equipment.

Echo's Superpower: Echo doesn't just write the script and hope for the best. It has a built-in gym where it actually tries to run the test.

• The Analogy: Imagine a chef writing a recipe. Most tools just print the recipe. Echo actually goes to the kitchen, turns on the stove, and tries to cook the dish. If the stove doesn't work or the ingredients are missing, Echo sees the smoke and knows, "Okay, I need to change the recipe."

3. The "Time-Traveler's Check" (The Ultimate Proof)

How do you know the test you wrote actually proves the bug? Maybe the test failed because of a mistake in the test itself, not the bug.

Echo's Superpower: Echo uses a clever trick. It asks another AI (or a tool) to guess how to fix the bug and creates a "patched" version of the software where the bug is supposedly gone.

• The Analogy: Echo runs its test on the Buggy Version (where the car crashes) and the Fixed Version (where the car drives smoothly).
  • If the test crashes on the Buggy Version but works on the Fixed Version, Echo knows: "Aha! I found the real bug!"
  • If the test crashes on both, or works on both, Echo knows: "My test is wrong. I need to try again."

4. The "One-Shot Wonder" (Efficiency)

Older AI tools would try to solve the problem 100 times, generate 100 different test scripts, and then pick the best one. This is like asking 100 detectives to write a report and hoping one of them gets it right. It's expensive and slow.

Echo's Strategy: Echo focuses on getting it right the first time. It uses the feedback from its "gym runs" and "time-travel checks" to refine one single test until it's perfect.

• The Analogy: Instead of throwing 100 darts at a board and hoping one hits the bullseye, Echo throws one dart, sees where it lands, adjusts its aim slightly, and throws again until it hits the center. It's smarter and uses less energy.

The Result

When the researchers tested Echo on a huge collection of real-world software bugs (called SWT-Bench), it became the champion.

• It successfully recreated the bug for 66% of the cases.
• It beat all other open-source tools, even those using more expensive or powerful AI models.
• It did this while being cheaper and faster than the competition.

In Summary

Echo is a smart system that:

1. Reads the map (Code Graph) to find the right clues.
2. Tries the test (Execution) to see if it works.
3. Checks the fix (Dual-Version) to prove the bug is real.
4. Refines one test until it's perfect.

It turns the difficult, manual job of "recreating a bug" into an automated, reliable process, helping developers fix software faster and more reliably.

Technical Summary

Here is a detailed technical summary of the paper "Echo: Graph-Enhanced Retrieval and Execution Feedback for Issue Reproduction Test Generation."

1. Problem Statement

Developers often struggle to identify the root causes of bugs because issue reports frequently lack reliable, executable test cases that reproduce the failure. Manually writing these "issue-reproducing test cases" is time-consuming and requires deep codebase understanding. While recent Large Language Model (LLM) agents have attempted to automate this, they face three critical limitations:

• Inadequate Context Retrieval: Existing methods often rely on simple hierarchical file localization (e.g., Agentless), which fails to capture deep structural relationships and long-range semantics in large codebases, leading to noisy or incomplete context.
• Execution Setup Challenges: Real-world repositories have implicit, scattered test execution commands and configurations. Many tools assume these are known, causing generated tests to fail simply due to execution environment mismatches.
• Unreliable Verification: Without a reliable "failure oracle," systems struggle to verify if a generated test fails for the correct reason. Most rely on LLMs to judge semantic equivalence between logs and issue descriptions, which is inherently unreliable.

2. Methodology: The Echo Agent

Echo is an autonomous agent designed to generate a single, high-quality issue-reproducing test case per issue. It operates through a six-phase pipeline:

A. Graph-Enhanced Context Retrieval

Instead of simple file search, Echo models the codebase as a heterogeneous directed code graph (stored in Neo4j) containing:

• Nodes: ResourceNodes (files/dirs), ParseNodes (AST nodes), and TextSpanNodes (comments/docs).
• Edges: Relationships like HAS_FILE, HAS_AST, PARENT_OF, etc.
• Automatic Query Refinement: Echo uses an LLM to iteratively refine retrieval queries. It assesses whether the retrieved context (focal code and regression tests) is sufficient. If not, it generates follow-up queries to fill gaps, ensuring the generator receives a compact, high-relevance context set.

B. Patch Generation

Echo generates a potential patch (using the open-source tool Prometheus) to create a "patched version" of the codebase. This patched version serves as a critical oracle for the Fail-to-Pass criterion (the test must fail on the original code and pass on the patched code).

C. Test Case Generation

Using the retrieved context (issue description, focal code, regression tests, and the patch), the LLM generates a self-contained test file.

• Role Prompting: The LLM is prompted to act as a QA automation expert.
• Constraints: The test must be minimal, follow project conventions, and reuse examples from the issue.

D. Autonomous Test Execution

Echo automatically infers the correct test execution command from the repository context (e.g., pytest, unittest) and runs the test in a container.

• Safety Constraints: To prevent the LLM from "cheating" (e.g., modifying files to force a pass/fail), strict rules are enforced: read-only access, no full test suite runs, no environment setup (assumed pre-installed), and immediate termination after the specific test runs.

E. Test Verification & Dual-Version Check

This is Echo's core innovation.

1. Semantic Verification: An LLM verifier checks if the execution logs align semantically with the issue description.
2. Dual-Version Check (Fail-to-Pass): Echo applies the generated patch to the codebase and runs the test on both the original and patched versions.
   • Success: The test fails on the original and passes on the patched version.
   • Refinement Loop: If the test fails the dual-version check, the execution logs are fed back to the generator to refine the test. This loop runs up to two times.

3. Key Contributions

1. Graph-Enhanced Retrieval: Introduction of a code-graph-based retrieval mechanism with automatic query refinement, significantly outperforming traditional file-localization methods in capturing relevant context.
2. Execution Feedback Loop: The first approach to seamlessly integrate automatic test execution and dual-version verification (using a generated patch as an oracle) into the generation loop. This moves beyond simple "fail" detection to rigorous "fail-to-pass" validation.
3. Efficiency via Single-Test Generation: Unlike prior works that generate large pools of candidates and rank them, Echo focuses on generating one high-quality test per issue through iterative refinement, offering a better cost-performance trade-off.
4. Open-Source SOTA: Echo is fully open-source and establishes a new State-of-the-Art (SOTA) on the SWT-Bench Verified benchmark.

4. Experimental Results

Evaluated on SWT-Bench Verified (433 real-world issues from popular GitHub repositories):

• Success Rate: Echo achieved a 66.28% success rate (Fail-to-Pass), surpassing all other open-source and closed-source approaches on the leaderboard.
  • Comparison: Outperformed OpenHands (62.4%) and e-Otter++ (62.1%) by significant margins.
• Uniqueness: Echo solved 33 unique instances that no other top-performing agent could solve, demonstrating its ability to complement existing techniques.
• Refinement Impact: Ablation studies showed that removing the refinement loop dropped the success rate to 59.12%, proving the dual-version check and execution feedback are critical for performance.
• Cost Efficiency:
  • Average cost per instance: $1.54 (using Gemini 2.5 Pro).
  • While slightly more expensive than simple zero-shot methods, it is competitive with other agentic approaches (e.g., e-Otter++ at ~$1.80–$2.75) while delivering higher accuracy.
  • Token consumption is dominated by the "Gen & Refine" stage (69.7%), highlighting the cost of deep reasoning and iterative debugging.

5. Significance

• Practical Workflow Integration: By automating the discovery of execution commands and validating tests against a patched version, Echo bridges the gap between theoretical test generation and practical developer workflows.
• Reliability: The use of a generated patch as an oracle provides a more robust verification mechanism than relying solely on LLMs to interpret error logs.
• Scalability: The "single-test" generation strategy makes the system more efficient and easier to integrate into CI/CD pipelines compared to methods requiring massive candidate pools.
• Future Direction: The paper highlights that while context retrieval is effective, reducing the token cost of long contexts and improving oracle reliability (beyond generated patches) are key areas for future research.

In summary, Echo represents a significant leap forward in automated bug reproduction, combining advanced graph-based retrieval with rigorous execution-based feedback to reliably generate tests that accurately reproduce software defects.