💻 computer science

Can Old Tests Do New Tricks for Resolving SWE Issues?

TestPrune is a fully automated technique that strategically minimizes large regression test suites to enhance LLM-based bug reproduction and patch validation, significantly improving issue resolution rates on SWE-Bench benchmarks with minimal API cost overhead.

Original authors: Yang Chen, Toufique Ahmed, Reyhaneh Jabbarvand, Martin Hirzel

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yang Chen, Toufique Ahmed, Reyhaneh Jabbarvand, Martin Hirzel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Too Much Noise, Not Enough Signal

Imagine you are a detective trying to solve a crime in a massive, chaotic city (a large software project). You have a giant notebook of "rules" (the regression test suite) that says, "If you walk down Main Street, you shouldn't get hit by a car." These rules are great for making sure the city doesn't fall apart when you make small changes.

However, when a new, weird crime happens (a new bug), the detective (an AI agent) needs to figure out exactly where and how to fix it. The problem is that the city has thousands of rules. If you try to read all of them to solve one specific crime, you get overwhelmed. It takes too long, costs too much money, and the detective gets distracted by rules about "Main Street" when the crime actually happened on "Elm Street."

In the world of software, these "rules" are tests. Current AI tools try to fix bugs by reading the whole library of tests. This is slow, expensive, and often confusing because most of those tests have nothing to do with the specific bug at hand.

The Solution: TestPrune (The Smart Librarian)

The authors of this paper created a tool called TestPrune. Think of TestPrune as a super-smart librarian who knows exactly which books you need for a specific research topic.

Instead of giving the detective the entire library, TestPrune looks at the description of the crime (the issue report) and the city map (the codebase). It then asks a smart AI (a Large Language Model) to guess which parts of the city are suspicious. Once it knows the suspicious areas, it scans the library of rules and pulls out only the 9 to 11 rules that are actually relevant to that specific crime.

It throws away the thousands of irrelevant rules. It's like going from reading a 10,000-page encyclopedia to reading a single, perfect 3-page cheat sheet.

How It Works (The "New Tricks")

The paper shows that these "old tests" (the ones already written by humans) can do "new tricks" if you just pick the right ones. TestPrune uses two main strategies to help AI fix bugs:

Recreating the Crime Scene (Reproduction):
Before fixing a bug, you need to prove it exists. The AI tries to write a new test that fails on the broken code but passes on the fixed code.
- Without TestPrune: The AI guesses blindly or tries to read too many old rules, getting confused.
- With TestPrune: The AI is handed the specific "old rules" that cover the broken area. This gives the AI better context, helping it write a perfect "crime scene recreation" test.
- Result: The AI gets better at proving the bug exists (a 6.2% to 9.0% improvement).
Checking the Fix (Validation):
Once the AI proposes a fix, it needs to make sure the fix didn't break anything else.
- Without TestPrune: The AI runs thousands of tests. If a test fails, the AI might panic and think the fix is bad, even if that test was irrelevant to the fix.
- With TestPrune: The AI only runs the small, relevant set of tests. If they pass, the fix is likely good. If they fail, the AI knows exactly what to tweak.
- Result: The AI fixes more bugs correctly (8.0% to 12.9% improvement) and does it faster.

The Results: Faster, Cheaper, and Smarter

The paper tested this on real-world software projects (using benchmarks called SWE-Bench Lite and SWE-Bench Verified). Here is what they found:

Massive Reduction: They reduced the number of tests the AI had to run by over 1,000 times. Instead of running 10,000 tests, they only ran about 9.
Speed: Running the full library took about 24 minutes. Running the TestPrune list took only 52 seconds.
Cost: It costs almost nothing extra to use TestPrune. It adds only about $0.02 to $0.05 per bug fix (using standard AI models).
Success Rate: By using this "smart filter," the AI agents solved significantly more bugs than they did before.

The Bottom Line

The paper answers its own title question with a resounding Yes. Old tests can do new tricks, but only if you stop treating them all as equal. By using AI to intelligently select the tiny, perfect subset of old tests that matter for a specific problem, we can make software repair faster, cheaper, and much more accurate.

Key Takeaway: You don't need a bigger library to solve a problem; you just need a better librarian to find the right book. TestPrune is that librarian.

Title: Can Old Tests Do New Tricks for Resolving SWE Issues?
Authors: Yang Chen, Toufique Ahmed, Reyhaneh Jabbarvand, and Martin Hirzel
Venue: Proc. ACM Softw. Eng. 3, FSE (July 2026)

1. Problem Statement

Real-world software projects possess large regression test suites with high code coverage, yet these suites often fail to detect all bugs, leaving many issues unresolved in open-source trackers. While regression tests are traditionally used to ensure backward compatibility, they can also serve a complementary role in debugging current versions by guiding the reproduction of reported issues and validating candidate patches. However, leveraging these suites for automated debugging presents significant challenges:

Scale and Noise: Full test suites are often too large and noisy to be useful in their entirety, especially within agentic Large Language Model (LLM) pipelines where context limits, inference costs, and reasoning accuracy degrade with excessive input.
The Minimization Gap: Existing test-suite minimization techniques focus on maintaining coverage or bug-detection capabilities of the existing suite. They do not address the specific challenge of issue-based test minimization: selecting a minimal subset of tests relevant to a newly reported issue where the resolving patch is unknown.
The Core Challenge: The set of methods that will eventually be modified to fix an issue is latent and unobservable at the time of test selection. The system must predict these "suspicious" methods based solely on the issue description and select tests that cover them, without knowing the final patch.

2. Methodology: TestPrune

The paper proposes TestPrune, a fully automated technique designed to extract a small, highly relevant subset of regression tests to assist LLM-based debugging workflows. The approach consists of three main stages:

A. Suspicious Function Localization
Instead of attempting precise line-level localization (which risks missing relevant tests), TestPrune uses an LLM to identify a list of "suspicious functions" based on the GitHub issue description and the repository structure. This serves as an approximation of the latent set of methods that will be modified.

B. Test File Retrieval and Coverage Generation

Retrieval: The LLM is prompted to select the top $k$ (set to 10 in experiments) candidate test files from the repository. The prompt includes the issue description, suspicious functions, and test file dependencies to provide semantic context.
Coverage: TestPrune executes the passing test methods from these candidate files to generate line-level coverage mappings against the suspicious functions. If no coverage is found, the process iterates by requesting additional test files.

C. Greedy Minimization Algorithms
Once coverage data is collected, TestPrune applies a greedy algorithm to minimize the test set. It formulates the problem as a weighted minimal hitting set problem (NP-hard) and offers two heuristic variants:

Greedy-Additional: Iteratively selects tests that cover the maximum number of currently uncovered lines in the suspicious functions. It recomputes coverage at each step.
Greedy-Total: Ranks tests based on their total coverage of all suspicious lines, regardless of previous selections, and iteratively adds the highest-ranking tests until no significant improvement is observed.

Tie-Breaking: When multiple tests offer equal coverage, the system selects up to three tests directly; if more exist, an LLM is invoked to break the tie.

Downstream Integration
The minimized test suite is plugged into two types of downstream clients:

Reproduction Test Generation: Providing focused context (e.g., to the Otter framework) to help LLMs generate "fail-to-pass" reproduction tests.
Patch Selection and Validation:
- Agentic Systems (e.g., SWE-agent, Trae Agent): Providing regression test feedback during the agent's iterative loop to guide patch refinement and prevent regressions.
- Non-Agentic Systems (e.g., Agentless): Using the minimized suite to rank and validate candidate patches, filtering out those that break existing functionality.

3. Key Contributions

Problem Formulation: The paper identifies and formalizes the issue-based test minimization problem, defining success metrics (precision and coverage recall) for selecting tests relevant to an unknown patch.
TestPrune Technique: The first approach to automatically localize a minimized, relevant set of regression tests from a repository based on an issue description, leveraging LLMs for localization and coverage analysis for selection.
Comprehensive Evaluation: A rigorous evaluation demonstrating that TestPrune improves both issue reproduction and issue resolution rates across multiple benchmarks and agent frameworks.

4. Experimental Results

The authors evaluated TestPrune on SWE-Bench Lite and SWE-Bench Verified using models like GPT-4o and Claude-3.7-Sonnet.

Efficiency (RQ1): TestPrune reduces the test suite size by over 1,000× compared to the full suite (averaging ~9–11 tests per instance vs. thousands). This results in a 27× reduction in runtime (from ~24 minutes to ~52 seconds per instance).
Quality (RQ2): The minimized tests achieve high relevance:
- Precision: 0.63 (TestPrune-Additional), meaning 63% of selected tests cover the ground-truth buggy functions.
- Coverage Recall: 0.71, indicating the selected tests cover 71% of the lines covered by the full "golden" regression suite.
Issue Reproduction (RQ3): Integrating TestPrune into Otter increased the "fail-to-pass" (F→P) reproduction rate by 6.2% to 9.0% relative to the baseline.
Issue Resolution (RQ4):
- Non-Agentic (Agentless): Replacing default regression tests with TestPrune-selected tests, combined with Otter-generated reproduction tests, improved the issue resolution rate by 9.4% to 12.9%.
- Agentic (SWE-agent, Trae Agent): Integrating TestPrune improved resolution rates by 8.0% to 9.4%.
- Cost Efficiency: The approach reduced the average cost per instance by 8.0% to 23.0% for agentic systems. The overhead of TestPrune itself is minimal, costing approximately $0.02 (GPT-4o) to $0.05 (Claude-3.7-Sonnet) per instance.

5. Significance and Claims

The paper claims that "old tests" can indeed perform "new tricks" if they are strategically selected. The significance of this work lies in:

Bridging the Gap: It demonstrates that regression tests, often viewed as static guards, can be dynamically repurposed to actively guide the debugging of new issues.
Enabling Agentic Workflows: By solving the context limit and noise problems inherent in large test suites, TestPrune makes LLM-based debugging pipelines more efficient, reliable, and cost-effective.
Generalizability: The improvements are observed across diverse frameworks (Otter, Agentless, SWE-agent, Trae) and models, suggesting that the strategy of issue-based test minimization is a generalizable enhancement for automated software engineering.
Cost-Benefit: The authors emphasize that the performance gains are achieved with negligible additional API costs, making the technique practical for real-world deployment.

The paper concludes that TestPrune provides a scalable, effective solution to the issue-based test minimization problem, significantly enhancing the capabilities of current automated software engineering agents.