💻 computer science

IssueExec: A Test-Driven Approach for Localizing Software Engineering Issues

This paper proposes IssueExec, a novel test-driven approach that leverages domain-enhanced test representations and hierarchical trace analysis to bridge the semantic gap between issue descriptions and code, achieving state-of-the-art performance in localizing software engineering issues by significantly improving recall and resolution rates.

Original authors: Jiawei Liu, Yun Lin, Chenyan Liu, Yu Qian, Yiming Liu, Jiaxin Chang, Weinan Zhang, Linpeng Huang

Published 2026-07-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiawei Liu, Yun Lin, Chenyan Liu, Yu Qian, Yiming Liu, Jiaxin Chang, Weinan Zhang, Linpeng Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery in a massive, chaotic library. The library represents a huge piece of computer software, and the mystery is a "bug"—a mistake that makes the software act weird. Usually, when someone reports a bug, they write a note in plain English, like "The map won't load when I click here." Your job is to find the exact page in the library where the mistake is hiding so you can fix it. This is called "issue localization."

For a long time, detectives tried to solve this by reading the plain English note and guessing which page in the library it matched. But this is like trying to find a specific book by looking for the word "map" in a library where the books are organized by "geography," "cartography," and "navigation," and the note just says "map." The words don't match, and the library is too big to search every shelf. The paper you are about to read suggests a brilliant new way to solve this: instead of guessing, use the library's own "practice tests." These are like rehearsal scripts that librarians run to make sure the books are in the right order. The authors suggest that these scripts act as a secret bridge, translating the messy human note into the precise language of the library shelves, making the search for the bug much faster and more accurate.

The Problem: The "Vocabulary Gap"

The authors of this paper, a team of researchers from China and Singapore, noticed a frustrating pattern in how computers try to fix software bugs. When a human says, "The serverless feature isn't working," the computer often looks for code named "serverless." But in the real world, the code might be named something technical like _get_db_cluster_kwargs (which is just a fancy way of saying "get the settings for the database cluster").

It's like if you asked a librarian for a "book about space," and they only looked for books with the word "space" in the title, missing the ones actually about "astronomy" or "cosmology." This mismatch between what humans say (requirements) and how programmers name things (code) creates a huge "semantic gap." Existing tools try to jump this gap directly, but they often stumble, leading to long, expensive searches where the computer guesses wrong.

The New Idea: Tests as "Executable Requirements"

The paper proposes a clever detour. Instead of jumping straight from the bug report to the code, the authors suggest going through the tests.

Think of a software test as a "practice run" or a "rehearsal." A programmer writes a test to check if a feature works. Crucially, the name of the test often sounds exactly like the bug report. If the bug is about "Serverless," the test might be named test_create_serverless_db_cluster.

The authors argue that tests are the perfect middleman because they are executable requirements. They are written in human-readable language (like the bug report) but are also tightly connected to the actual code (because they have to run and pass). By finding the right test first, you create a "two-hop" path:

Bug Report $\rightarrow$ Test (Easy match: both use words like "serverless").
Test $\rightarrow$ Code (Guaranteed match: the test actually runs the code).

The Theory: Reducing the "Confusion"

Before building their tool, the authors did some math to see if this idea actually makes sense. They used a concept called "entropy," which is a fancy way of measuring confusion or uncertainty. Imagine you are looking for a needle in a haystack.

Direct Search: If you just guess based on the bug report, you might have to look at 10,000 needles. That's high confusion.
Test-Mediated Search: If you first find the right "box" (the test) that contains the needle, you might only have to look at 100 needles.

Their calculations showed that using tests as a middleman reduces the "confusion" by an average of 7.73 bits. In plain English, this means the search space becomes significantly smaller and more focused, making it much easier to find the right spot.

The Solution: IssueExec

To put this theory into practice, the team built a tool called IssueExec. It works in three main steps:

Smart Test Retrieval: The tool looks at the bug report and tries to find the matching test. But it knows that programmers use abbreviations and inside jokes. So, it digs into the project's history (like reading old commit messages) to learn that "tz" means "timezone" or "ovr" means "OneVsRestClassifier." This helps it understand the test names better than a standard computer search could.
Trace Analysis: Once it finds the right test, it doesn't just stop there. It runs the test and watches exactly which lines of code the test touches. This creates a "trace," like a breadcrumb trail. However, tests often touch too much code, including boring infrastructure stuff that has nothing to do with the bug.
Noise Filtering: The tool uses a smart AI to look at the breadcrumb trail and filter out the noise. It asks, "Which of these touched lines actually caused the problem?" It ignores the boring parts and highlights the specific functions that are likely the culprits.

The Results: A Big Win

The team tested IssueExec on a famous benchmark called SWE-bench Lite, which contains 300 real-world software bugs. The results were impressive:

Better Accuracy: IssueExec found the correct code location (at the function level) 41.57% more often than the previous best method.
More Fixes: When they plugged IssueExec into an automated fixing system (called Agentless), that system solved 17.72% more bugs than before.
Cost Efficiency: Even though it does extra work (running tests and analyzing traces), it actually saved money compared to other complex methods, costing about 35% less on average per bug.

What It Can't Do (The Limits)

The authors are honest about where their tool might fail.

Missing Tests: If the software project doesn't have a test that covers the specific buggy code, IssueExec can't find it. Their study showed that existing tests cover about 96.98% of the files that need fixing, but that still leaves a small gap (about 33.30% of specific functions) where the tool might get stuck.
Deep Mazes: Sometimes, the code path is so long and twisted (like a deep underground tunnel system) that the tool gets lost in the middle layers and misses the final destination.

Why This Matters

This paper suggests that we don't need to teach computers to be better at guessing human language. Instead, we should teach them to use the tools developers already have: the tests. By treating tests as a bridge between human complaints and computer code, IssueExec turns a chaotic search into a guided tour. It suggests that the future of fixing software bugs isn't about bigger AI models guessing blindly, but about smarter, step-by-step reasoning using the evidence that's already there.

Technical Summary: IssueExec – A Test-Driven Approach for Localizing Software Engineering Issues

1. Problem Statement

Automated software maintenance relies heavily on issue localization, the task of identifying specific code locations (files, modules, or functions) that require modification to resolve a reported issue. Existing approaches predominantly follow a direct issue–code alignment paradigm, attempting to match natural language issue descriptions directly to code elements (e.g., via retrieval or LLM agents).

However, these methods often struggle due to an abstraction gap: issue descriptions express behavioral expectations (requirements), while code identifiers reflect technical implementation details. As illustrated in the paper, an issue requesting "ServerlessV2 support" may not semantically align with a function named _get_db_cluster_kwargs (parameter extraction), leading to weak semantic alignment and localization failures. While requirement documentation could theoretically bridge this gap, it often drifts from implementation. The authors posit that test suites, which must remain synchronized with implementation to pass, serve as a superior intermediate signal.

2. Methodology: The IssueExec Framework

The authors propose IssueExec, a test-mediated localization framework that treats test suites as "executable requirements." The framework operates on a two-hop pathway: Issue $\rightarrow$ Tests $\rightarrow$ Code. It consists of three main stages:

A. Theoretical and Empirical Motivation

Before detailing the framework, the authors establish the viability of this approach:

Theoretical Analysis: Using Hartley entropy, the authors model localization uncertainty. They demonstrate that the two-hop pathway (Issue $\rightarrow$ Tests $\rightarrow$ Trace-constrained Code) reduces uncertainty by an average of 7.73 bits compared to direct retrieval, primarily by compressing the search space.
Empirical Study: A study on 18 Python repositories (929 issues) reveals:
- Existing tests cover 96.98% of ground-truth files and 66.70% of ground-truth functions.
- The two-hop pathway yields stronger semantic connectivity than direct matching in 82.4% of cases.

B. Core Components of IssueExec

To address the challenges of semantic gaps and execution noise, IssueExec implements the following pipeline:

Domain Knowledge Enhancement (Offline Preprocessing):
- Challenge: Generic embeddings fail to capture domain-specific terminology (e.g., "tz" for timezone) or project-specific aliases.
- Solution: The system mines historical commit history to extract domain knowledge ( $D_t$ $D_{t}$ ) for each test. This includes:
  - Commit messages introducing the test (providing requirement-level context).
  - Co-changed entities (code frequently modified alongside the test).
- Result: An enriched test representation $repr(t) = [sig(t); doc(t); D_t]$ that bridges the terminology gap between issues and tests.
Relevant Test Retrieval (Two-Phase Filtering):
- Lexical Filtering: Uses BM25 to reduce the search space from thousands of tests to a manageable candidate set ( $N$ ).
- Semantic Selection: An LLM analyzes the enriched representations to select a small, high-quality subset of tests ( $k \ll N$ ) most relevant to the issue intent.
Trace-Guided Localization:
- Challenge: A single test may execute thousands of functions, including irrelevant infrastructure code, creating noise.
- Solution: The system executes the selected tests in a containerized environment to collect dynamic execution traces. It constructs a directed trace graph ( $G_t$ ) representing caller-callee relationships.
- Denoising: An LLM analyzes the trace graph in conjunction with the issue description to identify "suspicious locations." It filters out incidental infrastructure and highlights requirement-central code, assigning confidence levels (critical, high, medium).
Refinement and Reranking:
- The initial suspicious set is expanded by retrieving structural context (module skeletons) for files containing suspicious locations.
- A final LLM-guided reranking step uses full source code to prioritize candidates, reasoning about root causes, causal relationships, and architectural appropriateness to produce the final ranked list of edit locations ( $L^*$ ).

3. Key Contributions

Formalization of Executable Requirements: The paper formalizes the concept of using test suites as executable proxies for requirements, demonstrating how semantic alignment at the test level and execution grounding at runtime jointly bridge the issue–code gap.
The IssueExec Framework: A procedure-based framework that integrates domain-knowledge-enhanced test representations (via commit mining) and hierarchical trace analysis (via dynamic execution graphs) to localize issues.
State-of-the-Art Performance: Empirical validation showing significant improvements over existing baselines in both localization accuracy and downstream issue resolution.

4. Experimental Results

The framework was evaluated on SWE-bench Lite (300 issues from 11 Python repositories).

Localization Performance (RQ1):
- IssueExec achieves 70.07% file-level Recall@1, outperforming the strongest baseline (Agentless) by 17.78%.
- At the function level, it achieves 41.07% Recall@1, a 41.57% improvement over the strongest baseline (MoatlessTools).
- Improvements are consistent across different LLM backbones (GPT-4o and Claude-3.5-Sonnet).
Issue Resolution Performance (RQ2):
- When integrated into the Agentless pipeline, IssueExec resolves 17.72% more issues (37.67% vs. 32.00% resolution rate).
- The authors attribute this to the provision of highly focused, function-level context, which reduces distraction for downstream patch generation models.
Cost Efficiency (RQ3):
- IssueExec reduces the average cost per issue by approximately 35.47% compared to complex agent-based methods (e.g., SWE-Agent, OrcaLoca), while maintaining superior accuracy.
- It is slightly more expensive than the procedural baseline Agentless due to trace analysis overhead, but the authors argue this is justified by the performance gains.
Ablation Study (RQ4):
- Removing the test-based intermediate step ("w/o Tests") causes a significant drop in F1 scores (e.g., -12.94% at the function level), confirming the centrality of the test-mediated approach.
- Removing trace analysis or domain knowledge also results in substantial performance degradation, validating the necessity of both dynamic execution grounding and semantic enrichment.

5. Significance and Limitations

Significance:
The paper argues that precise, requirement-aware localization is a critical enabler for automated software maintenance. By shifting from direct issue–code matching to a test-mediated approach, IssueExec effectively reduces the abstraction gap and search space entropy. The results suggest that leveraging existing test suites as executable requirements provides a robust, cost-effective pathway for automated issue resolution, particularly for complex, multi-file issues.

Limitations and Failure Modes:
The authors explicitly acknowledge two primary failure modes:

Missing Test Coverage: If the repository's existing tests do not execute the buggy code (33.30% of ground-truth functions in the study fall outside full test coverage), the trace-guided localization cannot reach the target. This is a structural limitation of relying on pre-existing tests.
Long Call Chains: In projects with deep execution pipelines (e.g., build systems), the ground-truth location may be buried beneath high-frequency orchestration nodes. The trace graph may be dominated by these infrastructure layers, obscuring the specific fix site despite the location being technically covered.

The authors suggest future work could combine IssueExec with issue-to-test generation techniques to synthesize bug-triggering tests for cases with insufficient coverage, thereby extending the paradigm's applicability.