💻 computer science

TestWeaver: Execution-aware, Feedback-driven Regression Testing Generation with Large Language Models

TestWeaver is a novel LLM-based regression testing approach that overcomes coverage plateaus by integrating lightweight program analysis—specifically backward slicing, similar test case selection, and execution in-line annotations—to provide focused execution contexts that enhance test generation effectiveness and accelerate code coverage growth.

Original authors: Cuong Chi Le, Cuong Duc Van, Tung Duy Vu, Thai Minh Pham Vu, Hoang Nhat Phan, Huy Nhat Phan, Tien N. Nguyen

Published 2026-01-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Cuong Chi Le, Cuong Duc Van, Tung Duy Vu, Thai Minh Pham Vu, Hoang Nhat Phan, Huy Nhat Phan, Tien N. Nguyen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but slightly absent-minded robot (a Large Language Model or LLM) how to break into a specific room in a massive, complex mansion (a computer program). Your goal is to write a set of instructions (a test case) that will get the robot to walk through a specific door (a line of code) that it has never visited before.

The problem is that the mansion is huge. If you give the robot a map of the entire mansion, it gets overwhelmed, confused, and starts guessing. It might walk into the kitchen, then the library, then the garage, but it keeps missing that one specific door in the attic. This is called the "Coverage Plateau": the robot keeps trying, but it stops finding new rooms because it's just wandering around the same familiar halls.

TestWeaver is a new tool designed to fix this. Instead of just handing the robot the whole map and saying, "Go find that door," TestWeaver acts like a super-smart guide who uses three clever tricks to help the robot succeed.

1. The "Spotlight" Trick (Backward Slicing)

Instead of showing the robot the whole mansion, TestWeaver shines a spotlight only on the hallway that leads directly to the target door. It cuts away all the irrelevant rooms, the basement, and the garden.

The Metaphor: Imagine trying to find a specific key on a messy desk. If you show the robot the whole messy desk, it gets distracted. TestWeaver clears the desk, leaving only the key and the immediate area around it. This stops the robot from getting confused or "hallucinating" (making up things that aren't there) and helps it focus entirely on the path to the target.

2. The "Almost-There" Trick (Closest Test Case)

Sometimes, the robot tries to get to the door but stops just one step short. TestWeaver looks at the robot's previous attempts and finds the one that got closest to the door.

The Metaphor: Imagine the robot tried to climb a ladder but stopped at the third rung. TestWeaver says, "Look, you almost made it! You stopped at rung three. The door is just above rung four. Let's look at exactly where you stopped and figure out how to take that next step." Instead of starting from scratch, it uses the robot's near-miss as a stepping stone to guide the next attempt.

3. The "Live Commentary" Trick (Execution In-lines)

When the robot tries again and fails, TestWeaver doesn't just say, "Try again." It rewrites the instructions with a live commentary. It adds notes right next to the steps, saying things like, "At this moment, the robot is holding a red key," or "The door is locked because the light is off."

The Metaphor: It's like watching a video game with a coach shouting out the stats in real-time. Instead of the robot guessing what the game state is, the coach tells it, "You have 50 health points, and the enemy is to your left." This gives the robot a clear picture of what is actually happening inside the code, helping it understand exactly what change is needed to reach the target.

The Results

The authors tested this system on 35 real-world software projects (like a collection of different mansions). They found that:

Better Coverage: TestWeaver found more "rooms" (lines of code) than the previous best methods. It covered 68% of the code, while the others only managed 61% and 46%.
Faster Success: It reached its peak performance much faster. While other methods got stuck in loops of failure (the plateau) for a long time, TestWeaver kept making progress because its guidance was so specific.
Cost Effective: Even though it was smarter, it didn't cost significantly more money to run because it gave the robot smaller, clearer instructions (less "noise" to process).

In short, TestWeaver stops the robot from wandering aimlessly in a giant mansion. Instead, it gives it a focused spotlight, a reference to its best previous attempt, and a live commentary on the path ahead, ensuring it finally finds that specific door it was looking for.

Technical Summary: TestWeaver

Problem Statement

Recent advances in Large Language Models (LLMs) have shown promise in automating test generation for regression testing. However, these approaches often suffer from limited reasoning capabilities regarding actual program execution. Because LLMs are primarily trained on static code representations, they lack explicit runtime context or behavioral data. This limitation manifests as the coverage plateau: a phenomenon where, after initial rapid gains, the rate of discovering new execution paths diminishes significantly. Existing LLM-based approaches typically address this through "passive feedback," simply re-prompting the model with a list of uncovered lines without providing constructive guidance. Consequently, LLMs often generate test cases that fail to increase coverage, instead covering already-explored lines or hallucinating execution states due to the complexity of the full program context.

Methodology: TestWeaver

TestWeaver is a coverage-guided, LLM-based framework designed to overcome the coverage plateau by integrating lightweight program analysis (PA) to create a focused, execution-aware context. The system operates through a pipeline involving three core mechanisms:

1. Backward Program Slicing (Context Reduction)

Instead of feeding the entire program (which may be long and complex) into the LLM, TestWeaver extracts a dynamic backward slice from the specific target line of interest.

Mechanism: The system builds static Program Dependence Graphs (PDGs) and System Dependence Graphs (SDGs) to trace inter-procedural dependencies. Upon execution of a test case, it marks executed nodes and edges. The slicing algorithm then traverses the SDG backward from the target line, collecting only the statements that influence its execution.
Benefit: This reduces the input size to the LLM, minimizing distractions from irrelevant code (e.g., unrelated loops) and reducing the risk of hallucinations. It forces the LLM to focus on the critical control and data dependencies required to reach the target line.

2. "Closest" Test Case Retrieval (Constructive Feedback)

When a generated test case fails to cover a target line, TestWeaver avoids the passive strategy of simply listing all uncovered lines. Instead, it retrieves a "closest" test case from the current suite to provide constructive feedback.

Definition of "Closest": A test case $T$ is considered close to a target line $l_t$ if it executes a control-dependent condition of $l_t$ and covers the condition that is closest in source line distance to $l_t$ .
Mechanism: The system uses static control dependency analysis to identify the conditions governing $l_t$ . It then iterates through the test suite to find a case that executes one of these conditions but diverges from the path leading to $l_t$ .
Benefit: This provides the LLM with a relevant execution context that is structurally similar to the desired path, enabling contrastive reasoning to identify what specific input changes are needed to flip the execution path.

3. Execution In-line Annotations (Dynamic Context)

To further enhance the LLM's reasoning, TestWeaver augments the sliced code with execution in-line annotations.

Mechanism: The "closest" test case is executed, and a tool records the runtime values of variables at each statement. These values are inserted as comments directly into the code slice (e.g., # (k) var1 = v1; var2 = v2).
Benefit: This transforms the static code slice into a dynamic execution trace, allowing the LLM to observe the actual evolution of variable states. This helps the model infer the specific conditions required to steer execution toward the uncovered target line, effectively simulating the execution flow without requiring the LLM to predict it from scratch.

The workflow iterates: initial seeds are generated, uncovered lines are identified, and for each line, the system attempts generation using the slice. If unsuccessful, it triggers re-generation using the closest test case and execution in-lines.

Key Contributions

TestWeaver Framework: A novel integration of program slicing, closest-test retrieval, and execution inlining to guide LLMs in generating regression tests.
Focused Execution-Aware Context: A feedback mechanism that moves beyond passive retrying. By retrieving the most relevant failing test case and annotating it with runtime values, the system provides the LLM with the specific dynamic context needed to reason about program behavior.
Extensive Empirical Evaluation: The authors evaluated TestWeaver on the CodaMosa benchmark (35 real-world Python projects, ~100k lines of code) against state-of-the-art baselines (CoverUp and CodaMosa).

Experimental Results

The evaluation, conducted using the DeepSeek-v3-0324 model, yielded the following results:

Coverage Performance: TestWeaver achieved 68% line coverage and 62% branch coverage across the dataset. This outperformed CoverUp (61% line / 47% branch) and CodaMosa (46% line / 23% branch).
Handling Complexity: TestWeaver maintained superior performance across varying repository sizes and cyclomatic complexity levels. It showed particular robustness in complex modules where baselines frequently hit coverage plateaus.
Efficiency: TestWeaver reached its peak coverage 2.76× faster than CodaMosa. Compared to CoverUp, it achieved higher coverage (68% vs. 61%) in less time and with lower token costs.
Ablation Studies: Removing any of the three core components (slicing, closest-test selection, or execution inlining) resulted in a coverage drop of up to 12%, confirming the critical role of each component. Specifically, removing closest-test selection caused the largest drop (6–10%), highlighting the importance of structural similarity in guidance.

Significance and Claims

The paper claims that TestWeaver addresses the long-standing issue of the coverage plateau in LLM-based test generation. By shifting from passive, prompt-only strategies to a hybrid approach that integrates lightweight program analysis and execution-aware feedback, TestWeaver enables LLMs to reason more effectively about dynamic program states.

The authors argue that their approach demonstrates that static and dynamic analyses can meaningfully augment LLM reasoning, moving the field toward hybrid methods that combine symbolic information with execution context. They posit that this not only improves regression testing but also offers a foundation for broader software engineering tasks such as debugging and program repair. The paper emphasizes that the "closeness" metric between execution paths serves as a novel guidance strategy that reduces redundant exploration and hallucinations.