💻 computer science

Efficient Black-Box Fault Localization for System-Level Test Code Using Large Language Models

This paper introduces a fully static, black-box approach that leverages Large Language Models to localize faults in complex system-level test code without executing the test case, achieving high accuracy while significantly reducing inference time and token usage compared to existing methods.

Original authors: Ahmadreza Saboor Yaraghi, Golnaz Gharachorlu, Sakina Fatima, Lionel C. Briand, Ruiyuan Wan, Ruifeng Gao

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ahmadreza Saboor Yaraghi, Golnaz Gharachorlu, Sakina Fatima, Lionel C. Briand, Ruiyuan Wan, Ruifeng Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a massive, high-tech video conference system. One day, the system crashes during a critical meeting. You have two main suspects:

The System (SUT): The actual video conferencing software (the "engine").
The Test Script (TC): The automated robot you built to check if the engine works (the "inspector").

Usually, when a test fails, developers assume the engine is broken and spend hours tearing it apart to find the problem. But often, the engine is fine! The problem is that the inspector robot is holding the wrong checklist or looking at the wrong thing. It's like a security guard shouting "Intruder!" because they forgot to turn off the motion sensor, not because there's actually an intruder.

This paper introduces a new, super-smart way to find out if the inspector robot is the one who made the mistake, without having to run the test over and over again (which can be expensive, slow, or impossible if the crash happens randomly).

Here is the breakdown of their solution using simple analogies:

1. The Problem: The "Black Box" Mystery

In the real world, testers often don't have access to the "engine's" internal blueprints (source code). They only see the outside. When the robot inspector fails, they get a log file—a long, messy transcript of what happened.

The Old Way: To find the bug, you usually have to run the test 100 times, compare the "good" runs with the "bad" runs, and see what changed. But what if the test costs $1,000 to run? Or what if the crash only happens once in a million tries? You can't do that.
The New Way: The authors use a Large Language Model (LLM)—think of it as a super-intelligent detective who has read every coding book in existence. This detective can look at the messy transcript and the robot's code and say, "Ah, I see the problem right here," without needing to run the test again.

2. The Challenge: Too Much Noise

The problem is that the robot's code is huge and complex. It talks to hundreds of different parts of the system. If you feed the entire code (thousands of lines) to the detective, they get overwhelmed. It's like asking a detective to find a needle in a haystack while blindfolded. Also, reading that much text takes a long time and costs a lot of money (in "tokens," which are the currency of AI).

3. The Solution: The "Pruning" Trick

The authors invented a clever three-step process to help the detective focus only on the important parts. They call this Execution Trace Estimation.

Think of the test code as a choose-your-own-adventure book.

Step 1: The Clue Hunt (Log Matching): The detective looks at the "failure transcript" (the log). They find specific sentences in the code that match sentences in the transcript.
- Analogy: If the log says "Test 1 passed!" and the code has a line print("Test 1 passed!"), the detective knows that specific page of the book was read.
Step 2: Filling the Gaps (The "Fill-in-the-Gaps" Algorithm): The detective looks at the pages between the clues. If the book flows logically from Page 1 to Page 10, and we know Page 1 and Page 10 were read, the detective assumes the pages in between were read too.
Step 3: Cutting the Branches (CFG Analysis): This is the smartest part. The detective looks at the "If/Else" forks in the road.
- Analogy: If the log says "We took the left path," the detective knows the "right path" pages were never read. They physically cut those pages out of the book.
- They also check if the robot called any helper functions. If a helper function was never called, they cut that whole chapter out too.

The Result: They take a 1,000-page book and cut it down to just the 300 pages that were actually involved in the crash.

4. The Detective's Verdict

Now, the detective (the AI) only has to read those 300 pages.

Faster: It takes much less time to read a short book.
Cheaper: It costs less "tokens" to process.
Smarter: Because the book is shorter, the detective isn't distracted by irrelevant information and finds the bug more accurately.

5. The Results: Why It Matters

The authors tested this on real industrial data (from a company like Huawei).

Accuracy: Their "pruned" version of the code was so good that the AI found the bug just as well as if it had read the whole book. In fact, it found the bug 81% of the time in the top 3 guesses.
Speed: By cutting out the useless code, they made the AI 34% faster.
Cost: They reduced the amount of data the AI had to process by 93%.

The Big Picture

Imagine you are trying to find a typo in a 500-page novel, but you only have a note saying "There's a typo on page 400."

Old Method: You read the whole book 10 times to see where the story goes wrong.
This Paper's Method: You use a smart assistant to look at the note, realize the story only goes up to page 400, and then you tear out pages 401 to 500. Now you only have to read the first 400 pages to find the typo.

In summary: This paper teaches us how to use AI to find bugs in our testing tools (not just the software itself) by intelligently ignoring the parts of the code that didn't actually run. It saves time, money, and frustration for developers everywhere.

1. Problem Definition

The paper addresses the critical challenge of Test Code Fault Localization (TCFL) in system-level testing. Unlike traditional Fault Localization (FL) which targets the System Under Test (SUT), TCFL aims to identify faults within the test scripts themselves.

The Challenge: In industrial settings, system-level tests are complex, often black-box (no access to SUT source code), and expensive to execute repeatedly.
Limitations of Current Methods:
- Execution Cost: Traditional techniques (e.g., Spectrum-Based FL) require multiple passing and failing runs to calculate suspiciousness, which is often infeasible due to non-deterministic failures, high execution costs, or lack of test cases for the test code itself.
- Black-Box Constraint: Testers often lack access to the SUT source code, making it difficult to distinguish between a fault in the SUT and a fault in the test logic.
- LLM Context: Existing LLM-based FL methods focus on the SUT and often fail to handle the complexity of system-level test scripts or the specific constraints of black-box testing.
Goal: Develop a fully static, execution-free approach to localize faults in system-level test code using a single failure log, minimizing token usage and inference time while maintaining high accuracy.

2. Methodology

The proposed approach is a static, LLM-driven pipeline consisting of two main phases: Execution Trace Estimation and LLM-based Ranking.

A. Execution Trace Estimation (Static Analysis)

Since the actual execution trace is unavailable or too costly to obtain, the authors propose three novel algorithms to estimate which code statements were executed based on a single failure log and the test code source.

Source-to-Log Matching:
- Identifies Static Log Statements (calls to logging functions with at least one static string literal argument).
- Generates Regular Expressions (Regex) for these statements to match against the execution log messages.
- Unique Matching: If a log message uniquely matches a static log statement, that statement is marked as Executed. If no match is found, it is marked as Non-Executed. Ambiguous matches are handled conservatively.
Fill-in-the-Gaps Algorithm (Line-Level):
- Assumes a linear execution flow between known executed and non-executed points.
- Infers the state of intermediate lines: if a start point is unknown/executed and the end is executed, intermediate lines are marked executed; if the start is non-executed, subsequent lines are non-executed.
CFG-Based Algorithm (Block-Level):
- Constructs a Control Flow Graph (CFG) for each function.
- Identifies "Executable Paths" (pexe) that include all known executed nodes and exclude all known non-executed nodes.
- Prunes blocks that cannot be part of a valid execution path.
Call Site Refinement (CSR):
- Removes function definitions from the estimated trace if their call sites are not present in the trace, further reducing the search space.

B. LLM-Based Fault Localization

Pruning: The estimated trace is used to prune the original test code, removing non-executed lines/blocks.
Prompt Engineering: The pruned code, combined with the error message, is fed into an LLM (Qwen2.5-72B in the study).
Granularity: The system supports three levels of localization: Function, Block (basic block), and Line.
Output: The LLM ranks the top- $k$ most suspicious elements based on the probability of being the root cause.

3. Key Contributions

First Automated TCFL Technique: The first work to propose an automated, LLM-based solution specifically for localizing faults in system-level test code, distinct from SUT fault localization.
Novel Trace Estimation Algorithms: Development of three static algorithms (Fill-in-the-Gaps, CFG-based, and Call Site Refinement) that estimate execution traces with ~90% F1 accuracy without running the code.
Efficiency Optimization: Demonstrates that pruning the input code based on estimated traces reduces LLM inference time by up to 34% and token usage by 93% compared to using full code, without sacrificing accuracy.
Industrial Evaluation: Evaluation on a unique, large-scale industrial dataset of 785 faulty Python system-level test cases (from a codebase of ~5.1M lines) that were not part of the LLM's pre-training data, ensuring no data leakage.
Open Source Tool: Release of the tool and implementation of all algorithms.

4. Experimental Results

The study evaluated the approach using Precision@k, Recall@k, Hit@k, MAP, and MRR metrics against state-of-the-art baselines (FauxPy, IRFL, and FlexFL).

Trace Estimation Accuracy:
- The best combined trace estimator (CSR(T1,dbt=EXE ∩ T2)) achieved an F1 score of ~90% in predicting execution states.
- It successfully preserved at least one faulty location in 87.4% of test cases while pruning 26.1% of the code.
Fault Localization Performance:
- Granularity: Block-level localization offered the best practical trade-off.
  - Top-3 Block-Level: Achieved 81% Hit@3, 34% Precision, and 60.9% Recall.
  - Function-level had higher precision but lower granularity; line-level had lower precision due to the large search space.
- Comparison with Baselines:
  - Outperformed FlexFL (the latest LLM-guided SUTFL method) in efficiency and matched or exceeded its accuracy.
  - Required 85% less inference time and 93% fewer tokens than FlexFL.
  - Significantly outperformed traditional IRFL and FauxPy, especially at finer granularities (block/line).
Efficiency Gains:
- Using the most aggressive pruning (Tmax) reduced inference time by 34% per test case.
- Total inference time for the dataset dropped from ~26 hours (FlexFL) to ~4 hours (TCFL) for block-level FL.

5. Significance and Impact

Practicality for Industry: The technique is highly relevant for CI/CD pipelines and industrial QA teams where test execution is costly, non-deterministic, or where the SUT source code is unavailable (black-box testing).
Cost Reduction: By drastically reducing token usage and inference time, the method makes LLM-based debugging scalable and economically viable for large-scale systems.
Shift in Debugging Paradigm: It shifts the focus from "running tests to find faults" to "statically analyzing logs and code to infer faults," addressing a major bottleneck in modern software engineering.
Foundation for Future Research: This work establishes a baseline for execution-free FL and opens avenues for integrating data flow analysis, program slicing, and automated test repair.

In summary, the paper presents a robust, efficient, and practical solution for debugging complex system-level test scripts, proving that static trace estimation combined with LLMs can effectively localize faults without the overhead of repeated test executions.