💻 computer science

Beyond Coverage and Kill Scores: Empirically Measuring Test Suite Behavioural Gaps

This paper introduces an automated approach to quantify "behavioural gaps" by comparing expected behaviors extracted from documentation and code against actual test coverage, revealing that a significant portion of expected behaviors remain untested even in highly covered code and that these gaps are not detected by traditional structural metrics like line coverage or mutation scores.

Original authors: Partha Protim Paul, Reid Holmes

Published 2026-06-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Partha Protim Paul, Reid Holmes

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef who has written a recipe for a perfect chocolate cake. You've written down every step: "Mix the flour," "Add the eggs," and "Bake until golden."

Now, imagine you have a team of tasters (the test suite) who are supposed to check if your cake turns out right.

The Old Way: Counting Steps

Traditionally, software engineers check if the tasters did their job by counting steps.

Code Coverage: Did the tasters taste every single ingredient? (Did they touch the flour? The eggs? The sugar?)
Mutation Score: If we secretly swapped the sugar for salt, would the tasters notice and say, "Hey, this tastes wrong!"?

If the answer is "Yes" to all of them, the old metrics say, "Great job! The cake is perfect."

The Problem: The Missing "Singleton"

The authors of this paper argue that counting steps isn't enough. You can taste every ingredient and still miss the point of the recipe.

They give a real example from a popular software library:

The Recipe (Documentation): A method called emptyArray() is supposed to return an empty box. But the recipe also says, "This box is special; it's the only box of its kind. If you ask for it twice, you get the exact same physical box back, not a new one."
The Taster's Report (The Test): The tasters checked the box. They opened it, saw it was empty, and said, "Pass!" They even checked every line of code used to make the box.
The Gap: The tasters never checked if it was the same box both times. They missed the "special rule" in the recipe.

If a bug later changed the code to make a new box every time, the tasters wouldn't notice because they were only checking if the box was empty, not if it was the same box.

This missing check is called a Behavioural Gap. It's a gap between what the recipe says should happen and what the tasters actually verified.

The New Tool: BFINDER

The researchers built a tool called BFINDER (think of it as a super-smart, robotic recipe inspector).

Reads the Recipe: It uses AI to read the natural language documentation (the recipe) and the code.
Lists Expectations: It writes down a list of everything the code should do (e.g., "Must return an empty box," "Must return the same box every time").
Checks the Tasters: It looks at the existing tests to see which of those expectations were actually checked.
Finds the Gaps: It highlights the things the tasters missed.

What They Found

The team tested this on 10 very popular, well-tested software libraries (like a top-tier bakery chain). Here is what they discovered:

The Tool Works: BFINDER is very good at reading recipes and figuring out what the tasters should be checking. It was right about 93% of the time.
The Gaps are Real: Even in these high-quality, well-tested libraries, 17.5% of the expected behaviors were completely untested. The tasters were busy checking the ingredients but missed the rules.
Robots Miss Them Too: The researchers asked two famous AI test-generators (EvoSuite and ASTER) to write new tests. Even these robots missed 20.6% to 27.1% of the expected behaviors. This proves that missing these "rules" isn't just a human mistake; it's a fundamental blind spot in how we currently test software.
High Scores Don't Save You: This is the most surprising part. They looked at the methods that had 100% coverage (the tasters touched every single line of code). Even there, 38.2% of the methods still had untested behaviors.
- Analogy: You can have a taster who tastes every single crumb of the cake (100% coverage), but if they don't check if the cake is actually chocolate (the behavior), the cake could still be vanilla, and they wouldn't know.

The Big Takeaway

The paper concludes that Code Coverage and Mutation Scores are like checking if the tasters touched the cake. They are useful, but they don't tell you if the tasters actually understood the recipe.

Behavioural Coverage is a new, separate dimension. It asks: "Did we actually verify that the software does what the documentation promised?"

The authors suggest that to truly know if software is safe and correct, we need to measure not just how much code is touched, but whether the intended behavior is actually validated. It's the difference between checking if a car's engine parts are all present (coverage) and checking if the car actually drives down the road (behavior).

Technical Summary: Beyond Coverage and Kill Scores: Empirically Measuring Test Suite Behavioural Gaps

Problem Statement

Traditional test adequacy metrics, such as code coverage and mutation kill scores, are fundamentally implementation-centric. They measure whether a test suite exercises the existing code structure but fail to verify whether the code adheres to its expected behaviors as defined in natural language documentation. Consequently, these metrics are blind to "faults of omission," where missing logic or unhandled requirements yield high structural scores. While developers rely on these metrics to assess quality, there has been no reliable method to detect discrepancies between what a method is documented to do and what the test suite actually validates. This paper addresses the gap between documented intent and validated behavior, questioning whether structural metrics can fully capture test suite adequacy.

Methodology: BFINDER

The authors introduce BFINDER, a proof-of-concept approach designed to extract expected behaviors from natural language documentation and source code, map them to existing test cases, and identify gaps. The process involves three main stages:

Test Case Identification: BFINDER statically scans repositories for JUnit annotations and constructs per-method call graphs. It primarily uses dynamic analysis (via an AspectJ-based agent) to record exact runtime execution paths, linking test cases to the methods they invoke. This creates an enriched call graph connecting test cases to the target method's implementation, Javadoc, and variables.
Behavior Extraction: Using a Large Language Model (LLM), specifically the Qwen3-14B model, BFINDER analyzes the method signature, implementation, and full Javadoc block (including narrative text) to generate a list of discrete, expected behaviors (e.g., "Returns an empty array," "Returns the same singleton instance").
Behavior Mapping: The system uses an LLM-based "judge" to determine which of the extracted expected behaviors are validated by the existing test cases. By analyzing the call graph and test assertions, the model maps test cases to specific behaviors. Any expected behavior not mapped to a validating test case is classified as an Untested Behavior (UTB), representing a "behavioral gap."

Experimental Setup

The study evaluated BFINDER across ten popular open-source Java libraries (e.g., Apache Commons, Gson, jsoup) comprising 8,922 documented methods. The selected projects were chosen for their high structural coverage and mutation kill scores to ensure the findings represent a conservative lower bound on missed behaviors.

Dataset: 20,729 distinct expected behaviors were extracted.
Validation: A two-step validation process was employed. First, a separate LLM judge (Ministral 3:14B) validated the extracted behaviors. Second, human annotators verified a stratified random sample of 100 behaviors to establish ground truth and calculate precision.
Comparators: The study compared developer-written test suites against test suites generated by two state-of-the-art (SOTA) tools: EvoSuite (search-based) and ASTER (LLM-based).

Key Results

1. Feasibility of Extraction (RQ1)

BFINDER demonstrated high precision in extracting expected behaviors. The estimated precision of the extracted behaviors was 93.1% (±2.7%). The validation process confirmed that LLMs can reliably interpret unstructured documentation to identify discrete, testable behavioral guarantees.

2. Prevalence in Developer-Written Suites (RQ2)

Even in mature, well-maintained test suites with high structural coverage, significant behavioral gaps exist.

Gap Rate: At least 17.5% of the detected expected behaviors remained entirely untested.
Consistency: Every project in the dataset exhibited gaps, ranging from 6.6% to 29.3%.
Nature of Gaps: The untested behaviors were not inherently untestable. Common categories included exception handling, side effects, null/empty input handling, object construction rules, and specific algorithmic logic.
Correction: The authors reported five inconsistencies found during analysis to the respective open-source projects, all of which were subsequently fixed.

3. Generalization to Automated Testing (RQ3)

Behavioral gaps are not merely artifacts of human error; they persist in automated test generation.

ASTER: Missed 27.1% of expected behaviors.
EvoSuite: Missed 20.6% of expected behaviors.
Implication: Both search-based and LLM-based generators fail to validate a significant portion of documented intent, suggesting that current automated testing strategies are not immune to these gaps.

4. Relationship to Structural Metrics (RQ4)

The study investigated whether behavioral gaps are simply proxies for low coverage or low mutation scores. The results indicate they are not:

Line Coverage: There is a negligible correlation ( $\rho = 0.112$ ) between line coverage and behavioral gaps. 38.2% of methods with 100% line coverage still contained at least one untested behavior.
Mutation Kill Score: While the correlation is stronger ( $\rho = 0.377$ ), it remains weak ( $R^2 = 0.162$ ). 29.6% of methods with a perfect mutation kill score still had untested behaviors.
Conclusion: High structural scores do not guarantee that documented behaviors are validated. Behavioral coverage acts as an independent, complementary dimension of test adequacy.

Significance and Claims

The paper claims that behavioral coverage is a distinct and necessary dimension of test suite adequacy that complements traditional structural metrics.

Independence: The findings demonstrate that structural metrics (coverage and kill scores) cannot reliably predict whether a test suite validates the intent of the code. A test suite can be structurally "perfect" while leaving critical documented behaviors unverified.
Systemic Limitation: The persistence of gaps in both human-written and automatically generated suites suggests a fundamental challenge in current testing practices, rather than a simple lack of developer attention.
Actionable Insight: By identifying specific untested behaviors (e.g., missing assertions for singleton instances or specific exception conditions), BFINDER provides actionable feedback to strengthen test suites against regression faults that structural metrics would miss.

The authors conclude that while structural metrics tell developers how thoroughly the implementation has been exercised, behavioral gap analysis reveals whether the code was validated consistently with its expected behavior. They provide a public replication package including the BFINDER tool and all identified gaps to facilitate further research.