How Far Are We from Detecting Flaky Tests? On the Limits of Code-Based Detection

Technical Summary: On the Limits of Code-Based Flaky Test Detection

Problem Statement
Flaky tests—tests that pass and fail on the same code version—undermine continuous integration (CI) by eroding trust in test results and wasting developer time. While a significant body of research proposes static, code-based detectors (using lexical patterns, code metrics, or Large Language Models like CodeBERT) to predict flakiness without re-execution, industrial adoption remains limited. Recent studies suggest that reported high performance (e.g., F1 scores > 0.90) may be fragile, stemming from experimental flaws or the fundamental premise that flakiness is a static property of test code. This paper argues that the field is studying the wrong problem: flakiness often arises from runtime interactions with the execution environment, meaning test code alone frequently lacks the necessary information to distinguish flaky from non-flaky tests.

Methodology
The authors conducted a rigorous reproduction and re-evaluation of three prominent code-based detectors (Flakify, FlakyQ, and FlakyXBert) using a single, strict protocol. The study proceeds in three steps:

1. Reproduction and Artifact Audit: The authors reproduced the three detectors using their official replication packages. They audited the evaluation protocols and datasets for data leakage and shortcuts. Specifically, they addressed cross-validation data leakage (where model weights were not reinitialized between folds) and the "fix-commit" shortcut, where non-flaky labels were derived from developer-fixed versions of flaky tests, creating near-duplicate code pairs that models could exploit via superficial similarity rather than flakiness signals.
2. Construction of C-IDoFT (Controlled Counterfactual): To isolate the effect of labeling shortcuts, the authors created C-IDoFT. This dataset retains the developer-confirmed flaky tests from the original IDoFT benchmark but reconstructs the non-flaky class. Instead of using fixed versions of flaky tests, non-flaky labels were derived from 500 repeated executions of the test modules. This dataset serves as a control to test whether code-based detectors can generalize when the "fix-commit" shortcut is removed and projects are separated (project-disjoint evaluation).
3. Construction of FlakeCI (CI-Mined Corpus): To investigate the source of flakiness signals in complex systems, the authors mined over 450,000 CI runs from GitHub Actions to build FlakeCI. This dataset contains 86 confirmed flaky end-to-end (E2E) tests that passed and failed on the same commit. The authors manually analyzed the test code and CI logs to determine if the root cause of the flakiness could be attributed to these artifacts alone.

Key Results

• Reproduction and Data Leakage: When data leakage was fixed (reloading models per fold) and project-disjoint evaluation was applied, the performance of Flakify and FlakyQ collapsed. On the IDoFT benchmark, the weighted F1 score dropped from ~0.95 to 0.80, matching the "always-flaky" majority-class baseline. On the FlakeFlagger dataset, the flaky-class F1 score dropped to 0.07, failing to exceed the baseline. FlakyXBert's performance also dropped significantly (from 0.94 to 0.83) after correcting for notebook outputs that inadvertently used data from different projects.
• C-IDoFT Evaluation: On the controlled C-IDoFT dataset, which removes the fix-commit shortcut and uses rerun-confirmed non-flaky labels, both CodeBERT-based re-implementations (Flakify+ and FlakyLens+) performed no better than constant baselines in a project-disjoint setting. Their flaky-class F1 scores were 0.035 and 0.070, respectively, compared to an "always-flaky" baseline of 0.054. However, under the original random cross-validation protocol (which allows project overlap), the same models achieved high scores (F1 > 0.68), confirming that the apparent detection ability relies on within-project similarity and labeling shortcuts, not generalizable flakiness signals.
• FlakeBench Contrast: The models performed exceptionally well on the curated FlakeBench dataset (F1 > 0.96), suggesting that test code can contain separable signals for specific, curated types of flakiness (e.g., those recognizable from fix messages). However, this success did not transfer to the broader, developer-confirmed flakiness in C-IDoFT.
• FlakeCI Diagnosis: In the analysis of 86 flaky E2E tests, the authors could attribute a cause to only 42% of the failures using the test code and CI logs combined. For the remaining 58%, the root cause (often related to network instability, environment configuration, or runtime state) could not be determined without further execution evidence.

Significance and Claims
The paper claims that the high F1 scores reported in prior literature are largely artifacts of benchmark construction (specifically the fix-commit shortcut) and evaluation protocols (cross-validation data leakage and project overlap), rather than evidence of a model's ability to detect flakiness from code.

The authors argue that:

1. Flakiness is not static: It is not a property inherent to the test code but emerges from runtime interactions. Asking "is this test flaky?" from code alone is ill-posed for many flakiness types.
2. Code-based detection has reached a limit: Replacing traditional machine learning with LLMs does not overcome the limitation that the signal for flakiness is often missing from the source code.
3. Reframing the problem: The field should shift from predicting whether a test is flaky to classifying whether an observed failure is flaky based on execution evidence (logs, stack traces, CI context).

The paper concludes that code-based detectors are currently indistinguishable from majority-class baselines when evaluated rigorously on developer-confirmed flakiness with rerun-confirmed non-flaky labels. The path forward lies in execution-grounded approaches, such as failure-level classification and agents that probe execution environments, rather than static code analysis.