💻 computer science

Project-Aware Validation in Software Defect Prediction: A Controlled Simulation and Real-World Benchmark Study of Evaluation Optimism

This study demonstrates through controlled simulations and a secondary analysis of real-world benchmarks that using pooled random train/test splits in software defect prediction yields systematically optimistic performance estimates compared to project-aware validation methods, highlighting the critical need for evaluation protocols that respect project boundaries.

Original authors: Vladimir Tomilov

Published 2026-09-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vladimir Tomilov

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of software, code is the foundation of everything from banking systems to medical devices. Yet, like any human-made structure, it is prone to cracks and flaws. Software engineers and researchers have long tried to build computer programs that can predict where these flaws, or "defects," are most likely to hide before the software is even released. The goal is to catch errors early, saving time and preventing costly failures. To test if these prediction programs work, researchers typically gather a massive collection of code from many different software projects, mix them all together into one big pile, and then split that pile randomly into a training group and a testing group. If the prediction program does well on the test group, it is assumed to be ready for the real world. This approach is convenient and widely used, but it relies on a hidden assumption: that a piece of code from one project is just as likely to appear in another project as it is to appear in its own. In reality, software projects are distinct ecosystems. They have their own unique histories, coding styles, and teams. A model trained on a mix of these different worlds might be learning the specific quirks of the projects it has seen, rather than learning how to find defects in a brand-new, unseen project.

A recent study by independent researcher Vladimir Tomilov investigates whether this common testing method is giving researchers a false sense of confidence. The study asks a simple but critical question: if we test a prediction model on data it has never seen from a specific project, does it perform as well as the standard tests suggest? To find the answer, the researcher did not rely on a single historical dataset. Instead, he built a controlled simulation where he could create artificial software projects with known rules. In this simulation, he generated data for six different projects, ensuring that each project had its own unique characteristics, much like how real-world software teams operate differently. He then trained prediction models on this data using two different methods. The first method was the standard approach: mixing all the data together and splitting it randomly. The second method was more rigorous: the model was trained on five projects and then tested strictly on the sixth project, which it had never seen before. This "leave-one-project-out" method mimics the real challenge of deploying a tool into a new environment.

The results of the simulation were clear and consistent. When the models were tested using the standard random mix, they appeared to be more accurate than they actually were. For the most complex models tested, the standard method overestimated their ability to find defects by about three percentage points. While three points might sound small, in the world of software prediction, it represents a significant gap between expectation and reality. The study found that the models were essentially memorizing the patterns of the specific projects they were trained on, rather than learning a universal rule for finding bugs. When the researchers forced the models to prove themselves on a completely new project, their performance dropped noticeably. The gap between the optimistic random-split score and the realistic new-project score was not a fluke; it appeared across different types of prediction algorithms and held true even when the researcher adjusted the difficulty of the simulation.

To ensure these findings were not just an artifact of the computer simulation, the researcher also looked at real-world data. He re-examined a published benchmark that covered eleven major open-source software projects. In this real-world check, he compared how well the models performed when tested on the same project they were trained on versus how they performed when tested on a project they had never seen. The pattern was identical to the simulation. In every single one of the eleven projects, the models looked much better when tested on familiar ground than when tested on new ground. The difference was even larger in the real world, with the overestimation ranging from six to sixteen percentage points. This confirmed that the problem is not just a theoretical issue in a simulation, but a genuine phenomenon affecting how we evaluate software tools today.

The study also explored how different types of models reacted to this challenge. It turned out that the more complex models, which are designed to find intricate patterns, were the most susceptible to this overestimation. They were the ones that most eagerly memorized the specific details of their training projects, leading to the biggest drop in performance when faced with new data. Simpler models, which rely on fewer and broader rules, were more stable. They did not perform as spectacularly in the easy tests, but they held up better when the rules of the game changed. This suggests that in the rush to build the most sophisticated prediction tools, researchers might be favoring models that are good at guessing the past but poor at predicting the future.

The implications of these findings are significant for anyone building or using software defect prediction tools. The study does not claim that these tools are useless, but it does argue that the way we measure their success is flawed. If a researcher reports that a new tool is highly accurate based on a random mix of data, they are likely reporting a number that is too high for real-world application. The study suggests that the only fair way to test a tool intended for new projects is to test it on projects it has never seen. This requires a shift in how experiments are designed, moving away from the convenience of mixing all data together and toward a more honest, project-by-project evaluation. By doing so, the software community can avoid the disappointment of deploying tools that look perfect in the lab but struggle in the field, ensuring that the tools we build are truly ready for the complex, diverse world of software development.

Technical Summary: Project-Aware Validation in Software Defect Prediction

Problem Statement
Software defect prediction studies frequently evaluate models using pooled random train/test splits, treating observations from heterogeneous software projects as exchangeable. This protocol assumes that a model's performance on a random split of pooled data estimates its ability to predict defects in future observations. However, this approach fails to address the practical challenge practitioners face: deploying a model on a new project with distinct coding conventions, process characteristics, and defect mechanisms. When training and test data originate from the same projects, evaluation measures interpolation within familiar domains rather than generalization to unseen domains. This mismatch between the statistical unit of sampling (observation) and the unit of deployment (project) introduces "evaluation optimism," where performance metrics are systematically inflated compared to real-world transfer scenarios.

Methodology
The study employs a dual-stream approach to isolate and quantify this optimism:

Controlled Simulation (Primary Stream):
- Design: A synthetic data-generating process creates six distinct project domains, each with 350 instances and 12 continuous software metrics. The design explicitly incorporates covariate shift (project-specific feature means) and concept drift (project-specific perturbations in the coefficients mapping features to defect probability).
- Models: Three model families with distinct inductive biases are tested: Logistic Regression (linear), Random Forest (tree-based ensemble), and Histogram-based Gradient Boosting (non-linear boosting).
- Protocols: Each model is evaluated under two conditions:
  - Pooled Random Split: Standard stratified random splitting of all pooled data.
  - Leave-Project-Out (LPO): Training on $N-1$ projects and testing on the held-out project, repeated for all projects.
- Estimand: The "optimism gap" is defined as $\Delta = \text{Score}(\text{Random Split}) - \text{Score}(\text{LPO})$ .
- Sensitivity Analysis: The magnitude of concept drift (coefficient jitter) is varied (low, moderate, high) to test if the optimism gap scales with distributional differences.
Real-World Benchmark (Secondary Stream):
- Data: A secondary analysis of a published Just-in-Time (JIT) defect prediction benchmark covering 11 open-source projects (Kamei et al., 2016).
- Method: The study extracts the published within-project ROC-AUC values and the cross-project ROC-AUC matrix. For each target project, the gap is calculated as the within-project AUC minus the median cross-project AUC.
- Constraint: This is a secondary analysis of published AUC values, not a re-execution of raw data training, serving as an external validity check rather than an independent causal experiment.

Key Contributions

Definition of Evaluation Optimism: Formalizes the optimism gap as the paired difference between pooled random-split performance and leave-project-out performance under a shared generative process.
Causal Isolation: Demonstrates that protocol-induced optimism exists even when labels are perfectly controlled and the data-generating process is known, isolating the validation unit as the source of bias.
Model and Metric Sensitivity: Compares three model classes and five metrics (ROC-AUC, PR-AUC, MCC, Balanced Accuracy, Brier Score) to determine if the optimism effect is universal or dependent on model architecture and metric choice.
Sensitivity to Drift: Quantifies how the optimism gap evolves as project-level concept drift increases.
Empirical Triangulation: Validates the direction of simulation results against a real-world 11-project benchmark using a reproducible secondary-analysis estimand.

Results

Simulation Findings:
- Pooled random splitting consistently yields higher performance estimates than leave-project-out validation.
- Optimism Gaps (Mean $\Delta$ ):
  - Logistic Regression: +0.019 (ROC-AUC)
  - Random Forest: +0.033 (ROC-AUC)
  - Histogram-based Gradient Boosting: +0.031 (ROC-AUC)
- Statistical Significance: The optimism for Random Forest and Gradient Boosting remains statistically significant after Holm correction ( $p < 0.05$ ). Logistic regression shows a directional but less robust signal.
- Model Ranking: Under the project-aware (LPO) protocol, Logistic Regression outperformed tree-based models, whereas random splits often favored the more complex models. This indicates that validation protocols can alter the relative ranking of algorithms.
- Drift Sensitivity: The optimism gap increases with the magnitude of coefficient drift (from 0.042 at low drift to 0.054 at high drift for Logistic Regression), though the effect plateaus at higher drift levels.
Real-World Findings:
- In the 11-project JIT benchmark, the within-project AUC exceeded the median cross-project AUC for all 11 projects.
- The gap ranged from 0.060 to 0.165, with a mean of 0.093 (95% CI: 0.073–0.113).
- A Wilcoxon signed-rank test confirmed the median gap is significantly greater than zero ( $p = 0.00049$ ).
- The real-world gap (approx. 0.09) is larger than the simulated gap (0.02–0.03), reflecting the added complexity of real-world label noise and temporal dependencies, but confirms the same directional bias.

Significance and Claims
The paper argues that validation protocol is a first-class experimental variable that fundamentally changes the estimand. A pooled random split answers whether a model can discriminate defects within a known mixture of projects, whereas leave-project-out validation answers whether performance transfers to a new project.

Methodological Warning: Validation that mixes observations across project boundaries produces systematically optimistic estimates. This optimism is not merely an artifact of label noise but a structural consequence of the evaluation unit.
Modesty of Claims: The author explicitly states that the simulated effect sizes (0.02–0.03) should not be treated as a universal constant or a correction factor for all real-world datasets. The real-world gap is larger and context-dependent.
Implications:
- For Researchers: Reporting only pooled random-split performance is insufficient for claims of cross-project generalization. Studies must align the validation unit with the deployment unit (e.g., leave-project-out for new project prediction).
- For Practitioners: Selecting models based on random-split performance may lead to choosing models that overfit project-specific structures. Simpler models (like Logistic Regression in this study) may offer better robustness to cross-project shift despite lower interpolation scores.
- For Reviewers: Papers claiming transferability to new projects must provide project-aware validation results.

The study concludes that predictive performance is a property of the model, the data-generating process, and the evaluation unit combined. While the real-world analysis strengthens external validity, the primary contribution is the causal demonstration that the choice of validation protocol alone is sufficient to induce measurable evaluation optimism.

Technical Summary: Project-Aware Validation in Software Defect Prediction

More like this