An Audit of Machine Learning Experiments on Software Defect Prediction

Technical Summary: An Audit of Machine Learning Experiments on Software Defect Prediction

Problem Statement
Machine learning (ML) is increasingly proposed as a solution for predicting defect-prone software components. However, the credibility of findings in this field depends critically on experimental design, analysis, and reporting. Despite a growing body of literature, there is mounting evidence of questionable research methods, poor reporting, and a lack of "conclusion stability." Previous reviews have highlighted issues such as data leakage, flawed statistical analyses, and the influence of evaluation metrics on predictive power. Furthermore, the rise of "paper mills" and the difficulty in synthesizing a coherent body of knowledge due to methodological heterogeneity have raised concerns about the reproducibility and replicability of Software Defect Prediction (SDP) research. This paper addresses the need for a systematic technical review (audit) of recent SDP experiments to characterize the current state of practice and evaluate the reproducibility of published findings.

Methodology
The authors conducted a systematic audit of computational experiments in SDP published between 2019 and 2023.

• Data Source: The search was performed on the SCOPUS database using the query: ("software defect" OR bug OR "software fault") AND prediction AND "machine learning". This yielded 7,375 documents.
• Sampling: A stratified random sampling approach was used to select 20 relevant papers per year, resulting in a final sample of 101 papers (61 journal, 40 conference). The inclusion criteria required a focus on predicting software defects, presentation of new experimental results, use of real-world data, full content availability, and a minimum length of 3 pages.
• Audit Instrument: The study evaluated nine specific issues divided into two categories:
  1. Experimental Design and Implementation (Issues 1–5): Lack of Out-of-Sample (OOS) validation, use of problematic metrics (e.g., F1, Accuracy on imbalanced data), absence of benchmarks (better than random), multiple statistical tests without correction, and failure to report result dispersion (spread) alongside central tendency.
  2. Reporting Issues (Issues 6–9): No link to data, no link to code, low reproducibility scores, and articles behind paywalls.
• Reproducibility Assessment: The authors adapted the instrument proposed by González-Barahona and Robles to assess reproducibility. This involved scoring 27 indicators across five categories (Identification, Description, Availability, Persistence, Flexibility) regarding raw data, extraction tools, processed datasets, analysis tools, and results. Scores were normalized to a range of 0–1.
• Analysis: Two independent reviewers audited the papers, resolving disagreements through discussion. Data analysis was performed using R within an RMarkdown notebook.

Key Contributions

1. Systematic Audit: The paper provides the first in-depth technical audit of SDP computational experiments over a five-year period (2019–2023), moving beyond traditional systematic reviews to focus on methodological quality and reporting standards.
2. Quantification of Issues: It quantifies the prevalence of specific methodological and reporting flaws, identifying 427 distinct issues across 101 papers.
3. Reproducibility Baseline: It establishes a baseline for the reproducibility of SDP studies, revealing a wide variance in the availability of data and code.
4. Detection of Anomalies: The audit identified specific examples of "tortured phrases" (e.g., "novel insects" instead of "new bugs") indicative of potential "paper mill" activity, highlighting a threat to research integrity even within curated databases like SCOPUS.
5. Actionable Recommendations: The paper offers 16 specific recommendations for researchers, reviewers, and the community to improve experimental design, reporting, and open science practices.

Results

• Sample Characteristics: The 101 papers covered 74 unique venues. Approximately 50% of the papers were behind paywalls. Citation counts showed high variability, with a median of 4 citations and 25 papers having zero citations.
• Experimental Design Variability: There was significant diversity in the number of datasets used (1 to 365, median 10), learners evaluated (1 to 34, median 5), and metrics used (1 to 9, median 3). The Promise repository datasets were predominant.
• Methodological Flaws:
  • Validation: 34.7% of studies lacked clear OOS validation or had unclear reporting on the matter.
  • Metrics: Despite widespread criticism, 62.9% of studies used the F1 metric, and 39.2% used Accuracy. Only 16.8% used the Matthews Correlation Coefficient (MCC).
  • Chance-Anchoring: Approximately 40% of experiments did not use chance-anchored metrics (like AUC or MCC), making it impossible to determine if results were better than random guessing.
  • Statistical Rigor: Only 45% of studies used formal statistical inference. Of those, only 60% corrected for multiple testing, despite the median number of inferential tests being 24.
  • Dispersion: Many studies reported only central tendency (mean/median) without measures of dispersion (e.g., boxplots), obscuring the variability of stochastic algorithms.
• Reproducibility: The median reproducibility score was 0.52 (out of 1.0). Only one paper was entirely issue-free. Journal papers tended to be more reproducible and contain fewer issues than conference papers. Only 18.8% of studies provided links to analysis code, and 35.8% of existing links were broken.
• Trends: No evidence was found of improvement in methodological quality over the five-year period.

Significance and Claims
The paper claims that the SDP research community has "considerable scope for improvement." The audit reveals that almost half of the studies provide insufficient information for reproduction, and methodological choices often undermine the validity of the findings. The authors emphasize that the identified issues are not necessarily fatal to every paper but are "unnecessary and unhelpful," hindering independent scrutiny and the progress of Open Science.

Crucially, the authors state that the recommended improvements are "neither difficult nor costly to achieve." They argue that adopting chance-anchored metrics, ensuring OOS validation, reporting result dispersion, and sharing code and data are straightforward steps that would significantly enhance the reliability and impact of SDP research. The paper concludes by urging the community to move away from "sloppy experimental methods" and toward more robust, transparent, and reproducible practices, noting that the lack of reliability and insight in such a large body of work (over 1,500 studies in five years) is a call to action rather than a cause for despair.