Verifier Warnings Do Not Improve Comprehensibility Prediction

Technical Summary: Verifier Warnings Do Not Improve Comprehensibility Prediction

Problem Statement

Software developers spend a significant portion of their time attempting to understand existing code. To manage this, researchers use machine learning (ML) to predict code comprehensibility (the effort required to understand code). While traditional models rely on syntactic features (e.g., lines of code, cyclomatic complexity) and developer features (e.g., years of experience), recent meta-analyses suggested that formal verifier warnings—which signal semantic complexity or "accidental complexity"—might provide a valuable semantic signal to improve these ML models. The core problem addressed by this paper is whether incorporating the sum of verifier warnings as a feature actually enhances the predictive accuracy of code comprehensibility models.

Methodology

The authors conducted a control–treatment experiment to isolate the impact of verifier warnings.

1. Datasets: They utilized the two largest available Java comprehensibility datasets:
   • DS1 (Scalabrino et al.): 50 OSS methods with 13,860 human measurements across five proxies (AU, PBU, ABU50%, BD50%).
   • DS2 (Buse and Weimer): 100 OSS snippets with 12,100 readability (RL) measurements.
2. Experimental Setup:
   • Control Setting ($C$): ML models trained using only syntactic and developer features.
   • Treatment Setting ($T$): The same models trained with the addition of a new semantic feature: the sum of warnings from four state-of-the-art Java verifiers (Checker Framework, OpenJML, JaTyC, and Infer).
3. ML Models: Six different algorithms were evaluated: Naïve Bayes (NB), K-Nearest Neighbors (KNN), Logistic Regression (LR), Multilayer Perceptron (MLP), Random Forest (RF), and Support Vector Machine (SVM).
4. Evaluation Protocol: To ensure robustness, the authors used nested cross-validation (10-fold outer, 5-fold inner) for hyperparameter optimization and evaluated performance using weighted F1-scores ($wF1$) and weighted AUC ($wAUC$). They also performed statistical significance testing using the Mann-Whitney U test.
5. Statistical Analysis: To understand why the feature might fail, they analyzed the relationship between warnings and metrics using Kendall’s $\tau$ (rank correlation), Mutual Information (MI), and SHAP values (model-level contribution).

Key Contributions

• Empirical Validation/Refutation: The study provides a rigorous empirical test of the hypothesis that verifier warnings serve as a useful semantic proxy for human comprehension effort.
• Comprehensive Comparative Analysis: Unlike previous studies that focused on correlation, this work evaluates the predictive utility of these warnings within an ML pipeline.
• Granular Verifier Analysis: The paper investigates whether individual verifiers (rather than just the sum) provide unique signals, offering a more nuanced view of semantic feature engineering.
• Open Science: The authors provided a full replication package, including datasets, code, and experimental infrastructure.

Results

• No Significant Improvement: The primary finding is that adding the verifier warning sum does not significantly improve the predictive performance of the models. Most configurations showed either no change or a slight decline in performance.
• Metric-Specific Observations:
  • Subjective Metrics: Metrics driven by human perception (PBU and RL) showed very small, inconsistent gains, suggesting they might benefit slightly more from semantic data than objective metrics.
  • Difficulty of Prediction: Metrics like Readability Level (RL) remained extremely difficult to predict regardless of the features used.
• Individual Verifiers: While some individual verifiers (like JaTyC or CF) showed marginal improvements in specific configurations, no single tool provided a consistent or statistically significant advantage across all metrics and models.
• Statistical Independence: Kendall’s $\tau$, MI, and SHAP analyses all confirmed that the verifier warning sum carries a very weak or non-existent signal that the ML models can reliably learn.

Significance

The study has important implications for both researchers and practitioners:

1. For Researchers: It suggests that "warning counts" are a primitive way to represent semantic complexity. Future work should move toward richer semantic representations, such as control-flow/data-flow relations or specific types of logical violations, rather than simple aggregate counts.
2. For Practitioners: It highlights that current ML-based tools for estimating code comprehensibility are not yet reliable enough for real-world software maintenance.
3. Theoretical Insight: The results suggest that the "cognitive model" humans use to understand code is highly complex and is not yet fully captured by the combination of syntactic, developer, and basic semantic features currently used in the literature.