Evaluation-Strategy Gap in Fault Diagnosis of Deep Learning Programs

Technical Summary: Evaluation-Strategy Gap in Fault Diagnosis of Deep Learning Programs

Problem Statement
Deep Learning (DL) programs frequently fail during training due to configuration errors, implementation bugs, and numerical instability. Diagnosing these failures is a costly maintenance task typically addressed by dynamic analysis techniques that learn from runtime signals (e.g., loss trajectories, gradient statistics). However, existing evaluation practices for these techniques predominantly rely on within-program cross-validation, where runs from the same program are split between training and testing sets. This approach risks allowing classifiers to learn program-specific structural patterns rather than generalizable fault semantics. Consequently, it remains unclear how well these techniques generalize to unseen programs (cross-program deployment) or which runtime features (e.g., standard metrics vs. curvature-based metrics) remain effective in such settings.

Methodology
The authors introduce DynFault, a corpus of 5,542 labeled training traces derived from 38 distinct real-world DL programs (FFNN, RNN, CNN). Faults were injected using mutation operators covering seven categories (e.g., hyperparameters, layer configuration, optimizer). The study compares two evaluation strategies:

1. Within-program evaluation: Stratified K-fold cross-validation where runs from the same program appear in both training and test folds.
2. Program-held-out evaluation: Grouped K-fold cross-validation where entire programs are held out from the training set, simulating deployment on unseen codebases.

The study investigates three diagnostic tasks:

• Fault-type classification: Predicting the mutation category.
• Catastrophic instability triage: Detecting numerical divergence (loss > $10^6$).
• Training/Validation (T/V) mismatch detection: Identifying significant gaps between training and validation accuracy.

Two runtime feature configurations were compared:

• Curvature Features (CF): Derived from Hessian-vector products (HVPs), capturing second-order loss landscape properties.
• Optimizer Features (OF): First-order gradient statistics, optimizer states, and activation extrema.

The primary model used is Logistic Regression (LR) to provide a controlled linear probe of feature representations, with comparisons made against Random Forest (RF) classifiers used in prior work.

Key Contributions and Results

1. Quantification of the Evaluation-Strategy Gap (RQ1)
The study identifies a significant performance gap between evaluation strategies. For the four-class fault-type diagnosis task at $k=5$ epochs:

• Within-program balanced accuracy: 0.469
• Program-held-out balanced accuracy: 0.279
• Gap: 0.190 (95% CI: [0.053, 0.258])

Control experiments (program-identity prediction, label permutation, and per-program normalization) confirm that this gap is driven by program-level structure in the features. Classifiers trained under within-program splits exploit unique program signatures rather than fault semantics. When features are normalized per-program, the within-program advantage disappears, but cross-program performance remains low, indicating that the "shortcut" learning is the primary source of the inflated within-program scores.

2. Generalizability of Runtime Features (RQ2 & RQ3)

• Instability Detection: Catastrophic instability predominantly occurs at initialization (96% of unstable runs explode at epoch 0). Curvature features significantly improve cross-program detection of instability. Adding CF to a baseline model increased ROC-AUC from 0.918 to 0.977 at $k=1$ under program-held-out evaluation. An unsupervised clustering rule based on epoch-0 curvature geometry achieved 100% precision in terminating unstable runs, though with low recall on unseen programs.
• Mismatch Detection: Optimizer and activation features substantially improve T/V-mismatch detection within a program (ROC-AUC 0.899 $\to$ 0.971) but degrade significantly on unseen programs (0.826 $\to$ 0.612), creating a generalizability gap of 0.285. Feature ablation studies reveal that per-layer activation statistics are the primary source of these program-specific shortcuts, as removing them narrows the gap. Conversely, standard loss and accuracy metrics alone provide the most stable cross-program performance.

3. Re-evaluation of Existing Techniques (RQ4)
The authors re-implemented four existing fault diagnosis techniques (DeepFD, DEFault, AutoTrainer, DeepDiagnosis) on DynFault. The evaluation-strategy gap was observed across all learning-based techniques. For instance, using Random Forest, the gap widened to 0.34, suggesting that higher-capacity classifiers capture even more program-specific structure, further inflating within-program performance estimates.

Significance and Implications
The paper argues that within-program evaluation substantially overstates the diagnostic capability of DL fault diagnosis techniques for real-world deployment. The findings highlight a trade-off between observability and generalizability: richer logging (e.g., activation statistics) improves performance on known programs but introduces non-transferable shortcuts.

The authors propose three main implications:

1. Align validation with deployment: Cross-program claims require program-held-out evaluation to avoid overestimating performance.
2. Evaluate feature configurations separately: Additional logging channels should be tested under grouped evaluation before deployment, as they may introduce program-specific biases.
3. Monitor instability at initialization: Curvature features offer a generalizable signal for early instability detection, whereas optimizer features are less reliable for cross-program diagnosis.

The study concludes that while some fault-related information generalizes, the reliance on program-specific runtime patterns in current techniques necessitates a shift in evaluation protocols and feature design to ensure robustness in unseen DL programs.