Technical Summary: Cethraian-Shift

Problem Statement

Large-scale public chest radiograph datasets, such as NIH ChestX-ray14 and VinDr-CXR, facilitate reproducible model development but differ significantly in patient populations, acquisition pathways, label sources, and annotation densities. While internal validation is standard, it often fails to expose changes in discrimination, probability calibration, label-policy sensitivity, and post-hoc localization stability that occur when models are evaluated across different datasets or under different target definitions. Existing literature suggests that classifier performance can change materially outside the development setting, yet few studies isolate these variables using a strictly frozen evaluation protocol that separates model training from external testing and interpretation.

Methodology

The study, titled Cethraian-Shift, employs a "frozen evaluation" protocol to audit a fixed family of multi-label chest X-ray classifiers. The core methodology involves training, freezing, and then evaluating models without further modification across distinct datasets and label policies.

1. Model Architecture and Training

• Backbone: Three independent DenseNet-121 classifiers were initialized with ImageNet weights (DenseNet121_Weights.IMAGENET1K_V1).
• Training Data: Models were trained exclusively on the NIH ChestX-ray14 development partition (73,549 images).
• Seeds: Three distinct random seeds (42, 1337, and 2026) were used to generate three separate checkpoints. No seeds were excluded.
• Classes: Six findings were targeted: Cardiomegaly, Pleural effusion, Pneumothorax, Atelectasis, Consolidation, and Pleural thickening.
• Training Protocol: Unweighted binary cross-entropy loss, AdamW optimizer, and early stopping based on validation macro AUROC. No class weighting, focal loss, or oversampling was applied.
• Reproducibility: Deterministic algorithms, fixed random states, and specific hardware configurations (NVIDIA RTX 4080 SUPER) were enforced.

2. Evaluation Datasets and Policies

The frozen checkpoints were evaluated on two distinct datasets under specific constraints:

• NIH Official Test: The held-out test partition of NIH ChestX-ray14 (25,596 images).
• VinDr-CXR: The 15,000-image training partition of VinDr-CXR.
• Label Policies (VinDr): To isolate the effect of target definitions, the same model logits were evaluated against two reader-derived policies:
  • Majority: A positive label requires agreement from >50% of the three available radiologists (at least 2).
  • Threshold-One: A positive label requires agreement from at least one radiologist.
• Calibration: A single scalar temperature per seed was fitted on NIH validation data only. This temperature was applied to all subsequent evaluations (NIH test and VinDr) without refitting.

3. Analysis Dimensions

The study assessed six distinct technical dimensions:

1. Discrimination: Measured via macro AUROC and Average Precision (AP).
2. Calibration: Evaluated using Brier score, Negative Log-Likelihood (NLL), and Expected Calibration Error (ECE).
3. Label-Policy Sensitivity: Quantified by comparing metrics between the Majority and Threshold-One policies on VinDr.
4. Cross-Seed Variation: Analyzed prediction disagreement and probability variance across the three seeds.
5. Localization Overlap: Grad-CAM++ maps were generated from network.features.denseblock4 and compared against VinDr bounding boxes using the "pointing game" and energy-inside-box metrics.
6. Explanation Stability: Pairwise Pearson correlation and Top-Region IoU were calculated between maps generated by different seeds for the same image.

Key Results

Discrimination and Performance

• NIH Official Test: Macro AUROC was 0.798986 ± 0.002181; Macro AP was 0.307838 ± 0.004391.
• VinDr-CXR (Majority): Macro AUROC increased to 0.871625 ± 0.015777, while Macro AP dropped to 0.279830 ± 0.011922.
• VinDr-CXR (Threshold-One): Macro AUROC was 0.843146 ± 0.014760; Macro AP was 0.317946 ± 0.013857.
• Observation: The divergence between AUROC and AP trends across cohorts indicates that ranking discrimination and precision-recall behavior respond differently to cohort prevalence and case composition.

Label-Policy Sensitivity

Changing the target policy from Majority to Threshold-One on VinDr affected 1,809 images and added 2,374 positive image-class targets. The largest shift occurred in Pleural thickening (prevalence increased from 0.059 to 0.132). These metric changes reflect target-definition sensitivity rather than changes in the model's underlying output logits.

Calibration Transfer

Temperature scaling fitted on NIH validation improved calibration on the NIH test set (reducing mean ECE from 0.0287 to 0.0197). However, this improvement did not generalize uniformly:

• VinDr Majority: ECE slightly worsened (0.0292 raw → 0.0318 calibrated).
• VinDr Threshold-One: ECE showed negligible improvement (0.0448 raw → 0.0446 calibrated).
  This demonstrates that validation-fitted calibration does not guarantee improved performance under dataset or target shifts.

Localization and Stability

• Localization Overlap: Pneumothorax showed the highest pointing-game score (0.500), while Pleural thickening showed the lowest (0.046).
• Cross-Seed Stability: Cardiomegaly exhibited the highest map correlation between seeds (0.919), whereas Atelectasis showed the lowest (0.550).
• Atlas Audit: A deterministic atlas of 182 failure cases was selected; 83 were rendered, while 99 remained unavailable due to missing localization maps, highlighting limitations in the reproducibility of post-hoc visualization.

Significance and Claims

The paper positions Cethraian-Shift not as an architectural advancement, but as a procedural audit of model reliability. Its primary contributions are:

1. Decoupling Dimensions: The study demonstrates that discrimination, calibration, label-policy sensitivity, cross-seed variation, and explanation stability behave as distinct technical dimensions that cannot be summarized by a single metric.
2. Frozen Protocol Rigor: By locking the model, preprocessing, and evaluation policies before testing, the study isolates the effects of dataset shift and label definition from model training artifacts.
3. Reproducibility Transparency: The work provides a complete audit trail, including seed-specific results, deterministic atlas selection, and explicit reporting of unavailable data points (e.g., the 99 unrendered atlas records).
4. Limitations of Post-Hoc Explanation: The results reinforce that stable Grad-CAM++ maps do not imply clinical correctness or causal reasoning.

Modest Conclusions

The author explicitly states that these retrospective results do not establish clinical utility, causal reasoning, or deployment readiness. The study does not claim superiority over other architectures or models. Instead, it serves as a technical audit highlighting that:

• Performance on one dataset cannot be assumed to transfer unchanged to another.
• Calibration strategies fitted on internal data may fail or degrade under external shifts.
• Explanation stability varies significantly by pathology and seed, and stability is not a proxy for faithfulness.

The work concludes that such rigorous, multi-dimensional audits are necessary to understand the behavior of medical imaging models before any consideration of clinical deployment.