Prospective clinical indication, post-hoc report leakage, and fusion design in multi-image chest radiograph classification: a patient-clustered evaluation

Technical Summary: SectionGuard-MI Chest Radiograph Study

Problem Statement
Chest radiograph datasets often pair images with clinical text generated at different stages of the care process. Clinical Indication is recorded before image interpretation, whereas Findings and Impression are written after the radiologist reviews the images. A critical methodological flaw in multimodal learning is "report-label circularity": when models are trained to predict labels derived from a report (e.g., via CheXbert) using that same report as an input, high performance may reflect access to the linguistic source of the label rather than independent image interpretation. Furthermore, multi-image examinations often lack verified view labels (e.g., frontal vs. lateral), relying instead on file-order slots, which may introduce positional bias if models learn fixed-order fusion strategies. This study addresses the need to distinguish prospective prediction (using images and pre-interpretation text) from post-hoc reconstruction, while evaluating fusion strategies for unordered image pairs.

Methodology
The study utilized a subset of 15,000 examinations from the ReXGradient-160K dataset, each containing two readable images. The cohort was split into training (12,253), validation (1,398), and public-test (1,349) sets, with the test set representing 1,136 unique patient clusters.

• Data and Labels: Five observations (atelectasis, cardiomegaly, edema, pleural effusion, consolidation) were extracted using CheXbert from the concatenation of Findings and Impression. Two label mappings were evaluated: U-Ones (positive and uncertain outputs mapped to 1) as the primary endpoint, and U-Zeros (only explicit positives mapped to 1) as a sensitivity analysis.
• Feature Extraction: Frozen encoders were used to prevent fine-tuning on the specific dataset. Images were processed via a frozen ImageNet-pretrained DenseNet-121 (1,024-dim features). Text (Indication and report variants) was encoded using a frozen Bio+ClinicalBERT (768-dim features).
• Model Architectures:
  • Baselines: Single image, two images, Indication-only, and "Ordinary Fusion" (concatenation of modalities).
  • Proposed Model (SectionGuard-MI): A gated multimodal architecture where missing features are zeroed before and after projection to prevent bias. A joint gate network conditions on all projected features and availability indicators. It incorporates ModDrop (randomly dropping secondary image/text branches) and a prediction consistency penalty during training.
  • Comparators: Random-swap training (exchanging image positions during training), DeepSets (permutation-invariant aggregation), and parameter-matched concatenation baselines.
• Leakage Controls: Findings and Impression were used exclusively as post-hoc controls. Experiments included exact and expanded lexical masking to measure residual predictive signal in the report text.
• Evaluation: Models were trained with five random seeds. Uncertainty was estimated using 2,000 patient-cluster bootstrap replicates to account for repeated patients. Primary metrics were macro AUROC and macro AUPRC.

Key Results

• Prospective Performance: Under U-Ones, the Indication-only model achieved a macro AUROC of 0.749, outperforming the two-image model (0.694) and the primary image alone (0.643). The ordinary two-image-plus-Indication fusion reached 0.780.
• SectionGuard-MI Performance: SectionGuard-MI achieved a macro AUROC of 0.783 and a macro AUPRC of 0.260.
  • Compared to ordinary fusion, the AUROC difference was 0.0031 (95% CI: −0.0042 to 0.0104; adjusted p=0.374), which was not statistically significant.
  • The AUPRC difference was 0.0289 (95% CI: 0.0095 to 0.0413; adjusted p=0.004), indicating a significant improvement in precision-recall performance.
  • However, SectionGuard-MI showed worse calibration (higher Brier score and ECE) compared to ordinary fusion and random-swap fusion.
• Alternative Architectures: DeepSets (permutation-invariant) achieved the highest prospective AUROC point estimate (0.787), while random-swap fusion achieved the highest AUPRC (0.265) and best calibration among prospective models.
• Leakage and Masking:
  • Full report text (Findings + Impression) alone achieved near-ceiling performance (AUROC 0.979, AUPRC 0.836).
  • Even after exact and expanded lexical masking of target terms, the masked report models retained AUROC >0.973, demonstrating that descriptive report content encodes observations beyond specific keywords.
  • Adding images to the full report actually reduced performance (AUROC dropped by 0.0258), suggesting feature-level fusion interference in this setting.
• Indication Masking: Masking target terms in the Indication text resulted in negligible performance changes (AUROC difference ~0.001), suggesting the Indication's predictive power stems from broader clinical context (symptoms, history) rather than direct target term leakage.

Significance and Claims
The paper claims to provide a rigorous evaluation framework for multimodal chest radiograph classification that explicitly separates prospective inputs from post-hoc report leakage. Its primary contributions are:

1. Demonstrating Report-Label Circularity: It quantifies how much performance is driven by access to the report text itself, showing that even masked reports retain high predictive power.
2. Evaluating Fusion Strategies: It shows that permutation-invariant (DeepSets) and random-swap approaches are competitive with fixed-order fusion, suggesting that models need not assign permanent clinical meaning to file-order image slots.
3. Nuanced Model Comparison: It highlights that while SectionGuard-MI significantly improved AUPRC over ordinary fusion, it did not improve AUROC and suffered from poorer calibration. This underscores the trade-off between ranking performance and probability calibration in multimodal fusion.
4. Methodological Discipline: The study emphasizes the importance of patient-cluster bootstrapping for uncertainty estimation and the necessity of distinguishing between "report-derived targets" and "independently adjudicated diagnoses."

The authors explicitly state that their results do not establish diagnostic accuracy, clinical utility, or external validity to other datasets, as the targets are derived from reports rather than independent radiologist adjudication. The study serves as a diagnostic evaluation of model behavior under specific data constraints rather than a claim of clinical deployment readiness.