No gain from attention or graph modules in matched ResNet-50, ResNet-18, EfficientNet-B0 and MobileNetV2 baselines for nine-class chest radiographs

Technical Summary: No Gain from Attention or Graph Modules in Matched ResNet-50, ResNet-18, EfficientNet-B0 and MobileNetV2 Baselines for Nine-Class Chest Radiographs

Problem Statement

Deep convolutional networks, particularly ResNet-family backbones pretrained on ImageNet, dominate chest radiograph classification. A prevalent practice in medical imaging literature involves augmenting these backbones with lightweight attention (e.g., SE, CBAM, BAM) or graph modules (e.g., GCN, GAT) to sharpen feature representations and improve accuracy. However, reported gains are often inconsistent, module-dependent, and rarely subjected to rigorous paired statistical testing. Furthermore, when baselines are already strong (e.g., after fine-tuning), the margin for measurable improvement is narrow, making it difficult to distinguish genuine architectural gains from run-to-run variability caused by stochastic optimization and hardware non-determinism.

This study tests the specific, falsifiable hypothesis that five such modules (SE, CBAM, BAM, GCN, GAT) produce a measurable improvement over an identical, frozen-backbone ResNet-50 baseline on a non-saturated nine-class chest radiograph task.

Methodology

The study employs a strictly controlled benchmark design where the inserted module is the sole variable.

• Dataset: The NIH ChestX-ray14 collection was utilized, subsampled to a nine-class subset (atelectasis, cardiomegaly, effusion, infiltration, mass, nodule, pneumonia, pneumothorax, and no-finding). The training set consisted of 9,532 images (capped at 1,200 per class, with full usage for smaller classes) and the validation set 2,383 images. Class weights were applied to mitigate imbalance.
• Models & Architecture:
  • Baseline: ResNet-50 with ImageNet weights. The first two stages were frozen; the last two stages, the inserted module, and the classification head were trained.
  • Variants: Five modules were inserted once at the transition between the backbone and the classification head (post-backbone).
  • Ablation on Placement: For SE and CBAM, two additional placements were tested: (1) In-branch: inside the residual function of every trainable block (before identity addition), replicating original paper designs; and (2) Per-block output gating: applied to the complete output of every trainable block (after identity addition).
  • Backbone Robustness: Experiments were repeated on ResNet-18, EfficientNet-B0, and MobileNetV2 with proportional freezing.
• Experimental Protocol:
  • Seeds: Each model was trained across 10 random seeds to separate variability from module effects.
  • Training: Identical hyperparameters (Adam, LR 1e-4, batch size 16, 224×224 input) were used. No module-specific learning rate tuning was performed.
  • Evaluation: Primary metric was macro-averaged AUC (one-vs-rest).
• Statistical Analysis:
  • Paired Tests: Wilcoxon signed-rank tests on per-seed macro-AUC differences against the baseline, with Benjamini–Hochberg false-discovery-rate (FDR) control.
  • Equivalence Testing: Two One-Sided Tests (TOST) with a pre-specified equivalence margin of ±0.01 AUC.
  • Confidence Intervals: 2,000-iteration bootstrap CIs for ensemble predictions and 5,000-resample bootstrap CIs for paired differences.

Key Results

1. No Measurable Gain: None of the five modules (SE, CBAM, BAM, GCN, GAT) outperformed the identical ResNet-50 baseline. All five fell within the pre-specified equivalence margin of ±0.01 AUC.

   • Negative Findings: CBAM and GAT were statistically significantly worse than the baseline (CBAM: Δ = -0.0028, p=0.0195; GAT: Δ = -0.0053, p=0.0020) after FDR correction, though their effect sizes remained within the equivalence margin.
   • Neutral Findings: SE, BAM, and GCN showed no significant difference from the baseline.

2. Impact of Insertion Position:

   • In-Branch Placement: Inserting SE and CBAM inside the residual branch of every trainable block (the canonical form) resulted in performance equivalent to the baseline (+0.0013 and +0.0016 respectively), with no gain or harm.
   • Per-Block Output Gating: Applying the module to the complete output of every block (gating the identity path) caused significant degradation. SE dropped by -0.0274 and CBAM by -0.0703 against the baseline (10/10 seeds negative, p=0.0020). This penalty was specific to ResNet-50 and did not generalize to other backbones.

3. Robustness Checks:

   • Training Set Size: Increasing the training set 2.4-fold (to 22,898 images) did not yield gains for any module.
   • Backbone Variations: Repeating the experiment on ResNet-18, EfficientNet-B0, and MobileNetV2 confirmed the absence of gain (19 of 21 comparisons equivalent, 2 worse, 0 better).
   • Data Leakage: Switching from an image-level split to a patient-level split (removing patient overlap) reduced the baseline performance slightly (by ~0.01 AUC) but did not alter the conclusion that modules provided no gain.

Key Contributions

• Rigorous Benchmarking: The study provides a multi-seed, paired statistical comparison of attention and graph modules against an identical baseline, addressing the lack of formal testing in prior literature.
• Equivalence Demonstration: By using equivalence testing (TOST), the authors demonstrate that the absence of gain is not merely a failure to reject the null hypothesis but a statistically supported conclusion that the modules do not improve performance beyond a 0.01 AUC margin.
• Placement Sensitivity: The work highlights that insertion position is critical. While the canonical "in-branch" placement is neutral, gating the complete block output is actively harmful for ResNet-50, a nuance often overlooked in standard implementations.
• Methodological Standard: The paper advocates for multi-seed, paired comparisons with equivalence margins as the standard for evaluating small architectural changes in medical imaging.

Significance and Claims

The paper concludes that for the specific regime of fine-tuning a pretrained ResNet-50 (or similar backbones) on a nine-class chest radiograph task, lightweight attention and graph modules should not be assumed to improve performance. The burden of proof lies on the enhancement to demonstrate gains via multi-seed paired comparisons.

The authors explicitly state that their results do not prove attention mechanisms are incapable of helping in all contexts (e.g., segmentation or different datasets), but rather that in this specific, controlled setting, the baseline is strong enough that the modules offer no measurable benefit and, in some configurations (per-block gating), actively degrade performance. The study emphasizes that reported gains in the literature may often be artifacts of run-to-run variation or lack of rigorous statistical control.