Technical Summary: RadioAI – Physics-Grounded, Interpretable Deep Learning for Multi-Modal Medical Image Classification

Problem Statement

The paper addresses a diagnostic bottleneck caused by the global shortage of radiologists and rising medical imaging volumes. While deep learning has achieved radiologist-level performance on single-modality tasks, existing systems suffer from three critical limitations:

1. Single-Modality Design: Most architectures ignore the fundamental physical differences between imaging modalities (X-ray attenuation in CT, nuclear spin relaxation in MRI, and acoustic backscatter in Ultrasound), failing to span multiple modalities in a single deployable application.
2. Post-Hoc Interpretability: Current explainability methods (e.g., GradCAM, LIME) are applied after training and do not reference physical imaging phenomena, making it difficult for radiologists to verify why features were informative in physical terms.
3. Unvalidated Novelty: Novel architectural components are rarely systematically removed to confirm their contribution, making it impossible to distinguish genuine inductive-bias benefits from hyperparameter luck.

The authors explicitly state they do not claim state-of-the-art accuracy; rather, they aim to demonstrate that physics-grounded component design is empirically validatable and to provide a reproducible evaluation framework for interpretable medical AI.

Methodology

RadioAI is a multi-modal clinical decision support system integrating three physics-grounded deep learning architectures with a unified Streamlit interface and an automatic modality-routing "Gatekeeper."

1. Datasets and Preprocessing

• Chest CT: 1,000 images (4 classes: 3 cancer subtypes, normal). 2D slices were used with 8-fold augmentation to simulate 3D volumes.
• Brain MRI: 7,200 images (4 balanced classes). Preprocessing included denoising, CLAHE, and z-score normalization.
• Breast Ultrasound (BUSI): 780 images (3 classes: benign, malignant, normal). Preprocessing involved grayscale conversion and normalization.

2. Modality-Specific Architectures

Each architecture encodes a specific physical hypothesis as a structural constraint rather than a learned regularizer:

• DPMS-LSW (CT Lung Cancer):
  • Hypothesis: Cancer subtypes are discriminable by 2D texture at multiple spatial scales, and inter-slice context provides signal in 3D volumes.
  • Structure: A Local Path (EfficientNet-B2 + Multi-Scale Spatial Attention) and a Global Path (ResNet-34 + Learnable Slice Weighting + Transformer) are fused via Cross-Attention.
• TARNet (Breast Ultrasound):
  • Hypothesis: Acoustic phenomena (attenuation, backscatter, shadowing) are distinct and removable.
  • Structure: Three named branches:
    1. Attenuation Branch: Models exponential depth decay via a learnable sigmoid curve.
    2. Backscatter Branch: Captures echogenicity texture at multiple scales.
    3. Acoustic Shadow Branch: Models posterior acoustic artifacts.
  • Interpretability: An Acoustic Composition Gate (ACG) assigns weights to these branches, visualized as a clinical radar chart.
• MSCAF (Brain MRI Tumour):
  • Hypothesis: Tumour types are characterized by features at qualitatively different spatial scales.
  • Structure: Three CNN scales (fine texture, boundary sharpness, coarse morphology) fused with a global Transformer branch via Per-Scale Cross-Attention.

3. System Integration

• Gatekeeper: A MobileNetV3-Small model routes any input scan to the correct modality-specific model without clinician input. A confidence threshold of 0.60 triggers an override prompt if the routing is uncertain.
• Interpretability Stack:
  1. Layer 1: GradCAM activation maps (spatial, post-hoc).
  2. Layer 2: TARNet physics radar charts (model-intrinsic, causal, expressed in acoustic terminology).
  3. Layer 3: Structured clinical explanation templates (natural language bridging model output and clinician vocabulary).

4. Ablation Protocol

Nine controlled ablation experiments were conducted across 21 trained models. Components were removed one by one to measure the drop in accuracy, validating whether the physics-grounded design was necessary.

Key Results

The study reports accuracy and macro-F1 scores on held-out test/validation sets, with 95% Wilson confidence intervals calculated to assess statistical robustness.

• CT (DPMS-LSW):
  • Validation Accuracy: 90.28% (4 classes).
  • Ablation: Removing Multi-Scale Attention caused a −5.56% drop, validating the multi-scale texture hypothesis. The Global Path (inter-slice context) showed 0% drop, attributed to the use of pseudo-volumes (augmented 2D slices) lacking true 3D diversity.
• MRI (MSCAF):
  • Validation Accuracy: 89.62% (4 classes).
  • Ablation: Removing the multi-scale structure (SingleScale) caused the largest drop across all experiments (−11.62%), with non-overlapping confidence intervals confirming the multi-scale spatial hypothesis. Interestingly, removing the Transformer branch improved accuracy by +4.26%, indicating the scratch-trained ViT added noise to the 5,600-image dataset.
• Ultrasound (TARNet):
  • Test Accuracy: 83.76% (3 classes), outperforming ResNet50 by +4.27% despite limited training data (546 images).
  • Ablation: Removing the Acoustic Shadow Branch caused the most severe drop (−7.69%). The ranking of ablation impacts (Shadow > Backscatter = Attenuation) mirrored the clinical hierarchy of acoustic discriminators for malignancy.
• Gatekeeper: Successfully routed scans with high confidence, triggering manual override only when confidence fell below 0.60.

Significance and Claims

The paper frames its contribution as a measurement and design validation study rather than a pure accuracy benchmark. Its primary claims are:

1. Empirical Validatability: Physics-grounded component design is empirically validatable. The controlled ablation methodology successfully demonstrated that removing specific physical constraints (e.g., acoustic shadow modeling, multi-scale spatial attention) leads to statistically interpretable accuracy drops.
2. Inductive Bias over Complexity: The results suggest that "physics specificity"—not architectural complexity—drives interpretable benefits. Generic components (like a scratch-trained Transformer on limited data or global paths on pseudo-volumes) often failed to help or actively hurt performance.
3. Unified Deployable System: RadioAI is the first system to span CT, MRI, and Ultrasound in a single application with automatic modality routing and multi-layer interpretability.
4. Model-Intrinsic Explainability: The TARNet radar chart provides a model-intrinsic explanation in clinical acoustic terminology, distinct from post-hoc pixel attribution methods.

The authors acknowledge limitations, including the use of pseudo-volumes for CT, the noise introduced by scratch-trained Transformers on MRI, and the lack of prospective clinical validation. They position the deployed Streamlit application as a research demonstration tool to enable independent reproduction of inference results and ablation methodologies.