From pre-training to downstream performance: Does domain-specific pre-training make sense?

Technical Summary: From Pre-training to Downstream Performance

Problem Statement
Deep learning has significantly advanced medical imaging, particularly in diagnostic accuracy and disease detection. However, the relationship between pre-training strategies and downstream performance in medical contexts remains under-explored. While it is established that domain-specific data can improve performance and that self-supervised learning (SSL) addresses the scarcity of labeled data, critical questions persist: How closely must pre-training data align with the target modality to yield benefits? Does pre-training on large, domain-agnostic datasets (e.g., ImageNet) followed by domain-specific fine-tuning outperform direct domain-specific pre-training? Furthermore, how do different architectures (CNNs vs. Transformers) and evaluation metrics (beyond average AUC) influence the reliability and robustness of these models across diverse data collection sites and disease distributions?

Methodology
The study systematically compares Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) across various pre-training and fine-tuning configurations.

• Datasets: Eight datasets were utilized. For pre-training, 100,000 images were randomly sampled from ImageNet (natural images), NIH ChestX-ray, MIMIC-IV-CXR, CovidxCT (chest CT), and OCT (retina). For fine-tuning and evaluation, models were primarily trained on the NIH ChestX-ray dataset (75,312 training, 11,212 validation images, 14 classes). Evaluation was conducted on out-of-distribution (OOD) datasets: VinDr-CXR (Vietnam), PadChest (Spain), and CheXpert (USA), as well as in-distribution tests on NIH ChestX-ray.
• Architectures: The study evaluated ResNet50 (CNN) and ViT-small (Transformer).
• Pre-training Strategies:
  • Initializations: Random vs. ImageNet1k supervised weights.
  • Strategies: Supervised learning vs. Self-supervised learning (specifically GMML, DINO, MoCo-V3, and SiT).
  • Domains: Pre-training was conducted on natural images, chest X-rays, chest CT, and retina OCT to test modality alignment.
• Fine-tuning: All models were fine-tuned for 30 epochs on NIH ChestX-ray using five different random seeds to ensure statistical robustness. The authors argue that 30 epochs represent a brief adjustment rather than prolonged retraining.
• Evaluation Metrics: Beyond the standard Area Under the Curve (AUC), the study assessed reliability using subset accuracy (predicting all labels correctly), false negative rates (underdiagnosis) across decision thresholds, and "Oracle AUC" (performance if uncertain cases are referred to an expert).

Key Contributions

1. Modality Alignment Necessity: The research demonstrates that pre-training on data closely matching the target modality is a prerequisite for significant downstream performance improvements. Pre-training on a healthcare modality that does not align with the target task (e.g., OCT or CT for a chest X-ray task) can negatively impact performance compared to using a larger, domain-agnostic dataset like ImageNet.
2. Architecture Sensitivity: The study confirms that Transformer-based models (ViTs) are more sensitive to initialization than CNNs (ResNet50). While CNNs show robustness to random initialization, ViTs benefit significantly from pre-trained weights.
3. Limitations of Aggregated Metrics: The paper highlights that average performance metrics (like mean AUC) can be misleading. Significant performance disparities exist across specific diseases (e.g., models performed 20 percentage points better on Pleural Effusion than Atelectasis in some configurations) and across different data collection sites.
4. Reliability and Human-in-the-Loop: The study introduces the importance of evaluating models via "Oracle AUC," showing that performance and reliability can be markedly improved if the system can refer uncertain predictions to human experts, a critical feature for clinical deployment.

Results

• Pre-training Impact: Experiments (specifically 3.3–3.8 in Table 2) indicate that only pre-training on data closely aligned with the target modality (e.g., chest X-rays for a chest X-ray task) consistently improves downstream performance over baseline ImageNet initialization. Conversely, pre-training on misaligned medical modalities (e.g., CT or OCT) often resulted in performance equal to or worse than ImageNet pre-training, potentially due to a "knowledge-forgetting issue."
• Self-Supervised vs. Supervised: Self-supervised learning (SSL) can outperform supervised pre-training, but this is context-dependent. In some cases, SSL on domain-specific data (e.g., MIMIC-IV-CXR) yielded the best results (Experiment 3.4), while in others, the gains were marginal or non-existent compared to supervised methods.
• Cross-Site Performance: Models exhibited significant variance when evaluated on OOD datasets (VinDr, PadChest, CheXpert). Some models fine-tuned on NIH data performed surprisingly well on VinDr-CXR, while others struggled. Variance across fine-tuning seeds was notably higher for OOD datasets, raising concerns about model robustness.
• Metric Sensitivity: High AUC scores did not guarantee high subset accuracy or low underdiagnosis rates. For instance, even top-performing models often failed to predict all five disease labels correctly simultaneously. The Oracle AUC analysis revealed that referring uncertain cases to an oracle could significantly boost effective performance.

Significance and Claims
The paper claims to provide a systematic benchmark that clarifies the conditions under which domain-specific pre-training is beneficial. It argues that while domain-agnostic pre-training (ImageNet) serves as a strong baseline, the "one-size-fits-all" approach of pre-training on any large medical dataset is insufficient; the pre-training data must closely align with the target modality to outperform standard baselines.

The authors emphasize that the choice of pre-training strategy interacts with fine-tuning methods and that no single strategy dominates across all datasets. They conclude that for reliable clinical deployment, researchers must move beyond aggregate AUC metrics to evaluate per-disease performance, cross-site robustness, and the potential for human-in-the-loop collaboration (Oracle AUC). The study aims to guide resource prioritization, suggesting that domain-specific fine-tuning is crucial for improving model performance on target tasks, but only when the pre-training data is appropriately matched.