Technical Summary: Foundation Model Robustness to Technical Acquisition Parameters in Chest X-Ray AI

Problem Statement

While artificial intelligence (AI) for chest radiography has advanced rapidly, concerns regarding algorithmic fairness have predominantly focused on demographic factors (age, sex, race). Technical acquisition parameters, specifically the distinction between anteroposterior (AP) and posteroanterior (PA) views, have received comparatively little attention despite their profound impact on image characteristics and disease severity confounding. AP views, often acquired at the bedside for sicker patients, exhibit cardiac magnification and lower image quality compared to PA views.

The emergence of foundation models (pre-trained on large-scale self-supervised or vision-language datasets) has promised improved generalization and reduced bias. However, it remains unexplored whether these modern architectures exhibit greater robustness to technical acquisition parameter variations than traditional supervised convolutional neural networks (CNNs). Furthermore, it is unknown if any observed robustness advantages generalize across external datasets with different labeling schemes.

Methodology

This retrospective comparative study evaluated four distinct model architectures across two large-scale public datasets: the RSNA Pneumonia Detection Challenge (internal validation, n=26,684) and the NIH ChestX-ray14 (external validation, n=112,120).

Model Architectures

The study compared models representing different training paradigms:

1. DenseNet-121: A traditional supervised CNN trained on multiple chest X-ray datasets.
2. BiomedCLIP: A vision-language model trained on 15 million biomedical image-text pairs from PubMed Central (general biomedical domain).
3. RAD-DINO: A self-supervised vision transformer trained on >800,000 chest radiographs without text supervision.
4. CheXzero: A vision-language model trained on 377,110 MIMIC-CXR chest radiographs paired with radiology reports (chest X-ray specific domain).

Evaluation Metrics and Design

• Primary Outcome: The sensitivity gap between AP and PA views, defined as $Sensitivity(AP) - Sensitivity(PA)$.
• Statistical Analysis: Optimal thresholds were determined via Youden's J statistic. Bootstrap resampling (n=2,000) generated 95% confidence intervals. Permutation testing assessed significance.
• Secondary Outcomes: Area Under the Curve (AUC), calibration metrics (Expected Calibration Error, Brier score), variance decomposition (quantifying the contribution of view type vs. age/sex to performance variance), and inter-model agreement.
• External Validation: Models were tested on the NIH dataset using "Infiltration" as the primary proxy for pneumonia, with sensitivity analyses using "Consolidation" and explicit "Pneumonia" labels.

Key Results

1. Internal Validation (RSNA Dataset)

On the RSNA dataset, CheXzero demonstrated the smallest sensitivity gap (14.3%), significantly outperforming the other models. The ranking of robustness was:

1. CheXzero (14.3% gap)
2. RAD-DINO (25.2% gap)
3. DenseNet-121 (35.7% gap)
4. BiomedCLIP (36.1% gap)

This suggested that domain-specific training (CheXzero) provided superior robustness to view type variation within the training domain.

2. External Validation (NIH Dataset)

External validation revealed a critical limitation: the performance advantage of domain-specific models did not generalize. Model rankings reversed completely:

• RAD-DINO emerged as the most robust, maintaining a consistent gap of 22.3% (a change of only -2.9% from RSNA).
• CheXzero's performance deteriorated dramatically, with the gap increasing from 14.3% to 48.9%.
• BiomedCLIP and DenseNet-121 also showed increased gaps, though less severe than CheXzero.

The non-overlapping 95% confidence intervals between datasets for CheXzero and BiomedCLIP indicated statistically significant performance shifts.

3. Systematic Blind Spots and Variance Decomposition

• Systematic Blind Spot: On the NIH dataset, 31% of PA view pneumonia cases were missed by all four models, compared to only 6% on AP views. This indicates a fundamental challenge in PA pneumonia detection across all architectures.
• Variance Dominance: View type explained 61–100% of the performance variance across models and datasets. In contrast, age explained 0–38%, and sex contributed less than 4%. This confirms that technical acquisition parameters are the dominant source of performance variance, far exceeding demographic factors.
• Label Independence: The view type bias persisted across six different pneumonia-related label definitions (e.g., Infiltration, Consolidation), confirming the bias is not an artifact of specific label choices.

4. Calibration and Clinical Impact

• Calibration: RAD-DINO demonstrated excellent calibration stability across view types. In contrast, vision-language models showed paradoxical calibration behaviors on PA views (statistically aligning with the majority class despite failing to detect positives).
• Clinical Impact: In a hypothetical scenario of 10,000 pneumonia patients, BiomedCLIP resulted in 2,475 excess missed diagnoses on PA views compared to a hypothetical model with equal sensitivity, highlighting significant patient safety risks.

Key Contributions and Claims

The paper makes the following specific claims regarding its findings:

1. Foundation Models Do Not Eliminate Technical Bias: Contrary to assumptions, foundation models do not inherently eliminate biases related to technical acquisition parameters. In fact, domain-specific training can lead to overfitting to specific labeling conventions, reducing external generalizability.
2. Self-Supervised Learning Offers Generalizable Robustness: RAD-DINO, trained via self-supervised learning without text supervision, demonstrated the most consistent robustness across datasets with different labeling schemes. The authors suggest that avoiding text-based learning signals may allow models to learn more fundamental visual features.
3. The "Domain-Specific" Trap: While CheXzero (trained on MIMIC-CXR) performed best internally, its advantage collapsed on external data. The authors argue this reflects overfitting to MIMIC-CXR-specific annotation practices and a semantic mismatch with the NIH dataset's NLP-extracted labels.
4. Acquisition Parameters Trump Demographics: The study provides empirical evidence that view type (AP vs. PA) is a far more significant driver of performance variance than age or sex, challenging the current focus of algorithmic fairness research.
5. Regulatory Implications: The authors claim that current regulatory frameworks, which emphasize demographic subgroup analysis, are insufficient. They argue that acquisition parameter auditing and multi-site external validation must be mandatory for chest X-ray AI regulatory submissions to ensure patient safety.

Significance

The paper concludes that the systematic 31% miss rate on PA pneumonia cases across all models represents a critical patient safety concern. It challenges the prevailing narrative that foundation models are a panacea for generalization issues, demonstrating instead that without rigorous external validation across diverse acquisition parameters and labeling schemes, AI models may fail systematically in clinical deployment. The findings advocate for a shift in development strategies toward self-supervised approaches and a regulatory shift toward auditing technical acquisition biases.