Overview of the CXR-LT 2026 Challenge: Multi-Center Long-Tailed and Zero Shot Chest X-ray Classification

1. Problem Definition

The paper addresses two critical limitations in current Chest X-ray (CXR) AI research:

• Long-Tailed Distribution: Clinical data is heavily imbalanced, with a few common diseases dominating while rare conditions (the "tail") are underrepresented. Standard deep learning models tend to bias toward common classes, failing to detect rare but critical pathologies.
• Open-World Generalization: Existing benchmarks often rely on "closed-set" assumptions (training and testing on the same known classes). In real-world clinical environments, models must handle Out-of-Distribution (OOD) data, including novel findings or rare diseases never seen during training (Zero-Shot scenarios).

The CXR-LT 2026 challenge was designed to evaluate models on both Robust Multi-Label Classification (known classes) and Open-World Generalization (unseen classes) across multiple medical centers.

2. Methodology and Dataset Design

Dataset Construction

The challenge utilizes a multi-center dataset comprising over 145,000 images from two primary sources to ensure domain shift and diversity:

1. PadChest: The primary source for common diseases and the base distribution of "seen" classes.
2. NIH Chest X-ray: Used to introduce domain shift and additional rare findings not present in PadChest.

Key Data Curation Changes (vs. Previous Editions):

• Radiologist Annotations: Unlike previous versions where labels were extracted from free-text reports, the Development and Test sets in CXR-LT 2026 use high-quality, image-level annotations provided by three radiologists.
• Training Data: The large training set (142,928 images) still relies on automatically extracted labels from Spanish radiology reports, simulating real-world scenarios where massive weakly-labeled data is available for training, but rigorous evaluation requires expert curation.

Task Definitions

The challenge defines two distinct tasks:

• Task 1: Robust Multi-Label Classification (In-Distribution)

  • Goal: Classify 30 "Seen" classes.
  • Classes: Includes 20 overlapping with previous challenges plus 10 new high-prevalence categories (e.g., Aortic Atheromatosis, Hydropneumothorax, Vertebral Degenerative Changes).
  • Evaluation: Uses a development set (1,620 images) and a test set (1,400 images) from the radiologist-annotated PadChest-GR subset.

• Task 2: Open-World Generalization (Zero-Shot / OOD)

  • Goal: Detect 6 "Unseen" classes strictly excluded from training.
  • Classes: Adenopathy, Bulla, Goiter, Infarction, Osteopenia, Scoliosis.
  • Evaluation:
    • Internal Test: 305 images from PadChest-GR.
    • External Test: 79 images from NIH Chest X-ray, validated by three radiologists (requiring consensus of at least two) to ensure high reliability.

Evaluation Metrics

Due to severe class imbalance, mean Average Precision (mAP) is the primary ranking metric. Secondary metrics include mean AUROC (ranking capability) and mean F1-score (threshold-dependent binary decision performance).

3. Key Contributions

1. New Benchmark Standard: Introduced a multi-center dataset with a strict separation between weakly-labeled training data (report-derived) and expert-annotated evaluation data (radiologist-derived), better reflecting clinical AI deployment realities.
2. Zero-Shot Evaluation: Established a rigorous protocol for evaluating models on rare, unseen diseases using both internal and external test sets, moving beyond closed-set assumptions.
3. Comprehensive Leaderboard: Provided a detailed analysis of 72 registered teams, with 21 teams submitting final results, offering a baseline for future research in long-tailed and open-world medical imaging.

4. Results

Task 1: Robust Multi-Label Classification

• Top Performer: CVMAIL × MIHL achieved the highest mAP of 0.5854 and AUROC of 0.9259.
• Observation: There was a significant performance gap between the top team and the rest (2nd place mAP: 0.4827).
• Metric Divergence: The 3rd place team (VIU) had lower mAP/AUROC but the highest F1-score (0.4504). This indicates that while their ranking capability was weaker, their model's decision threshold was better calibrated for binary detection, highlighting the trade-off between ranking metrics and clinical decision utility.

Task 2: Open-World Generalization

• Top Performer: CVMAIL × MIHL again ranked first with an mAP of 0.4315 and AUROC of 0.8073.
• Performance Drop: As expected, performance dropped significantly compared to Task 1, reflecting the difficulty of zero-shot detection under distribution shift.
• General Trend: Beyond the top three teams, performance approached random chance (AUROC ~0.5), indicating that robust open-world detection of rare conditions remains a major unsolved challenge.

5. Significance and Discussion

• Dominance of Foundation Models: The consistent success of the top team suggests that Vision-Language Pre-training (VLP) is the most effective strategy. Aligning visual features with medical text embeddings allows for significant semantic transfer, enabling models to generalize to unseen classes (Zero-Shot) and handle distribution shifts better than traditional CNNs.
• Calibration vs. Discrimination: The results emphasize that high AUROC/mAP (discrimination) does not guarantee optimal clinical utility at a fixed threshold. Models must be carefully calibrated (F1-score optimization) to be clinically useful.
• Future Directions: While foundation models have mitigated the performance drop in zero-shot diagnosis, the paper concludes that reliable detection of rare conditions in open-world settings is still an unresolved problem. The released dataset and baselines aim to drive the development of more trustworthy clinical AI tools.

Conclusion: CXR-LT 2026 successfully demonstrates that large-scale vision-language pre-training significantly improves generalization in long-tailed and zero-shot CXR classification, yet substantial challenges remain in achieving robust, reliable detection for rare diseases in real-world clinical settings.