Development and evaluation of CADe systems in low-prevalence setting: The RARE25 challenge for early detection of Barrett's neoplasia

1. Problem Statement

The paper addresses the critical challenge of Computer-Aided Detection (CADe) in low-prevalence surveillance settings, specifically for the early detection of neoplasia (cancerous or pre-cancerous lesions) in patients with Barrett's Esophagus (BE).

• The Core Issue: In routine clinical surveillance, the vast majority of endoscopic images show normal or benign tissue, while clinically relevant neoplastic lesions are extremely rare (often <1% prevalence).
• The "Prevalence Shift" Gap: Most existing CADe research trains and evaluates models on artificially balanced datasets (e.g., 50% normal, 50% abnormal) to optimize accuracy. However, when these models are deployed in real-world settings with extreme class imbalance (e.g., 100:1 ratio), they suffer from:
  • Optimistic Metrics: High accuracy or AUROC scores that do not translate to clinical utility.
  • False Alarm Fatigue: An unacceptably high number of false positives, which burdens clinicians and can lead to alarm fatigue or missed diagnoses if thresholds are raised to compensate.
• Goal: To create a benchmark that forces researchers to confront training-time class scarcity and deployment-time prevalence shift, evaluating systems under clinically realistic conditions.

2. Methodology: The RARE25 Challenge

The RARE25 challenge was organized as part of the EndoVis/MICCAI 2025 conference to provide a rigorous, prevalence-aware evaluation framework.

A. Dataset Design

• Public Training Set: 3,095 fully anonymized endoscopic images (2,937 non-dysplastic, 158 confirmed neoplasia). This is the largest publicly available BE dataset to date. Images were acquired using Olympus Exera III systems from Dutch medical centers.
• Private Test Set: A "locked" test set of >25,000 images (23,176 non-dysplastic, 3,232 neoplasia). Crucially, this set was sampled to reflect the real-world incidence of Barrett's neoplasia (approx. 100:1 imbalance), rather than a balanced distribution.
• Validation Set: A smaller private cohort (1,143 images) used for internal validation during the challenge.

B. Evaluation Protocol

To address the limitations of standard metrics in imbalanced settings, the challenge employed a prevalence-aware evaluation strategy:

1. Bootstrapping: 1,000 bootstrap iterations were performed where the negative class (normal) was retained, and the positive class (neoplasia) was resampled to maintain a fixed 100:1 imbalance ratio.
2. Key Metrics:
   • AUROC & AUPRC: To measure global discriminative and retrieval performance.
   • PPV@90% Recall (Primary Metric): The Positive Predictive Value at 90% sensitivity. This metric explicitly quantifies the false-positive burden a clinician would face when the system is tuned to catch 90% of all cancers. This is the primary ranking metric.
3. Constraints: Only fully automated methods were allowed. Participants could use public pre-trained models but were restricted to the provided training data (plus public datasets) to ensure fairness.

3. Participating Methods (11 Teams)

The challenge attracted 11 teams from 7 countries. A comparative analysis of their approaches revealed several trends:

• Ensembling: A dominant strategy. Top teams (e.g., IMSY, agaldran, sk) used large ensembles (10–40 models) combining different architectures (ResNet, ViT, ConvNeXt) to reduce variance.
• Pretraining Strategies:
  • Domain-Specific: Many teams (IMSY, MEDAI, iRARE) utilized GastroNet-5M or similar gastrointestinal-specific pretraining, which outperformed generic ImageNet pretraining in several cases.
  • Foundation Models: Teams leveraged large vision transformers (ViT-L, DINOv3) often fine-tuned with parameter-efficient methods (LoRA).
  • Staged Pretraining: Team Jmees-inc used a unique pipeline starting with polyp segmentation, moving to Barrett's segmentation, and finally to classification.
• Loss Functions & Calibration:
  • Most teams relied on supervised classification with modified loss functions (Focal Loss, Asymmetric Loss, Balanced MixUp) to handle imbalance.
  • Explicit Calibration: Rarely used. Only a few teams (IMSY, MEDAI) applied post-hoc calibration (affine re-calibration, Platt scaling) to adjust probability outputs for the expected prevalence.
• Notable Absence: No team submitted an anomaly detection or one-class learning approach. All methods relied on fully supervised classification, despite the dominance of normal data.

4. Key Results

• Performance:
  • Team IMSY achieved the highest PPV@90% Recall (0.035) on the bootstrap median and 0.320 on the full dataset, ranking first.
  • Team UT achieved the highest median AUPRC (0.532) in bootstrapped samples, highlighting that ranking metrics (AUPRC/AUROC) can differ significantly from operating-point metrics (PPV).
  • Discriminative Power: Top models achieved high AUROC (>0.90), indicating they can distinguish between classes well in a theoretical sense.
  • Clinical Reality: Despite high discrimination, PPV remained low (<4%) for all teams at high recall. This confirms that even state-of-the-art models generate a massive number of false positives when deployed in a 1% prevalence setting.
• Robustness: Performance varied significantly across bootstrap samples, indicating that internal cross-validation on small positive sets is an unreliable predictor of real-world performance.
• Qualitative Findings: Top models struggled with subtle, early-stage lesions where visual cues were faint, but performed well on advanced, clearly visible neoplasia.

5. Key Contributions

1. RARE25 Benchmark: The creation of the largest public Barrett's Esophagus dataset (3,095 images) coupled with a massive, prevalence-realistic private test set (>25,000 images).
2. Evaluation Framework: The introduction of a rigorous evaluation protocol that prioritizes PPV at high recall and uses bootstrapping to account for uncertainty in low-prevalence settings, moving beyond standard AUROC/Accuracy.
3. Empirical Evidence of the "Prevalence Gap": The challenge empirically demonstrated that models optimized on balanced data fail to translate to clinical utility without specific prevalence-aware adjustments, as evidenced by the low PPV scores despite high AUROC.
4. Identification of Methodological Gaps: The paper highlights the field's over-reliance on supervised classification and the lack of exploration into anomaly detection or one-class learning, which may be better suited for surveillance tasks where normal data is abundant.

6. Significance and Future Directions

• Clinical Impact: The results serve as a cautionary tale for the deployment of AI in surveillance. High sensitivity (recall) is useless if it comes with an unmanageable false-positive rate that overwhelms clinicians. Future systems must optimize for clinical operating points, not just global metrics.
• Research Direction: The paper advocates for a shift toward prevalence-agnostic approaches (e.g., anomaly detection, self-supervised learning on normal data). These methods could learn a robust model of "normal" mucosa and flag deviations, potentially offering better generalization and lower false-positive rates in rare-event detection.
• Standardization: RARE25 establishes a new standard for evaluating medical AI in low-prevalence scenarios, urging the community to report operating-point-specific metrics and uncertainty estimates alongside traditional ranking metrics.

In summary, the RARE25 challenge successfully exposed the limitations of current CADe development practices in low-prevalence settings and provided a roadmap for developing more robust, clinically viable systems through prevalence-aware benchmarking and methodological innovation.