Validation of an AI-based end-to-end model for prostate pathology using long-term archived routine samples

Technical Summary: Validation of an AI-based End-to-End Model for Prostate Pathology Using Long-Term Archived Routine Samples

Problem Statement
Artificial intelligence (AI) has shown promise in automating prostate cancer histological grading (Gleason scoring and ISUP grade groups), yet its ability to generalize across variations in sample preparation, preservation, and long-term archival storage remains poorly understood. Existing models, particularly those based on foundation models (FMs) pretrained on massive datasets, often suffer from performance degradation when applied to external cohorts with different staining protocols, scanner characteristics, or temporal variations in tissue processing. Furthermore, the specific impact of time-related factors (e.g., slide age, fading, historical processing differences) on AI-based grading is uncharacterized. There is a critical need for rigorous external validation on fully independent, diverse, and temporally extensive datasets to determine if AI models have learned disease-relevant morphological features rather than lab-specific signatures or batch effects.

Methodology
The study conducted an independent, pre-specified validation of GleasonAI, a fully end-to-end, attention-based multiple instance learning (ABMIL) model previously developed by Mulliqi et al. The validation utilized the ProMort I and ProMort II cohorts, derived from the National Prostate Cancer Register of Sweden (NPCR).

• Dataset: The validation set comprised 10,366 biopsy cores from 1,028 patients across 14 Swedish regions, collected between 1998 and 2015. These archival diagnostic specimens were digitized using a modern 3DHistech Pannoramic 250 Flash II scanner and re-annotated by expert pathologists according to contemporary 2014 ISUP guidelines and 2016 WHO classifications.
• Model Architecture: GleasonAI employs an EfficientNet-V2-S encoder to extract feature representations from tissue patches, which are processed by a gated-variant ABMIL aggregator. The model is trained end-to-end (joint optimization of encoder and aggregator) on a large, diverse dataset of over 61,000 whole slide images (WSIs) from three clinical cohorts.
• Evaluation Protocol:
  • Cancer Detection: Sensitivity, specificity, and Area Under the Curve (AUC) were calculated for distinguishing benign from malignant cores.
  • Grading Performance: Agreement with the reference pathologist was measured using quadratic-weighted Cohen's kappa (QWK) and linear-weighted kappa (LWK) for both ISUP grades and Gleason scores.
  • Sensitivity Analyses: Performance was stratified by geographic region (14 regions), sample collection time (17-year span), and inter-observer variability (comparing AI against multiple pathologists).
  • Comparative Analysis: GleasonAI was compared against two general-purpose pathology foundation models (UNI and Virchow2) adapted for the same task using frozen encoders and identical aggregation modules.
  • Prognostic Analysis: An exploratory patient-level survival analysis assessed the association between AI-assigned grades and prostate cancer-specific mortality using Cox proportional hazards regression.

Key Results

• Overall Performance: The model achieved a QWK of 0.86 for ISUP grading across all cores (including benign) and 0.86 for Gleason score classification. For cancer detection, the model demonstrated high sensitivity (96.9% in ProMort I, 96.2% in ProMort II) and specificity (93.2% and 92.1%, respectively), with AUCs of 0.990 and 0.980.
• Geographic Generalization: Performance remained consistent across all 14 geographic regions, with ISUP QWK values ranging from 0.79 to 0.92, indicating robust generalization across diverse laboratory settings.
• Temporal Robustness: Crucially, the model maintained stable performance across the 17-year collection period (1998–2015). ISUP QWK values ranged from 0.83 to 0.87 across temporal subsets, showing no significant degradation on older specimens. In contrast, the foundation model-based pipelines (UNI and Virchow2) exhibited a pattern of declining performance on older specimens, with progressively lower QWK values for samples collected further in the past.
• Inter-Observer Agreement: The AI model's agreement with reference pathologists was comparable to the agreement observed between experienced human uropathologists. In the ProMort I subset (diagnostically challenging cases), the AI ranked third among four observers; in ProMort II, it achieved intermediate agreement levels within the range of human variability.
• Prognostic Utility: The AI-assigned grades demonstrated a clear prognostic gradient for prostate cancer-specific mortality. Higher ISUP grades and specific Gleason patterns (e.g., 4+3 vs. 3+4) showed statistically significant increases in hazard ratios, suggesting the model captures biologically relevant morphological features.
• Segmentation: The tissue segmentation module successfully identified tissue in 99.97% of cores, with manual review confirming that the few failures were largely due to annotation errors rather than model limitations.

Significance and Claims
The paper claims that this study provides the first comprehensive external validation of an AI-assisted cancer diagnosis system across an extensive temporal span (17 years) of archival routine samples. The primary significance lies in demonstrating that task-specific end-to-end models can achieve robustness to temporal variations in tissue processing and preservation, a property not consistently observed with foundation model-based approaches.

The findings support the generalizability of the GleasonAI model across diverse geographic regions and inter-observer variations. Furthermore, the study highlights the underutilized potential of pathology archives accumulated over decades. By proving that digitized historical slides can be directly repurposed for AI development and validation without requiring tissue reprocessing (re-sectioning or re-staining), the authors argue that these archives represent a valuable resource for large-scale AI development and retrospective prognostic research, particularly for diseases like prostate cancer where long-term follow-up is essential. The results suggest that such archives can accelerate the clinical translation of AI-assisted diagnostics.