Understanding Human AI Discrepancy in Breast Cancer TIL Assessment: A Multi-Rater and Perceptual Bias Study

Technical Summary: Understanding Human–AI Discrepancy in Breast Cancer TIL Assessment

Problem Statement
Tumor-infiltrating lymphocytes (TILs) in breast cancer, particularly in triple-negative and HER2-positive subtypes, serve as critical prognostic and predictive biomarkers. Despite the existence of International TILs Working Group guidelines, TIL assessment remains subject to substantial inter-observer variability due to the inherent subjectivity of visual estimation, heterogeneity in lymphocyte distribution, and perceptual biases. While Artificial Intelligence (AI) has been proposed to standardize this process, previous studies have largely focused on model accuracy against reference annotations rather than directly comparing the agreement between multiple human observers and AI models on identical regions. Furthermore, the specific sources of discrepancy between human visual estimation and algorithmic quantification remain under-investigated.

Methodology
The study employed a multi-rater, multi-model design involving 184 breast cancer cases (186 cases from 179 patients) and 416 manually annotated stromal regions of interest (ROIs).

• Human Assessment: Three pathologists (one expert, one senior resident, one junior resident) independently evaluated the ROIs to estimate stromal TIL percentages and stromal tumor area proportions according to standard guidelines.
• AI Assessment: Two distinct AI pipelines were applied to the same ROIs without local fine-tuning (trained solely on public datasets):
  1. YOLO Pipeline: Utilized U-Net for stromal segmentation and YOLO-v8 for lymphocyte detection.
  2. HoVer-NeXt Pipeline: Utilized U-Net for stromal segmentation and HoVer-NeXt (an instance segmentation model) for lymphocyte detection.
• Statistical Analysis: Agreement was quantified using Intraclass Correlation Coefficients (ICC), Concordance Correlation Coefficients (CCC), Spearman and Pearson correlations, and Kappa statistics (Cohen's and Fleiss') for both continuous and categorical (dichotomized and multi-tier) scores. Bland–Altman analysis was used to assess bias.
• Perceptual Bias Study: To isolate human cognitive limitations, a complementary experiment was conducted using synthetic black-and-white images mimicking lymphocyte distributions. Ten pathologists estimated the white-area percentage in these images against known ground truths to measure inherent visual estimation error.

Key Results

• Inter-Pathologist Agreement: High agreement was observed among human raters, with a global ICC of 0.81 (95% CI 0.76–0.84) and strong pairwise correlations (ρ = 0.82–0.85). Categorical agreement (Fleiss' kappa) was moderate (0.50–0.58).
• Human–AI Agreement: Agreement between pathologists and AI models was significantly lower. The global ICC for all five raters (3 humans + 2 AI) was 0.41. Pairwise ICCs between individual pathologists and AI models ranged from 0.10 to 0.21.
• Model Performance: The HoVer-NeXt model demonstrated better alignment with pathologists (ICC ~0.21) compared to the YOLO model (ICC ~0.10–0.16). However, both AI models exhibited a systematic negative bias, consistently underestimating TIL scores by approximately 15% compared to pathologists, with the discrepancy widening at higher TIL scores.
• Correlation vs. Agreement: While absolute agreement was poor, Spearman correlations between pathologists and AI were moderate to good (ρ = 0.48–0.81), indicating that AI models could rank cases relatively well but failed to match absolute scoring values.
• Perceptual Bias Findings: The synthetic image study revealed that pathologists' estimates of white-area percentages had low-to-moderate agreement with ground truth (ICC 0.16–0.57). This suggests that human visual estimation of scattered objects is inherently error-prone, contributing significantly to the observed human–AI discrepancy.

Significance and Claims
The paper claims that the discrepancy between human and AI TIL scoring is not solely attributable to algorithmic limitations but is also driven by intrinsic perceptual and cognitive biases in human visual estimation. The study demonstrates that while experienced pathologists achieve high internal consistency, their visual estimation of scattered cellular patterns is subject to error, a limitation that pixel-based AI models bypass.

The authors conclude that current AI models, even when technically accurate in segmentation and detection, do not yet achieve the absolute agreement required for standalone clinical implementation. They emphasize that future AI integration should focus on:

1. Standardization: Training on larger, more diverse datasets to improve generalizability.
2. Perceptual Calibration: Developing strategies to align AI outputs with human visual tendencies rather than treating human scores as the sole ground truth.
3. Decision Support: Positioning AI as a tool to reduce observer variability and enhance standardization rather than as a direct replacement for pathologist scoring in its current form.

The study underscores the need for a realistic understanding of human–AI dynamics, suggesting that the "gold standard" in TIL assessment may need to be re-evaluated considering the cognitive constraints of human observers.