Comprehensive framework for evaluation of deep neural networks in detection and quantification of lymphoma from PET/CT images: clinical insights, pitfalls, and observer agreement analyses

Technical Summary: Comprehensive Framework for Evaluation of Deep Neural Networks in Detection and Quantification of Lymphoma from PET/CT Images

Problem Statement
Automated lymphoma segmentation from Fluorodeoxyglucose (18F-FDG) PET/CT images is critical for diagnosis, staging, and therapy response evaluation. However, existing deep learning approaches face significant gaps:

1. Lack of Generalizability: Few studies evaluate models on external or out-of-distribution (OOD) data, limiting real-world clinical applicability.
2. Insufficient Variability Analysis: Intra- and inter-observer variability analyses are rare, obscuring the inherent difficulty of the segmentation task and the baseline for human performance.
3. Over-reliance on Global Metrics: Most studies prioritize global overlap metrics like the Dice Similarity Coefficient (DSC), which can mask failures in lesion-level detection and neglect clinically crucial metrics such as Total Metabolic Tumor Volume (TMTV) and Total Lesion Glycolysis (TLG).
4. Small Datasets: Prior works often rely on small datasets (<100 cases) and lack transparency regarding lesion measure distributions.

Methodology
The authors propose a clinically-relevant evaluation framework utilizing a large, multi-institutional dataset of 611 PET/CT cases spanning four cohorts:

• Internal Cohorts: DLBCL-BCCV (107 scans), PMBCL-BCCV (139 scans), and DLBCL-SMHS (220 scans).
• External Cohort: AutoPET lymphoma (145 scans) from a different center and scanner manufacturer.

Ground Truth (GT): Annotations were performed by nuclear medicine physicians using semi-automatic tools. Intra- and inter-observer variability were assessed via re-segmentation of specific subsets and consensus generation (STAPLE).

Models Evaluated: Four state-of-the-art deep segmentation networks were trained and compared:

1. ResUNet (3D CNN)
2. SegResNet (3D CNN, winner of HECKTOR 2022)
3. DynUNet (3D full-resolution nnUNet implementation)
4. SwinUNETR (3D Transformer-based)

Evaluation Framework:
Beyond standard DSC, the study introduces a multi-faceted evaluation strategy:

• Volumetric Metrics: False Positive Volume (FPV) and False Negative Volume (FNV).
• Detection Criteria: Three per-lesion criteria were defined:
  • Criterion 1 (Weak): Any voxel overlap (TP if $|G_l \cap P_{l'}| > 0$).
  • Criterion 2 (Strong): Intersection-over-Union (IoU) > 0.5.
  • Criterion 3 (Proposed): Detection of the voxel with maximum Standardized Uptake Value ($SUV_{max}$) within the lesion.
• Clinical Measure Reproducibility: Equivalence testing (TOST) and Bland-Altman analysis on six clinically relevant measures: $SUV_{mean}$, $SUV_{max}$, Lesion Count ($L$), TMTV, TLG, and Lesion Dissemination ($D_{max}$).
• Stratified Analysis: Performance was analyzed as a function of GT lesion measure values (e.g., low vs. high $SUV_{mean}$, small vs. large TMTV).

Key Results

• Model Performance: SegResNet achieved the highest median DSC (0.76 internal, 0.68 external) and lowest FPV. ResUNet performed competitively on specific internal cohorts. SwinUNETR achieved the best median FNV on the internal set.
• Generalizability: All models showed a performance drop on the external OOD dataset, confirming the challenge of domain shift.
• Lesion Characteristics: Networks performed best on large, metabolically active lesions (high $SUV_{mean}$, high TMTV/TLG). Performance plateaued or declined for small, faint lesions ($\le$ 20th percentile of measures).
• Observer Variability:
  • Intra-observer agreement (Physician 3) mirrored network performance: both struggled with small/faint lesions (low TMTV/TLG) but were consistent on larger lesions.
  • Inter-observer agreement (Physicians 1 & 2) showed high correlation (ICC > 0.93) on lesion measures, yet segmentation DSC varied significantly, particularly in difficult cases.
• Detection Criteria:
  • Criterion 1 yielded the highest sensitivity but high false positives.
  • Criterion 2 (IoU > 0.5) showed low sensitivity for faint lesions for both networks and physicians.
  • Criterion 3 (SUVmax detection) demonstrated superior sensitivity for localizing lesions compared to Criterion 2, even for small lesions, suggesting that identifying the metabolic "hotspot" is more robust than delineating full boundaries in low-resolution PET.
• Clinical Measure Reproducibility: Despite high DSC, networks generally failed to achieve equivalence (within $\pm$20%) for $SUV_{max}$ and TMTV, especially in external testing. This indicates that high spatial overlap does not guarantee accurate quantitative biomarker extraction.

Significance and Claims
The paper claims that its primary contribution is a comprehensive, clinically-relevant benchmarking framework that moves beyond proof-of-concept DSC metrics. By contextualizing network performance against human observer variability and analyzing performance across specific lesion characteristics, the study:

1. Highlights Limitations: Demonstrates that current models, while effective for large lesions, struggle with the "long tail" of small/faint lesions and fail to consistently reproduce quantitative biomarkers (SUV, TMTV) required for precise clinical decision-making.
2. Validates Human Baseline: Shows that the difficulty of segmenting small/faint lesions is intrinsic to the imaging modality (partial volume effects, noise) and affects both AI and human experts similarly.
3. Proposes Better Metrics: Introduces Criterion 3 and lesion-measure stratification as essential tools for evaluating the clinical utility of segmentation models, arguing that detecting the $SUV_{max}$ voxel is often more clinically actionable than perfect boundary delineation.
4. Promotes Transparency: Provides a public code repository and detailed dataset characterization to enable reproducibility and future research into robust, histology-agnostic lymphoma segmentation.

The authors conclude that while deep learning offers promise for automating lymphoma segmentation, current models require further development—specifically in handling small lesions and preserving quantitative fidelity—to be fully reliable for clinical decision support.