Benchmarking Foundation Models for Renal Lesion Stratification in CT

Technical Summary: Benchmarking Foundation Models for Renal Lesion Stratification in CT

Problem Statement
The stratification of renal lesions via Computed Tomography (CT) is a clinically critical yet data-scarce task. While guidelines recommend contrast-enhanced CT for characterizing renal masses, non-invasive differentiation between specific histological subtypes (e.g., distinguishing benign renal oncocytoma from malignant chromophobe RCC, or papillary RCC Type I from other entities) remains challenging. Deep learning approaches have shown promise but are often hindered by the low prevalence of specific subtypes, leading to datasets with negligible sample sizes for rare conditions (e.g., as few as 17 cases in cross-validation). This scarcity limits the efficacy of standard convolutional neural networks (CNNs) trained from scratch. The emergence of open-source medical Foundation Models (FMs) offers a potential solution by leveraging large-scale pre-training to extract robust feature representations. However, it remains unclear whether these generalist pre-trained representations can effectively transfer to specific, data-constrained renal lesion classification tasks without fine-tuning.

Methodology
The study conducted a rigorous benchmark of three open-source medical FMs against established baselines using a frozen feature-probing protocol.

• Datasets: The training set comprised 2,854 lesions from 1,279 participants, aggregated from an in-house cohort (2007–2022) and the KiTS23 dataset. An external test set of 234 lesions from 140 CT scans was sourced from The Cancer Imaging Archive (TCIA). Lesions were categorized into six classes: cysts, clear cell RCC (ccRCC), papillary RCC (pRCC), chromophobe RCC (chrRCC), renal oncocytoma (RO), and "Other."
• Model Selection: Three FMs were evaluated: FMCIB (Foundation Model for Cancer Imaging Biomarkers), CT-FM (Vision Foundation Models for Computed Tomography), and MMM (MedicalMultitaskModeling).
• Experimental Strategies:
  1. Strategy A (End-to-End Learning): A 3D ResNet-50 was trained from scratch and compared against an FMCIB model fine-tuned with specific architectural recommendations (larger classification head, split learning rates).
  2. Strategy B (Frozen Feature Probing): The primary evaluation method. Embeddings were extracted from the three FMs and a handcrafted radiomics pipeline using frozen encoders. These features were fed into XGBoost classifiers.
• Preprocessing: Images were cropped to lesion bounding boxes with a 1 cm margin, resized to model-specific dimensions (e.g., 50×50×50 voxels for FMCIB), and augmented. Radiomics features were extracted using PyRadiomics (107 features) from resampled, windowed volumes.
• Evaluation: Performance was measured using macro-averaged Area Under the Receiver Operating Characteristic curve (AUC) and Average Precision (AP). Internal validation used 10-fold cross-validation, while external validation utilized a held-out TCIA set with non-parametric bootstrapping for confidence intervals.

Key Results
The study yielded two primary findings regarding the comparative performance of FMs, deep learning from scratch, and radiomics:

1. Radiomics Superiority: The handcrafted radiomics baseline significantly outperformed all deep learning approaches. On the external test set, radiomics achieved an AUC of 0.88 (AP 0.64). This was statistically superior to the best-performing Foundation Model (MMM, AUC 0.77) with a mean difference of 0.11 (95% CI [0.03, 0.18], p=0.002).
2. Foundation Model vs. From-Scratch ResNet: The performance of the frozen FM embeddings (AUC range 0.69–0.77) was comparable to the 3D ResNet-50 trained from scratch (AUC 0.72). No statistically significant differences were found between the FMs and the ResNet baseline. Furthermore, fine-tuning the FMCIB model did not yield improvements over training the ResNet from scratch (AUC 0.68 vs. 0.64) and failed to match the performance of the frozen probing approach (AUC 0.71).
3. Computational Efficiency: While the ResNet baseline required 16 hours of GPU training, the FM probing pipeline (specifically MMM) required only 2.5 hours for feature extraction on a CPU, with subsequent classifier training taking mere seconds. This demonstrates a drastic reduction in hardware demand for FMs.
4. Feature Analysis: UMAP visualizations indicated that while cysts and ccRCC formed distinct clusters across all embeddings, other subtypes (pRCC, chrRCC, RO) showed substantial overlap. Radiomics feature importance analysis revealed that texture features contributed most (65%) to the classifier's decision, followed by intensity statistics and shape.

Significance and Claims
The paper concludes that while medical Foundation Models offer a computationally efficient alternative to training deep networks from scratch in data-scarce settings, they do not yet surpass established radiomics baselines for renal lesion stratification.

The authors argue that current generalist FM embeddings fail to capture the fine-grained texture and shape heterogeneity required to discriminate between specific histological subtypes, which are the primary drivers of classification difficulty in this domain. The study highlights that the interpretability and explicit engineering of radiomics features currently provide a more discriminative signal than the latent representations of generalist FMs.

Furthermore, the paper underscores the lack of consensus on the optimal downstream utilization of medical FMs (e.g., patch size, resampling, feature aggregation), suggesting that current benchmarks reflect implementation variability as much as architectural capability. The authors assert that until optimization gaps are addressed and standardized evaluation protocols are established, radiomics remains the state-of-the-art for texture-dependent renal lesion classification tasks. The study does not claim that FMs are obsolete, but rather that their current generalist representations are insufficient for this specific high-stakes, fine-grained diagnostic task without further domain-specific adaptation.