Probing, Fusion, and Trustworthiness: A Systematic Evaluation of Foundation Model Representations for Multimodal Cancer Analysis

Technical Summary: Probing, Fusion, and Trustworthiness of Foundation Model Representations for Multimodal Cancer Analysis

Problem Statement
Foundation models (FMs) have emerged as powerful representation extractors for medical data, yet their generalizability to real-world datasets under distribution shift remains underexplored. While prior evaluations of pathology and transcriptomic FMs (e.g., CONCH, UCE) show promise, they are often conducted on public benchmarks that may overlap with pretraining corpora. Real-world industrial datasets, such as those from commercial oncology sources, often follow different data collection pipelines and distributions. This paper addresses three critical gaps: (1) whether FM representations transfer reliably to out-of-distribution (OOD) clinical data; (2) whether multimodal fusion of image and omics data yields consistent gains over unimodal baselines; and (3) the trustworthiness of these pipelines, specifically regarding uncertainty quantification and calibration in high-stakes diagnostic settings.

Methodology
The study utilizes two real-world, in-house (IH) commercial cohorts: IH-BC (breast cancer) and IH-NSCLC (non-small cell lung cancer). These datasets, derived from the Flatiron Health-Caris Life Sciences Clinical-Molecular Database, provide paired whole-slide H&E images and transcriptomic profiles (TPM RNA gene expression). The data is split into training, calibration, validation, and test sets (7:3:1:1 ratio).

The experimental workflow consists of three stages:

1. Unimodal Probing:

   • Image Representations: Four tile-level pathology foundation models are evaluated: CONCH, UNI, MUSK, and Virchow.
   • Omics Representations: Three methods are compared: a transcriptomic FM (UCE), a variational autoencoder (scVI), and a classical dimensionality reduction baseline (PCA).
   • Downstream Tasks: Eight classification tasks are defined, including biopsy site prediction, molecular subtype identification, biomarker status (PR, PIK3CA, LOH, TMB), and tumor site identification.
   • Learning: Unimodal performance is assessed using Multiple Instance Learning (HEMIL) for images and Multilayer Perceptrons (GeneMLP) for omics.

2. Multimodal Fusion:

   • Three fusion strategies are implemented to combine paired image and omics representations:
     • CONTACT: Concatenation of omics features with every image tile embedding, followed by a fusion encoder.
     • MCAT: A co-attention transformer that exchanges information between image and omics tokens before MIL pooling.
     • LateMIL: A late-fusion approach combining independent WSI logits and omics logits via a weighted sum.

3. Trustworthiness Analysis:

   • Conformal Prediction (CP): Split conformal prediction is applied to all selected pipelines to generate prediction sets with marginal coverage guarantees ($\alpha = 0.10$).
   • Metrics: Evaluation includes empirical coverage, average set size, singleton rate, and "rescue rate" (the frequency with which the true label is recovered in the prediction set when the top-1 point prediction is incorrect).

Key Results

• Unimodal Performance:

  • Image FMs achieve competitive performance on OOD data, though results vary significantly by task difficulty (e.g., high AUC >0.9 on IH-BC Biopsy Site vs. ~0.64 on IH-NSCLC Biopsy Site).
  • Omics representations provide complementary signals. Notably, the classical PCA baseline often outperforms learned transcriptomic foundation models (UCE, scVI) on several tasks, achieving the highest AUC on LOH, PR, PIK3CA, Subtype, and TMB tasks. This suggests that building effective transcriptomic FMs remains an open challenge compared to classical statistical methods for capturing biological perturbations.

• Multimodal Fusion:

  • Fusion is not universally beneficial. Multimodal methods (LateMIL, MCAT, CONTACT) outperform unimodal baselines only when no single modality dominates the predictive signal.
  • When one modality carries the dominant signal (e.g., GeneMLP/PCA on NSCLC-TMB), fusion can actually degrade performance compared to the best unimodal model.
  • LateMIL demonstrates the most consistent performance across tasks compared to CONTACT and MCAT.

• Trustworthiness and Uncertainty:

  • Conformal prediction achieves mean coverage at or above the nominal 0.90 target across all tasks.
  • Task Difficulty Drives Set Size: NSCLC tasks (Biopsy Site, Tumor Site) exhibit over-coverage (0.93–0.94) with large average set sizes (2.5+) and near-zero singleton rates, reflecting the intrinsic difficulty of distinguishing lung subregions. Conversely, BC tasks show tighter coverage closer to the target with smaller set sizes (~2.0).
  • Architecture Agnosticism: The conformal behavior (coverage and set size) is largely robust to the choice of backbone model; task difficulty, rather than model architecture, governs the prediction set characteristics.
  • Rescue Rate: In cases where the point prediction fails, the true diagnosis is recovered within the prediction set in 72.6% to 87.3% of cases, validating the safety net provided by uncertainty-aware inference.

Significance and Claims
The paper claims that foundation model representations can transfer reasonably well to unseen, real-world commercial oncology data, but their utility is highly task-dependent. A primary contribution is the finding that multimodal fusion does not guarantee performance gains and may be detrimental if one modality already provides a dominant signal. Furthermore, the study highlights that classical methods like PCA can outperform modern transcriptomic foundation models in specific clinical contexts.

Regarding trustworthiness, the authors argue that shifting from single point predictions to conformal prediction sets is critical for clinical support. This approach ensures that even when a model's top-1 prediction is wrong, the correct diagnosis is rarely discarded, thereby reducing the risk of silent errors in human-AI collaboration. The work underscores that while model architecture matters, the inherent difficulty of the clinical task is the primary driver of uncertainty and prediction set size.