Multimodal Cancer Modeling in the Age of Foundation Model Embeddings

Technical Summary: Multimodal Cancer Modeling in the Age of Foundation Model Embeddings

Problem Statement
The Cancer Genome Atlas (TCGA) serves as a premier resource for cancer research, offering harmonized genomics, clinical, and imaging data for over 11,000 cases across 33 cancer types. While numerous studies have developed bespoke deep learning models for tasks like cancer survival prediction using TCGA data, these approaches often require significant computational resources and task-specific training. Furthermore, despite the availability of free-text pathology reports within TCGA, these data have historically been underutilized due to the challenges of processing raw, scanned PDF documents. The authors investigate whether classical machine learning models, when applied to zero-shot embeddings derived from modern Foundation Models (FMs), can effectively predict cancer survival using multimodal data without the need for extensive fine-tuning.

Methodology
The study employs a modular, late-fusion framework that combines zero-shot FM embeddings with simple classical machine learning models (specifically Cox Proportional Hazards models).

• Data Curation: The authors assembled a dataset of 7,982 patient cases from TCGA, ensuring the availability of survival data, pathology reports, tumor diagnostic slides, and tumor RNA-seq gene expression. Demographic data (age, sex, race, ethnicity) and cancer type were one-hot encoded.
• Foundation Model Embeddings: The authors utilized pre-trained, zero-shot embeddings without further fine-tuning to prevent data leakage:
  • Histology: UNI2-h embeddings for diagnostic slides.
  • Gene Expression: BulkRNABert (trained on GTEx/ENCODE) and Universal Cell Embeddings (UCE).
  • Text: BioMistral-7B and Mistral-7B-Instruct-v0.1 for pathology reports.
  • Aggregation: Patient-level embeddings were derived by averaging embeddings from multiple samples of the same modality.
• Pathology Report Processing: To address issues with OCR errors and report length, the authors used Llama-3.1-8B-Instruct to summarize pathology reports, focusing on microscopic descriptions, diagnoses, and clinical history while omitting gross descriptions.
• Modeling Pipeline:
  1. Unimodal Modeling: For each modality, embeddings were standardized and reduced via Principal Component Analysis (PCA) to varying dimensions (4–256). A CoxPH model was trained on the training split to predict risk scores.
  2. Multimodal Fusion: A "late-fusion" approach was used where the predicted risk scores from unimodal models served as input features for a final multimodal CoxPH model. This allowed for the integration of diverse modalities (demographics, cancer type, expression, histology, and text) without complex architectural changes.
  3. Evaluation: Models were evaluated using 5-fold cross-validation with metrics including the Concordance Index (C-index), mean cumulative/dynamic Area Under the Curve (AUCC,D), and Integrated Brier Score (IBS).
• Hallucination Analysis: A subset of 40 randomly selected cases was manually reviewed to correct LLM-generated summaries for factual hallucinations. These corrected summaries were re-embedded to assess the impact of hallucinations on downstream survival prediction.

Key Contributions

1. Quantification of FM Embeddings for Survival: The study demonstrates that zero-shot FM embeddings combined with small, linear models (CoxPH) can achieve competitive survival prediction performance (peak C-index ~0.75 for single modalities) on TCGA, rivaling more complex deep learning approaches.
2. Additive Multimodal Fusion: The authors introduce a simple, extensible late-fusion framework showing that cancer data modalities are additive and non-redundant. Combining expression, histology, and text embeddings with clinical features (demographics and cancer type) yielded the best performance.
3. Impact of Text Summarization: The paper provides evidence that summarizing pathology reports using LLMs significantly improves survival prediction compared to using raw, unsummarized text embeddings.
4. Hallucination Robustness: Through manual correction of a subset of summaries, the study evaluates the effect of LLM hallucinations, finding that minor factual corrections did not significantly alter risk stratification in the sampled subset.

Results

• Unimodal Performance: Single-modality models achieved a peak mean cross-validated C-index of approximately 0.75. Performance generally plateaued at a PCA dimension of 256. Notably, unimodal models over FM embeddings did not substantially outperform models using only tabular clinical features (demographics + cancer type, C-index 0.747) in all cases, highlighting the importance of a strong baseline.
• Multimodal Performance: Multimodal fusion consistently outperformed unimodal models. The combination of all modalities (demographics, cancer type, expression, histology, and text) achieved a C-index of 0.795.
• Modality Specificity:
  • Gene Expression: Domain-specific models (BulkRNABert) outperformed general cell biology models (UCE) for bulk RNA-seq data.
  • Text: Summarized reports yielded a C-index of 0.752, significantly higher than unsummarized reports (0.694). Domain adaptation (BioMistral vs. Mistral) showed minimal difference compared to the impact of summarization.
  • Cancer Types: Multimodal models consistently outperformed unimodal models across the 8 most prevalent cancer types (e.g., BRCA, KIRC, UCEC), whereas the best unimodal modality varied by cancer type.
• Hallucination Impact: Manual correction of hallucinations in 24 of 40 sampled reports did not change the risk stratification of the subset, suggesting that the salient information for survival prediction (e.g., invasion, metastasis) remained intact despite minor factual errors.

Significance and Claims
The authors claim that their work establishes a modular, low-resource framework for multimodal cancer modeling that leverages the power of foundation models without requiring the computational infrastructure for training bespoke deep learning models. By using zero-shot embeddings and simple linear models, the approach addresses data privacy concerns and avoids overfitting on small datasets.

The study emphasizes that while complex deep learning methods exist, the synergy between feature-rich FM embeddings and simple statistical models is sufficient for high-accuracy survival prediction. The authors position their work as a quantification of this "embedding-centric" approach, demonstrating that multimodal data is additive and that pathology report summarization is a critical preprocessing step for text-based modeling. They conclude that this paradigm offers a scalable and effective pathway for applying foundation models to cancer multi-omics and broader biomedical tasks, while noting limitations regarding sample sizes for hallucination correction and the need for external validation beyond TCGA.