Unifying Multiple Foundation Models for Advanced Computational Pathology

1. Problem Statement

While pathology foundation models (PFMs) like Virchow, UNI, and H-Optimus-1 have advanced computational pathology by learning transferable visual representations from large datasets, they face three critical limitations:

• Performance Variability: No single model consistently outperforms others across all tasks (diagnostic, prognostic, molecular, and region-level) due to differences in pretraining data composition and institutional biases.
• Data Accessibility & Scalability: High-performing models often rely on proprietary, non-shareable datasets. This prevents the research community from cumulatively expanding the pretraining corpus, unlike the open-data evolution seen in natural image and language models.
• Inefficiency of Offline Distillation: Existing attempts to combine models via offline knowledge distillation (e.g., GPFM) require dedicated distillation datasets and full retraining of a "student" model whenever a new foundation model is introduced. This is computationally expensive, inflexible, and limits the ability to capture task-specific nuances.

2. Methodology: The Shazam Framework

The authors propose Shazam, an online integration model that adaptively combines multiple pre-trained pathology foundation models without requiring access to their original training data or offline retraining.

Core Architecture

Shazam operates as a unified ensemble system with the following components:

1. Multi-Teacher, Multi-Scale Feature Extraction:
   • Shazam utilizes five state-of-the-art PFMs as frozen "teacher" models: UNI 2, Virchow2, H-optimus-1, Prov-Gigapath, and Phikon-v2.
   • Instead of using only the final layer, Shazam extracts features at three semantic depths (Low, Middle, and High levels) from each teacher by hooking into specific transformer blocks (approx. 33%, 66%, and 100% depth).
2. Feature Alignment and Scale-wise Fusion (Mixture-of-Experts):
   • Features from all teachers are concatenated and passed through a gating network (a lightweight MLP) to compute adaptive weights.
   • This implements a Mixture-of-Experts (MoE) mechanism, allowing the model to dynamically emphasize the most relevant teacher features for a specific input or task.
   • The weighted features are organized into a matrix and processed by self-attention layers to model interactions across different teachers and scales.
3. Online Distillation:
   • A "student" model is trained to align with the fused teacher representations.
   • The loss function combines the task-specific loss (e.g., cross-entropy for classification, negative log-likelihood for survival) with a multi-level distillation loss.
   • The distillation loss uses a combination of Cosine Distance and Huber Loss to align student features with teacher features at low, middle, and high semantic scales simultaneously.
4. Task Adaptation:
   • Tile-level tasks: Directly use fused tile embeddings.
   • Whole-slide image (WSI) tasks: Employ Attention-Based Multiple Instance Learning (ABMIL) to aggregate patch-level features into slide-level representations before fusion.

3. Key Contributions

• Online Integration Paradigm: Shazam introduces a shift from offline distillation to online, task-specific integration. It can incorporate new foundation models immediately without retraining the entire system from scratch or curating new distillation datasets.
• Unified Representation Learning: It successfully consolidates complementary strengths from heterogeneous models (different architectures and pretraining data) into a single, robust representation.
• Scalability: The framework is designed to be extensible, allowing the community to continuously integrate emerging models as they become available.
• Comprehensive Benchmarking: The authors provide a rigorous evaluation across 30 diverse benchmarks, covering spatial transcriptomics, survival analysis, tile-level classification, and visual question answering (VQA).

4. Results

Shazam was evaluated against the five constituent foundation models across four major task categories:

• Overall Performance: Shazam achieved an average ranking score of 1.17 across 30 benchmarks, significantly outperforming the next best model, Virchow2 (3.20). It ranked 1st in 26 out of 30 tasks.
• Spatial Transcriptomics (ST) Prediction:
  • On the HEST-1k dataset (8 organ types), Shazam achieved the highest Pearson Correlation Coefficient (PCC) for all organs, with an average improvement of 0.08–0.17 over the best individual model.
  • It demonstrated superior ability to capture heterogeneous spatial gene expression patterns (e.g., IDH1 upregulation in dysplastic glands) that single models missed.
• Survival Prognosis:
  • Across 10 TCGA cancer cohorts, Shazam achieved a mean C-index of 0.666, surpassing the best individual model (H-optimus-1 at 0.642).
  • Kaplan-Meier analyses showed statistically significant separation between high-risk and low-risk groups in 9 out of 10 cohorts.
• Tile-Level Classification:
  • On 11 diverse datasets (e.g., PCAM, PanCancer-TIL, BACH), Shazam achieved a mean ranking of 1.36 vs. 2.46 for the best baseline.
  • Notable gains included a 0.914 F1 score on PCAM (metastasis detection) and 0.852 on PanCancer-TIL.
• Visual Question Answering (VQA):
  • On the PathVQA dataset, Shazam achieved 57.5% accuracy, outperforming the second-best model (UNI 2) by 1%. This is significant given that the underlying models were trained only on images, not text.

Ablation Studies confirmed that:

• Removing the strongest teachers degraded performance significantly, validating the value of diverse expert supervision.
• Fusing features from all three semantic levels (Low, Mid, High) yielded the best results, proving the complementarity of multi-scale representations.
• The MoE module provided measurable gains over uniform feature aggregation.

5. Significance

• Overcoming Data Silos: Shazam offers a practical solution to the "data silo" problem in medical AI. It allows the field to leverage the collective knowledge of proprietary models without needing access to their private training data.
• Continuous Evolution: By enabling online integration, Shazam supports a modular approach where the system can evolve as new foundation models are released, avoiding the stagnation caused by isolated, monolithic model development.
• Clinical Utility: The consistent performance across molecular, regional, and slide-level tasks suggests that Shazam is a robust, generalizable tool ready for diverse clinical decision-support applications, from prognosis to biomarker discovery.
• Future Direction: The paper advocates for a paradigm shift in computational pathology from training isolated foundation models to building compositional, modular systems that can adaptively fuse heterogeneous knowledge sources.