A comprehensive benchmark of publicly available image foundation models for their usability to predict gene expression from whole slide images

This study benchmarks five publicly available image foundation models for predicting gene expression from whole-slide images in breast cancer, demonstrating that histopathology-specific models, particularly Phikon, significantly outperform general-purpose encoders in morphology-to-transcriptome inference.

The Gist

Imagine you have a massive library of microscopic photographs of human tissue (called Whole Slide Images or WSIs). These photos are so detailed they look like high-resolution maps of a city, showing individual cells and structures.

Now, imagine you want to know the "secret recipe" (the gene expression) of the cancer in that tissue just by looking at the map, without needing to run expensive chemical tests. This is the challenge the paper tackles: Can an AI look at a picture of tissue and accurately guess the genetic activity inside it?

To solve this, the authors acted like a taste-test judge for five different "super-eyes" (AI models) to see which one is best at this task.

The Contestants: Five Different "Super-Eyes"

The researchers tested five different AI models, each trained differently. Think of them as different types of students taking a final exam:

1. DINOv2 (The Generalist): This AI was trained on millions of pictures of cats, cars, and landscapes. It's great at recognizing general shapes but has never seen a microscope slide before.
   • Analogy: Like a brilliant art student who has studied every painting in the Louvre but has never stepped foot in a hospital.
2. MedSigLIP (The Medical Generalist): This AI learned from a mix of medical images and text descriptions. It knows some medical stuff but isn't a specialist in tissue slides.
   • Analogy: A medical student who has read the textbooks but hasn't done many internships in the pathology lab.
3. UNI, H-Optimus-0, and Phikon (The Pathology Specialists): These three AIs were trained specifically on millions of images of human tissue slides. They have "seen" billions of cells and know exactly what healthy and cancerous tissue looks like.
   • Analogy: These are veteran pathologists who have spent decades staring at microscope slides. They know the difference between a normal cell and a cancer cell just by a glance.

The Test: The "Morphology-to-Genome" Challenge

The researchers took a specific set of breast cancer cases (from the TCGA-BRCA dataset). For each patient, they had:

• The Picture: The high-res tissue slide.
• The Answer Key: The actual genetic data (RNA-seq) from that patient.

They fed the pictures into the five AI models. The AIs tried to predict the genetic data based only on the visual patterns in the image. The researchers then compared the AI's guess to the real answer key using a score called Spearman Correlation (a number between -1 and 1, where 1 is a perfect match).

The Results: Who Won?

The results were clear and followed a strict hierarchy:

• 🏆 The Winner: Phikon
  This model was the clear champion. It predicted the genetic activity with the highest accuracy and consistency.

  • Why? Because it was trained specifically on the "language" of tissue slides. It learned that specific patterns in the tissue (like how crowded the cells are or how they are arranged) directly correlate to specific genetic switches being turned on or off.

• 🥈 The Runners-Up: UNI and H-Optimus-0
  These two also performed very well, significantly better than the general models, but they didn't quite reach Phikon's level of precision. They are still excellent "specialists."

• 🥉 The Middle Pack: MedSigLIP
  It did okay, better than the generalist, but not as good as the tissue specialists. It had some medical knowledge but lacked the deep, specific training on tissue structure.

• 📉 The Loser: DINOv2
  The generalist model struggled the most. While it could recognize that "this is a picture of cells," it couldn't decode the subtle biological secrets hidden in the arrangement of those cells.

  • Why? It was like asking someone who only knows how to drive a car to perform heart surgery. They know the basics of movement, but they lack the specific domain knowledge required for the task.

The Big Takeaway

The paper proves a simple but powerful rule: Specialization wins.

If you want an AI to understand the complex relationship between what a tissue looks like and what its genes are doing, you shouldn't just give it a general education (like DINOv2). You need to give it a specialized medical degree (like Phikon).

In everyday terms:
If you want to guess a person's personality just by looking at their messy desk, you'd want someone who has studied psychology and office habits (the specialist), not someone who just knows how to organize a bookshelf (the generalist). The "specialist" AIs learned that the "mess" on the tissue slide (the morphology) is actually a direct map to the genetic instructions inside.

This study provides a "menu" for doctors and scientists: if you are building tools to predict cancer genetics from images, choose the specialist models (Phikon, UNI, H-Optimus) over the general ones. It saves time, money, and leads to more accurate medical insights.

Technical Summary

1. Problem Statement

The digitization of histopathology has enabled the extraction of morphological features from Whole Slide Images (WSIs) that correlate with molecular phenotypes. While large-scale self-supervised foundation models have revolutionized visual representation learning, their systematic evaluation for transcriptomic prediction (predicting gene expression from tissue morphology) remains limited.

• The Gap: It is unclear whether general-purpose vision models (trained on natural images) or domain-specific pathology models are better suited for inferring RNA-seq profiles from H&E-stained slides.
• The Challenge: Gene expression prediction is a stringent task requiring models to detect subtle, morphology-linked transcriptomic variations across heterogeneous disease states.

2. Methodology

Dataset and Cohort

• Source: TCGA-BRCA (Breast Invasive Carcinoma) cohort from The Cancer Genome Atlas.
• Data: Matched Hematoxylin and Eosin (H&E) stained diagnostic WSIs and bulk RNA-seq profiles.
• Sample Size: After quality control and filtering for tumor tissue/artifacts, 987 cases with complete, usable WSI and RNA-seq data were retained.
• Preprocessing:
  • RNA-seq: Aligned via STAR, quantified using FPKM-UQ, log-transformed, and min-max normalized.
  • WSI: Partitioned into tiles; a single representative slide was selected per patient.

Foundation Models Evaluated

The study benchmarked five state-of-the-art encoders representing different pretraining paradigms:

1. DINOv2: General-purpose self-supervised ViT trained on natural images.
2. Phikon: Pathology-specific DINO-based model trained on pan-cancer histology.
3. UNI: Large-scale pathology model trained on >100M histology patches.
4. H-Optimus-0: Billion-parameter ViT-g pathology model.
5. MedSigLIP: Medical vision-language pretraining framework (multimodal).

Prediction Framework

• Feature Extraction: Frozen tile embeddings were extracted from each model (dimensions ranging from ~768 to ~1536).
• Aggregation: An Attention-based Multiple Instance Learning (MIL) framework aggregated tile embeddings into a single slide-level representation.
• Regression: A fully connected regression head performed multi-target regression to predict continuous gene expression values for ~60,000 genes.
• Optimization: Minimized prediction error between predicted and observed RNA-seq values.

Evaluation Metrics

Performance was assessed using gene-level Spearman correlation ($\rho$) across samples. Analysis included:

• Distributional comparisons (Boxplots, Histograms).
• Empirical Cumulative Distribution Functions (ECDF).
• Rank-based correlation curves.
• Threshold-based summaries (proportion of genes with $\rho > 0.3$ and $\rho > 0.5$).

3. Key Results

The study established a clear performance hierarchy: Phikon > UNI $\approx$ H-Optimus-0 > MedSigLIP > DINOv2.

• Overall Performance:

  • Phikon achieved the strongest overall performance, exhibiting the highest median correlation, a compact interquartile range (indicating stability), and the highest fraction of genes exceeding correlation thresholds.
  • UNI and H-Optimus-0 followed with moderately high medians and competitive performance, though they showed slightly more variance than Phikon.
  • MedSigLIP demonstrated moderate performance.
  • DINOv2 (general vision) consistently underperformed, showing the lowest median correlation, the widest dispersion, and a significant number of genes with near-zero or negative correlations.

• Distributional Insights:

  • ECDF Analysis: Phikon and H-Optimus showed the most favorable rightward shifts, indicating a higher fraction of genes achieving moderate-to-high correlations.
  • Rank-Based Analysis: Phikon maintained superior correlation values across the entire gene spectrum, particularly for the best-predicted genes. DINOv2 showed a steep decline for lower-ranked genes.
  • Thresholds: Phikon achieved the highest proportion of genes with biologically meaningful correlations ($\rho > 0.3$ and $\rho > 0.5$).

4. Key Contributions

1. First Comprehensive Benchmark: Provides the first systematic evaluation of publicly available image foundation models specifically for transcriptomic inference from WSIs.
2. Domain Alignment Validation: Quantitatively demonstrates that histopathology-specific pretraining is superior to general-purpose or multimodal pretraining for molecular prediction tasks.
3. Model Selection Guidance: Offers a principled framework for selecting foundation models in computational pathology, identifying Phikon as the current state-of-the-art for this specific task.
4. Methodological Pipeline: Establishes a robust end-to-end pipeline (Tile extraction $\to$ Attention-MIL $\to$ Multi-target regression) for morphology-to-transcriptome inference.

5. Significance and Conclusion

The findings underscore that domain relevance is a critical factor in transfer learning for computational pathology. While large-scale self-supervised learning improves feature robustness, models trained directly on histology (like Phikon and UNI) leverage domain-specific inductive biases to capture the systematic morphological phenotypes (e.g., proliferation, immune activation, stromal remodeling) that encode gene expression signatures.

• Implication: For molecular pathology tasks, relying on general-purpose vision models (like DINOv2) is suboptimal. Researchers should prioritize domain-aligned foundation models to achieve accurate, biologically interpretable predictions of gene expression from routine H&E slides.
• Future Impact: This benchmark provides a baseline for future development of "morphology-to-transcriptome" models, potentially reducing the need for expensive RNA-seq sequencing in clinical settings by inferring molecular profiles directly from standard pathology slides.