InSTaPath: Integrating Spatial Transcriptomics and histoPathology Images via Multimodal Topic Learning

InSTaPath is a multimodal topic modeling framework that integrates spatial transcriptomics and histology images by converting image features into discrete "words" to jointly infer interpretable latent topics linking gene expression programs with tissue morphology, thereby improving spatial domain identification and revealing biologically meaningful structure-function relationships.

The Gist

Imagine you are trying to understand a bustling city. You have two different maps of this city:

1. The "Gene Map": This map lists every single person living in the city, what jobs they do, and what they are saying to each other. It tells you who is there and what they are doing, but it doesn't show you what the buildings look like.
2. The "Photo Map": This is a giant, high-resolution aerial photograph of the city. It shows you the skyscrapers, the parks, the slums, and the factories. It tells you what the city looks like, but it doesn't tell you who lives inside or what they are saying.

For a long time, scientists studying diseases like cancer have had to choose one map or the other, or try to glue them together clumsily. They could see the cells (the people) and the tissue structure (the buildings), but they struggled to understand how the people's conversations (genes) actually shaped the shape of the buildings (tissue).

Enter InSTaPath.

Think of InSTaPath as a super-smart translator and architect that finally merges these two maps into one perfect, understandable guide. Here is how it works, using simple analogies:

1. Turning Pictures into "Words" (The Dictionary)

The biggest problem is that computers see photos as millions of tiny, continuous dots of color (pixels), which is like trying to read a novel written in pure static noise. It's hard to count or organize.

InSTaPath solves this by using a "magic dictionary."

• It looks at a tiny piece of the tissue photo and asks a super-intelligent AI (trained on millions of other medical photos) to describe it.
• Instead of saying "this is a shade of pink with a curve," the AI assigns it a code word, like "Muscle Fiber" or "Tumor Cluster."
• Suddenly, the entire photo is no longer a picture; it's a bag of words. Just like a gene list counts how many times a gene appears, InSTaPath counts how many times "Muscle Fiber" or "Tumor Cluster" appears in a specific spot.

2. The "Topic" Detective (Finding the Story)

Now that we have a list of genes (words) and a list of image codes (words) for every spot in the tissue, InSTaPath plays a game of "detective."

It looks for Topics.

• Imagine you are reading a library of books. You notice that whenever the word "Fire" appears, the words "Smoke," "Heat," and "Brick" also appear. You deduce a topic: "A Burning Building."
• In the body, InSTaPath notices that whenever certain genes (like "Tumor Growth") appear, certain image words (like "Dense Cell Clumps") also appear.
• It groups these together into a Topic. One topic might be "The Immune Defense Zone" (full of immune cells and specific genes). Another might be "The Fat Storage Area."

This is powerful because it doesn't just say "this is a tumor." It says, "This is a tumor because these specific genes are talking to these specific building shapes."

3. Why This Matters (The "What If" Game)

The coolest part of InSTaPath is its ability to run "What If" simulations (called in silico perturbation).

Imagine you have a recipe for a cake (the tissue).

• Old way: You just look at the cake and guess what ingredients made it rise.
• InSTaPath way: You can take the recipe, delete the "Eggs" (a specific gene), and ask the computer: "If we remove the eggs, what will the cake look like?"

The computer then predicts: "If you remove these genes, the 'Tumor' building blocks will disappear, and the tissue will start to look like normal, healthy tissue."

This allows scientists to test drugs or understand disease mechanisms without cutting up a single real patient. They can virtually "knock out" genes and watch the tissue morph on the screen to see if the disease signature vanishes.

The Bottom Line

InSTaPath is a tool that translates the visual language of tissue (what it looks like) into the same language as genetics (what the cells are saying).

• Before: Scientists had two separate languages and couldn't see the connection.
• Now: InSTaPath speaks both fluently. It finds the hidden stories (topics) where genes and tissue shapes dance together, helping us understand exactly how a disease builds its "house" and, more importantly, how to tear it down.

It turns a blurry photo and a long list of numbers into a clear, readable story about how our bodies work and how they break.

Technical Summary

1. Problem Statement

Spatial Transcriptomics (ST) technologies provide gene expression data while preserving tissue spatial context, often paired with histological (H&E) images. However, current computational approaches face two main limitations:

• Underutilization of Histology: Most pipelines use histology images only for visualization or manual annotation, relying primarily on gene expression for downstream analysis. This fails to capture the complex interactions between tissue architecture (morphology) and transcriptional programs.
• Lack of Interpretability: Existing deep learning methods that integrate both modalities (e.g., graph-based or contrastive learning approaches like OmiCLIP) often produce latent embeddings that are difficult to interpret. They do not explicitly model shared latent "programs" that link specific gene sets to specific morphological patterns in a human-readable way.

2. Methodology: InSTaPath Framework

InSTaPath is a multimodal topic modeling framework designed to jointly analyze gene expression and histological morphology. It operates in three distinct stages:

Stage 1: Image Embedding Discretization (Vector Quantization)

To apply topic modeling (which typically requires discrete count data) to continuous histology images, InSTaPath converts image features into "image words":

1. Token Extraction: Whole-slide images (WSIs) are tiled, and a pretrained histology foundation model (UNI, based on ViT-Giant) extracts token-level embeddings (1536 dimensions).
2. Codebook Construction: Token embeddings are partitioned into sub-vectors. Mini-batch k-means clustering is applied to these sub-vectors to create a visual codebook (e.g., 512 entries).
3. Vector Quantization (VQ): Each token embedding is mapped to its nearest cluster center in the codebook. This converts continuous embeddings into discrete image-word counts, analogous to gene expression counts.

Stage 2: Multimodal Topic Modeling

InSTaPath treats each ST spot as a "document" containing two modalities: gene expression counts and image-word counts.

• Architecture: It employs modality-specific Variational Autoencoder (VAE) encoders (two-layer fully connected networks) to map inputs to a shared latent space.
• Integration: A Product-of-Gaussians mechanism integrates the modality-specific latent variables into a unified spot-level topic distribution ($\theta$).
• Decoding: A shared topic embedding matrix and modality-specific feature matrices reconstruct the original modalities. This yields:
  • $\beta_{gene}$: Topic–gene associations (probability of genes given a topic).
  • $\beta_{img}$: Topic–image-word associations (probability of image words given a topic).

Stage 3: Downstream Analyses

The learned topic representations enable three key analyses:

1. Spatial Domain Detection: Clustering spots based on their topic proportions ($\theta$) to identify tissue regions.
2. Biological Interpretation: Gene Set Enrichment Analysis (GSEA) on $\beta_{gene}$ to link topics to biological pathways; visualization of $\beta_{img}$ to map morphological patterns across the entire WSI (beyond the ST capture area).
3. In Silico Perturbation: Simulating gene knockouts by zeroing out top-ranked genes for a specific topic, then reconstructing the image-word distribution to observe predicted changes in histological signals.

3. Key Contributions

• Discrete Image Representation: The novel conversion of continuous histology embeddings into count-based "image words" via vector quantization, enabling the direct application of probabilistic topic models to pathology images.
• Interpretable Multimodal Integration: Unlike black-box deep learning models, InSTaPath provides explicit, interpretable links between latent topics, specific gene programs, and specific morphological features (image words).
• Whole-Slide Context: By deriving image words from whole-slide features, the framework can visualize topic-associated morphology across the entire tissue section, not just within the limited ST capture spots.
• Causal Inference Capability: The framework supports in silico perturbation experiments to predict how altering gene expression influences tissue morphology.

4. Results

The authors validated InSTaPath on multiple datasets, including CRC-100k (colorectal cancer), 10x Visium breast cancer, and Visium HD colorectal cancer.

• Image-Only Performance: On the CRC-100k dataset, the discretized image-word representation (ViT-VQ) combined with topic modeling produced clearer separation of tissue classes (e.g., tumor vs. normal) compared to raw embeddings or standard clustering.
• Spatial Domain Detection (Breast Cancer):
  • InSTaPath achieved an Adjusted Rand Index (ARI) of 0.82, significantly outperforming gene-only baselines (ARI = 0.60) and image-only baselines (ARI = 0.47).
  • It successfully separated morphologically distinct but transcriptionally similar regions (e.g., stromal vs. immune regions) that gene-only methods failed to distinguish.
• Biological Alignment:
  • Topics showed strong correspondence to ground-truth annotations (e.g., Topic 8 = invasive tumor, Topic 4 = adipocytes).
  • GSEA confirmed biological relevance (e.g., tumor topics enriched for oxidative phosphorylation; immune topics for T-cell differentiation).
• In Silico Perturbation:
  • Knocking out top genes associated with tumor topics caused a significant drop in the predicted probability of "tumor" regions in the reconstructed histology.
  • Perturbation of smooth muscle topics resulted in slower declines, demonstrating the model's ability to distinguish tissue-specific gene-morphology dependencies.
  • Differentially expressed image words after perturbation spatially aligned with the expected tissue structures (e.g., tumor regions vs. smooth muscle).

5. Significance

InSTaPath represents a paradigm shift in spatial omics analysis by moving from "feature extraction" to "probabilistic topic modeling." Its significance lies in:

1. Bridging the Gap: It unifies the discrete nature of transcriptomics with the continuous nature of pathology images into a single, interpretable probabilistic framework.
2. Enhanced Interpretability: Researchers can now directly ask, "What genes and what tissue shapes define this specific biological program?"
3. Predictive Power: The ability to simulate gene perturbations and predict morphological outcomes offers a powerful tool for hypothesis generation in cancer biology and drug discovery, potentially identifying how specific genetic alterations drive tissue remodeling.
4. Scalability: By leveraging pretrained foundation models (UNI) and efficient vector quantization, the method is scalable to large whole-slide images and diverse tissue types.