SpatialFusion: A lightweight multimodal foundation model for pathway-informed spatial niche mapping

SpatialFusion is a lightweight multimodal foundation model that integrates histopathology, gene expression, and pathway activity to identify biologically coherent spatial niches, successfully uncovering pre-malignant and stage-predictive microenvironments in colorectal and lung cancers.

The Gist

Imagine you are trying to understand a bustling, chaotic city. You have two main ways to look at it:

1. The Aerial Photo (H&E Images): You can see the buildings, the parks, and the streets. You know where things are, but you can't hear what the people inside are saying or what they are planning.
2. The Phone Records (Gene Expression): You have a list of every text message sent by the citizens. You know what they are talking about (e.g., "We need more food," "Let's build a wall"), but you don't know exactly where they are or what the neighborhood looks like.

For a long time, scientists studying cancer had to choose one or the other, or try to manually glue them together. They missed the big picture: how the neighborhood's shape influences what the people are saying, and how their conversations change the neighborhood.

Enter SpatialFusion. Think of it as a super-smart, lightweight detective that can look at both the aerial photo and the phone records simultaneously to figure out the true "vibe" of every neighborhood in the city.

The Problem: The "Just Neighbors" Trap

Previous tools were like a real estate agent who only cares about who lives next door. If a bakery and a library are next to each other, they are grouped together. But what if the bakery is actually a secret front for a criminal gang, and the library is a hub for peaceful protesters? Just because they are neighbors doesn't mean they are part of the same "functional group."

In cancer, cells that look similar and sit next to each other might be doing completely different things. Some might be quietly growing, while others are screaming "Attack!" to the immune system. Old tools missed these subtle but critical differences.

The Solution: The "Vibe Check" Detective

SpatialFusion is a new AI model that acts like a master detective. Here is how it works, using simple analogies:

1. The "Pre-Trained" Brain (Foundation Models)
Instead of teaching the detective to read from scratch, the researchers gave them a brain that has already read millions of books (scGPT) and looked at millions of photos (UNI).

• Analogy: Imagine hiring a detective who has already studied every city in the world. They don't need to learn what a "house" or a "street" is; they already know. They just need to apply that knowledge to this specific city. This makes the detective incredibly fast and efficient.

2. The "Neighborhood Watch" (Spatial Graph)
The model doesn't just look at one cell; it looks at a cell and its 30 closest neighbors, forming a little "neighborhood."

• Analogy: It's not just about the person in the house; it's about the block party. Who is talking to whom? Is the whole block tense, or is it relaxed?

3. The "Secret Decoder Ring" (Pathway Activity)
This is the magic sauce. The model doesn't just ask "Who is here?" It asks, "What are they doing?" It checks for specific "secret codes" (pathways) like the "Inflammation Signal" or the "Growth Signal."

• Analogy: Imagine two neighborhoods that look identical on a map. In Neighborhood A, everyone is gardening peacefully. In Neighborhood B, everyone is secretly building a bomb. Old tools would say, "These are the same neighborhood." SpatialFusion looks at the "secret codes" and says, "No! Neighborhood B is in a state of high alert!"

What Did They Discover?

The researchers tested this detective on two real-world "cities" (cancer patients) and found things no one else could see:

1. The "Trojan Horse" in the Colon (Colorectal Cancer)
They looked at the tissue next to a tumor. To the naked eye (and standard pathologists), it looked like perfectly healthy, normal tissue.

• The Discovery: SpatialFusion found a "hidden neighborhood" right next to the tumor. Even though the cells looked normal, they were acting weird. They were switching to a "defensive mode," producing extra mucus and sending out distress signals.
• Why it matters: This suggests that the tumor is already "poisoning" the healthy tissue next to it, preparing the ground for the cancer to spread, even before the healthy tissue looks sick. It's like finding a neighborhood that looks peaceful but is secretly stockpiling weapons.

2. The "Two Faces" of Lung Cancer
They looked at lung cancer patients to see if they could predict how aggressive the cancer was (the "stage").

• The Discovery: The model found two distinct types of "bad neighborhoods" (malignant niches).
  • Type A: A busy, growing city with lots of blood vessels (good for growth, but maybe slower spread).
  • Type B: A chaotic, war-torn city with high inflammation and cells that are trying to escape (very aggressive).
• Why it matters: By simply looking at the "vibe" of the tumor, the model could predict if the cancer was in an early stage or a late, dangerous stage with 90% accuracy. It's like a doctor being able to tell if a patient is sick just by listening to the tone of their voice, without needing a full blood test.

Why is this a Big Deal?

• It's Lightweight: Most AI models for this are like supercomputers that need a massive data center. SpatialFusion is so small and efficient it can run on a standard laptop.
• It's Versatile: It can work with paired data (photos + genes) or just photos. This means doctors can use it on old slides from the past that don't have gene data yet.
• It Finds the Invisible: It sees the "functional" reality of the tissue, not just the "visual" reality.

In summary: SpatialFusion is a new tool that helps doctors and scientists see the true story of a tumor. It stops looking at cancer cells as just "neighbors" and starts understanding them as a complex, communicating society with specific moods, plans, and hidden dangers. This could lead to earlier detection of cancer spread and better ways to treat it.

Technical Summary

1. Problem Statement

Current computational approaches for spatial biology face several critical limitations:

• Lack of Integration: Existing methods often fail to fully integrate transcriptomic (gene expression) and morphological (histopathology) information to define functional tissue niches.
• Cell-Centric vs. Niche-Centric: Many foundation models focus on single-cell embeddings rather than multicellular "niches" (microenvironments), missing the coordinated signaling between cell types.
• Pathway Blindness: Most models cluster based on cell identity or spatial proximity alone, failing to distinguish regions with similar cell compositions but distinct functional states (e.g., different pathway activation patterns).
• Resource Intensity: Existing spatial foundation models are often computationally heavy, requiring specialized hardware and large-scale training, limiting accessibility.
• Generalization: Specialist niche-detection tools (e.g., NicheCompass, BANKSY) are not pretrained on diverse corpora, leading to poor batch robustness and an inability to generalize to unseen tissues or technologies without retraining.

2. Methodology: SpatialFusion Architecture

SpatialFusion is a lightweight, multimodal foundation model designed to generate biologically grounded spatial neighborhood embeddings at single-cell resolution. Its architecture consists of three primary modules:

A. Unimodal Encoding (Frozen Foundation Models)

Instead of training encoders from scratch, SpatialFusion leverages pretrained foundation models to extract features, ensuring batch robustness and biological fidelity:

• Transcriptomics: Uses scGPT to embed raw gene expression counts.
• Histopathology: Uses UNI (a vision foundation model) to embed 256×256 pixel H&E image patches centered on each cell.
• Note: These encoders are frozen; only the subsequent alignment and spatial modules are trained.

B. Multimodal Alignment Module (mAE)

A Multimodal Autoencoder (mAE) aligns the transcriptomic and histological embeddings into a shared latent space.

• Architecture: Two independent encoders map modalities to a shared latent space, followed by decoders that reconstruct both within-modality and cross-modality inputs.
• Objective: Minimizes reconstruction loss and enforces latent alignment, ensuring the joint representation captures concordant biological variation across modalities.

C. Spatial Encoding Module (GCMAE)

The aligned embeddings are processed by a Graph Convolutional Masked Autoencoder (GCMAE) to model spatial neighborhoods.

• Graph Construction: Tissues are represented as k-nearest-neighbor (k=30) graphs where nodes are cells and edges represent spatial proximity.
• Masking Strategy: A high masking ratio (randomly masking nodes) forces the model to learn robust neighborhood representations by reconstructing missing features from neighbors.
• Pathway Regression Loss (Key Innovation): An auxiliary loss function predicts PROGENy pathway activity scores (e.g., EGFR, JAK-STAT, NF-κB) for each cell. This explicitly forces the latent space to encode functional signaling states, allowing the model to distinguish niches with similar cell types but different pathway activations.

D. Efficiency

• Parameters: The model has fewer than 300,000 learnable parameters (~280k in mAE, ~2k in GCMAE).
• Hardware: Requires <5 MB VRAM for training and <2 MB for inference, runnable on standard laptops.
• Inference Modes: Supports paired H&E + ST inference and H&E-only inference (using the image branch of the mAE), enabling application to cohorts lacking spatial transcriptomics.

3. Key Contributions

1. Pathway-Informed Niche Mapping: Unlike previous models that cluster by cell identity, SpatialFusion explicitly learns to separate spatial neighborhoods based on pathway activation patterns, revealing functional microenvironments.
2. Lightweight Multimodal Foundation Model: It successfully combines the generalization power of foundation models (via frozen encoders) with the niche-centric design of specialist tools, all while maintaining extreme computational efficiency.
3. Zero-Shot Generalization: The model demonstrates strong zero-shot performance on unseen tissue types (e.g., prostate, lung) and unseen technologies (e.g., Visium HD) without retraining.
4. Discovery of "Invisible" Niches: By integrating morphology and pathway data, it identifies biologically distinct niches that are morphologically indistinguishable on standard H&E staining.

4. Results

Benchmarking Performance

• Ovarian Cancer (OVCA): SpatialFusion outperformed specialist methods (NicheCompass, BANKSY) and other spatial foundation models (NicheFormer, scGPT-spatial, OmiCLIP) in rediscovering pathologist-annotated niches. It achieved superior Adjusted Rand Index (ARI) and spatial coherence metrics, particularly in delineating complex structures like fallopian tubes and necrotic zones.
• Breast Cancer (BRCA): Successfully resolved distinct edge and core compartments within Ductal Carcinoma In Situ (DCIS), capturing the myoepithelial barrier and associated TGF-β signaling gradients that other methods missed.
• Unseen Tissues/Technologies:
  • Prostate (PRAD): Showed competitive performance on unseen Xenium data; fine-tuning significantly improved results.
  • Lung (Visium HD): Achieved the best performance among all models on an unseen Visium HD platform, demonstrating robust cross-platform generalization.

Biological Discoveries

1. Pre-malignant Niche in Colorectal Cancer (CRC):

   • Applied to a Visium HD CRC cohort, SpatialFusion identified a "defensive" niche (Niche 2) in morphologically normal mucosa adjacent to tumors.
   • While indistinguishable from normal mucosa (Niche 1) by H&E, Niche 2 exhibited a secretory, inflamed phenotype with upregulated antimicrobial programs and altered JAK-STAT signaling. This suggests a field cancerization effect invisible to standard pathology.

2. T-Stage Predictive Niches in Non-Small Cell Lung Cancer (NSCLC):

   • Applied to a Visium HD NSCLC cohort, the model identified malignant niches predictive of pathological tumor stage (pT).
   • Niche 10: Associated with smaller tumors; characterized by proliferative, well-vascularized, basal-like phenotypes (high PI3K/VEGF).
   • Niche 16: Associated with advanced stages (pT3/4); characterized by an aggressive, inflammation-activated phenotype with high JAK-STAT, TNFα, and NF-κB activity, and features of Epithelial-to-Mesenchymal Transition (EMT).
   • An Attention-based Multiple Instance Learning (ABMIL) classifier using SpatialFusion embeddings achieved 0.8 AUC in predicting tumor stage, significantly outperforming gene-expression-only baselines.

5. Significance

• Clinical Utility: SpatialFusion provides a scalable framework for discovering morpho-molecular niches that correlate with disease progression and therapeutic targets (e.g., JAK-STAT inhibitors).
• Accessibility: Its lightweight design democratizes access to spatial foundation models, removing the barrier of high-performance computing requirements.
• Paradigm Shift: It moves the field from "cell-type mapping" to "functional niche mapping," acknowledging that tissue organization is defined by coordinated pathway activation rather than just cell identity.
• Future Directions: The ability to perform H&E-only inference suggests potential for retrospective analysis of vast archival pathology datasets to uncover spatial biomarkers for prognosis and treatment response.

In summary, SpatialFusion represents a significant advancement in spatial biology by creating a computationally efficient, pathway-aware foundation model that uncovers biologically and clinically relevant tissue microenvironments previously obscured by single-modality analysis.