Towards Cross-Sample Alignment for Multi-Modal Representation Learning in Spatial Transcriptomics

This paper introduces a deep representation learning framework that integrates foundation models for transcriptomics and pathology to robustly align multi-modal spatial transcriptomics data across diverse patient cohorts, significantly outperforming conventional batch-correction methods in clustering cells by type rather than dataset-specific conditions.

The Gist

Imagine you are trying to solve a massive, complex jigsaw puzzle. But here's the catch: the puzzle pieces don't all come from the same box. Some pieces are from a box labeled "Patient A," others from "Patient B," and they were all taken by different people using slightly different cameras and lighting.

If you try to put these pieces together just by looking at the colors (which represents gene expression or the chemical instructions inside cells), the picture gets messy. The pieces from Patient A might stick together just because they were photographed in the morning, while Patient B's pieces stick together because they were photographed in the afternoon. You end up with a picture of "Morning vs. Afternoon" instead of the actual image of a healthy heart or a tumor.

This is the problem scientists face with Spatial Transcriptomics (ST). They have a goldmine of data showing where genes are active in tissue, but the data is "noisy" because of technical differences between patients and labs.

The Paper's Big Idea: A Three-Part Recipe

The authors of this paper propose a new way to clean up this mess and build a perfect picture. They call their method AESTETIK (a fancy name for their framework). Think of it as a recipe that mixes three ingredients to get the best result:

1. The Chemical Recipe (Transcriptomics): This is the list of genes turning on and off. It's like the text written on the puzzle pieces.
2. The Visual Texture (Morphology): This is the actual photo of the tissue. It shows the shape of the cells, the texture of the skin, or the structure of a tumor. It's like the picture printed on the puzzle piece.
3. The Map (Spatial Context): This is knowing exactly where the piece fits in the room. It's the "neighborhood" information.

How It Works: The "Smart Glue"

The researchers realized that if you only look at the text (genes), you get confused by the "batch effects" (the morning/afternoon lighting issue). But if you also look at the picture (morphology) and the map (location), you can see the real pattern.

They built a "Smart Glue" (a deep learning model) that does two things at once:

• Horizontal Alignment: It takes pieces from different patients and smoothes out the differences caused by the camera or the lab. It says, "Hey, this tumor cell from Patient A looks and acts just like this tumor cell from Patient B, even though the lighting was different."
• Vertical Integration: It looks at the gene text, the tissue photo, and the location all at the same time to figure out what the cell actually is.

The Analogy: The International Potluck

Imagine a potluck dinner where everyone brings a dish, but the descriptions on the labels are written in different languages and some are smudged (the batch effects).

• Old Method: You try to sort the dishes just by reading the smudged labels. You end up grouping all the "English" dishes together and all the "French" dishes together, even if they are both lasagna.
• This Paper's Method: You also look at the food itself (does it look like pasta?) and where it's sitting (is it next to the salad?). By combining the label, the look, and the location, you can correctly identify that the "English Lasagna" and the "French Lasagna" are actually the same dish. You can now build a "Global Lasagna Atlas" that shows you all the variations of lasagna from around the world, rather than just a list of languages.

The Results: A Clearer Picture

The team tested this on three very different "puzzles":

1. Human Brains: Sorting out the different layers of neurons.
2. Skin Melanoma: Identifying different parts of a skin cancer tumor.
3. Lung Cancer: Finding specific zones within a lung tumor.

The Outcome:
Their new method was a huge success.

• For the brain data, it was 38% better at finding the right groups than old methods.
• For the lung cancer data, it was 2 times (200%) better.
• For the skin cancer data, it was 58% better.

In the lung cancer example, the old methods couldn't tell the difference between "Patient A's tumor" and "Patient B's tumor." The new method successfully ignored the patient differences and grouped the cells by what they actually were (e.g., "Tumor Core," "Healthy Edge," "Immune Attack Zone").

Why This Matters

Before this, scientists had to study patients one by one, like looking at a single star in the night sky. This paper gives them a telescope that lets them see the constellation.

By aligning data from many different people, they can now find universal rules of biology. They can discover:

• "Oh, every lung cancer tumor has this specific little neighborhood of immune cells trying to fight it."
• "This specific gene pattern always appears in the same spot in the brain, no matter who the patient is."

In a Nutshell

This paper is about teaching computers to ignore the "noise" of different labs and patients so they can focus on the "signal" of what cells are actually doing. By combining genes, photos, and maps, they created a tool that helps us see the true, shared architecture of our bodies across different people, leading to better understanding of diseases like cancer and brain disorders.

Technical Summary

1. Problem Statement

Spatial Transcriptomics (ST) datasets are rapidly expanding, offering simultaneous mapping of tissue morphology, gene expression, and spatial context. However, integrating these datasets across different patients (horizontal integration) and modalities (vertical integration) faces significant challenges:

• Batch Effects & Variability: Technical artifacts, patient-specific variability, and local microenvironment differences often dominate the signal, causing cells to cluster by donor or dataset rather than by biological cell type.
• Limitations of Existing Methods:
  • Transcriptomics-only batch correction (e.g., Harmony, scVI, Scanorama) effectively removes technical noise but ignores spatial context and morphology, limiting their applicability to ST data where location is critical.
  • Per-sample analysis fragments data, preventing the discovery of conserved biological patterns across patients.
  • General image models used for morphology often fail to capture fine-grained histopathological structures relevant to biology.

The authors hypothesize that combining specialized transcriptomics correction methods with deep multi-modal representation learning can align morphology, transcriptomics, and spatial information across multiple tissue samples, enabling the construction of comprehensive, multi-donor ST atlases.

2. Methodology

The proposed framework, built upon the AESTETIK architecture, integrates data through a two-step process: Horizontal Alignment followed by Vertical Multi-Modal Integration.

A. Data Preprocessing & Horizontal Alignment

1. Input Representation: Each ST spot $i$ is defined by a transcriptomics vector ($x_i$), a morphology vector ($m_i$), and spatial coordinates ($s_i$).
2. Batch Correction: To mitigate donor-specific and technical variation, established batch-correction algorithms (Harmony, scVI, or Scanorama) are applied independently to the transcriptomics and morphology features.
   • Transcriptomics: $\tilde{x}_i = \text{BatchCorrect}(x_i | \text{sample}_i)$
   • Morphology: $\tilde{m}_i = \text{BatchCorrect}(m_i | \text{sample}_i)$ (using Harmony).
3. Foundation Models: The framework leverages recent foundation models to generate richer embeddings:
   • Transcriptomics: Uses models like CancerFoundation (trained on malignant cells) instead of standard PCA.
   • Morphology: Uses Pathology Foundation Models (e.g., UNI2-h) trained on histopathology data, outperforming general-purpose image models (e.g., Inception v3).

B. Vertical Multi-Modal Integration (AESTETIK)

The corrected features are integrated into a unified representation using a convolutional autoencoder:

1. Grid Construction: Principal components of transcriptomics and morphology are concatenated and augmented with local spatial neighborhoods to form a tensor image-like grid ($spot_i \in \mathbb{R}^{N_{grid} \times N_{grid} \times 2n_{PCA}}$).
2. Embedding Learning: A convolutional autoencoder ($f_{AESTETIK}$) computes spot embeddings ($z_i$).
3. Loss Function: The model is trained with a composite loss function balancing morphology and transcriptomics contributions:
   $$L_{AESTETIK} = \alpha \cdot (L^m_{MSE} + L^m_{triplet}) + (3 - \alpha) \cdot (L^{tr}_{MSE} + L^{tr}_{triplet})$$
   • Multi-Triplet Loss: Encourages spots with similar labels (derived from precomputed K-Means clusters) to be close in embedding space while pushing dissimilar spots apart. This enables self-supervised learning without ground-truth labels.
   • Hyperparameter $\alpha$: Controls the relative weight of morphology vs. transcriptomics.

C. Evaluation Strategy

The framework is evaluated using Nested Cross-Validation (nCV) to prevent data leakage:

• Single-Donor Integration: Evaluates alignment of adjacent tissue sections from the same donor.
• Multi-Donor Integration: Evaluates alignment of samples across different donors.
• Metrics: A composite score balancing biological conservation (ARI, NMI, Silhouette) and batch mixing (iLISI, kBET).

3. Key Contributions

1. Unified Framework: Proposes a novel pipeline that combines horizontal batch correction with vertical multi-modal deep learning to align ST data across patients and modalities.
2. Foundation Model Integration: Demonstrates that replacing standard linear projections with specialized foundation models (CancerFoundation for genes, UNI for pathology) significantly enhances representation quality.
3. Spatial Context Preservation: Introduces a grid-based convolutional approach that explicitly models local spatial neighborhoods, proving that spatial context is crucial for preserving biological niches.
4. Scalability: The architecture is scalable to millions of cells, leveraging efficient convolutional autoencoders and self-supervised triplet losses.

4. Results

The framework was benchmarked on 18 melanoma, 12 human brain, and 4 lung cancer datasets.

• Performance Gains: The proposed method outperformed conventional batch-correction approaches (Harmony, Scanorama, scVI) by:
  • 58% in skin melanoma.
  • 38% in human brain.
  • 2-fold in lung cancer (increasing ARI from 0.18 to 0.50).
• Multi-Modal Synergy: Joint integration of morphology, transcriptomics, and spatial data consistently produced clusters closer to biological ground truth than transcriptomics-only integration.
• Foundation Model Impact: Using pathology foundation models (UNI2-h) for morphology improved domain identification over general image models. Similarly, using CancerFoundation embeddings improved alignment in tumor datasets.
• Ablation Study:
  • Spatial Window Size: Performance peaked at a spatial window size of 5; larger windows (7) diluted local signals.
  • Modality Weight: The model effectively leveraged multiple data sources rather than relying on a single modality.
• Biological Validation: Downstream pathway analysis (via decoupleR) on the integrated clusters revealed biologically consistent signals, such as increased PI3K/MAPK activity in tumor clusters and WNT activity in repair/regeneration zones.

5. Significance

This work addresses a critical bottleneck in spatial transcriptomics: the inability to systematically integrate data across diverse patient cohorts.

• Robust Atlas Construction: It enables the creation of comprehensive multi-modal ST atlases that span multiple donors, facilitating the discovery of conserved cellular programs and spatial niches.
• Biological Insight: By aligning cells by cell type rather than dataset-specific conditions, the method reveals true biological heterogeneity and tissue organization that was previously obscured by batch effects.
• Future Direction: The paper advocates for a shift from separate batch correction and representation learning steps toward end-to-end frameworks that jointly learn complex tissue patterns, paving the way for applications in high-resolution technologies (e.g., Visium HD) and precision medicine.

Code Availability: The implementation is open-source at https://github.com/ratschlab/aestetik.