Multimodal spatial alignment and morphology mapping with MOSAICField

MOSAICField is a unified deep learning framework that integrates diverse spatial omics data across multiple tissue slices to perform both physical 3D reconstruction and morphological alignment of anatomical structures, thereby enabling comprehensive multimodal tissue and tumor atlas building.

The Gist

Imagine you are trying to build a perfect 3D model of a city, but you only have a stack of 2D photos. Some photos are taken from a drone (showing the whole neighborhood), some are close-up street-level shots, and some are infrared thermal images. The problem? The photos were taken at different times, the camera angles are slightly off, the streets have shifted, and the images don't share any common landmarks.

This is exactly the challenge scientists face when studying tissues like tumors. They slice a piece of tissue into hundreds of thin layers. They use different "cameras" (technologies) on different slices: one slice might show which genes are active (transcriptomics), another shows protein levels (proteomics), and another just shows the tissue's color and shape (histology).

The goal is to stack these slices back together to see the whole 3D picture of the disease. But because the slices are different and the tissue gets squished or stretched during preparation, they don't line up perfectly.

Enter MOSAICField, a new computer program that acts like a super-smart, magical puzzle solver.

The Two Big Problems It Solves

The authors realized there are actually two different types of "lining up" needed, and they gave them distinct names:

1. Physical Alignment (The "Stacking" Problem):
   Imagine you have a deck of cards that has been shuffled and bent. You need to straighten the deck so the cards are in the right order and the edges match up. This is Physical Alignment. It corrects for the fact that the tissue was cut at a slight angle or stretched. It puts every slice in its correct 3D position relative to its neighbors, creating a solid, coherent block of tissue.

2. Morphological Alignment (The "Worm" Problem):
   Now, imagine a worm (or a pipe, or a blood vessel) running through that deck of cards. If you cut the deck at an angle, the worm will look like it jumps around from card to card. It doesn't follow a straight line up and down; it weaves through the tissue.
   Morphological Alignment is about tracking that worm. It asks: "Even though this slice is tilted, where does this specific duct or blood vessel go next?" It connects the dots of complex shapes that cut across the slices at weird angles, ensuring the "worm" looks like one continuous tube in the final 3D model, not a broken string of beads.

How MOSAICField Works (The Magic Trick)

Most old methods tried to match the slices by looking for common features (like saying, "This red spot on the gene map must match this red spot on the protein map"). But what if the gene map has no red spots, or the protein map uses a completely different color system? They fail.

MOSAICField is different. It doesn't need the slices to speak the same language.

• Step 1: The Rough Sketch (Affine Alignment):
  First, it looks at the big picture. It figures out how much to rotate, stretch, or shift a slice to make it roughly fit next to its neighbor. It's like roughly shoving two puzzle pieces together to see if they are in the right neighborhood.

• Step 2: The Fine-Tuning (Neural Deformation):
  This is where the magic happens. The program uses a deep learning "brain" (a neural network) to learn a deformation field.

  • Think of the tissue slice as a rubber sheet.
  • The program learns exactly how to stretch, squish, and warp that rubber sheet so that the patterns (like the shape of a cell cluster or the texture of the tissue) line up perfectly with the next slice.
  • Crucially, it does this by looking at the structure of the image (like how a neighborhood looks), not just the specific data points. It's like recognizing a face even if the person is wearing a different hat or the photo is black and white.

Why This Matters

The researchers tested this on a prostate cancer sample from the Human Tumor Atlas Network. They had slices from three completely different technologies that didn't share any common data points.

• The Result: MOSAICField successfully built a stunning 3D model of the tumor.
• The "Worm" Tracking: It didn't just stack the slices; it successfully tracked the complex network of prostate ducts (the "worms") through the entire 3D tumor, showing exactly how they twist and turn.
• The Bonus: Because the slices are now perfectly aligned, scientists can finally ask questions like, "Do the cells with high gene activity sit right next to the cells with high protein levels?" Previously, this was impossible because the slices were misaligned.

The Takeaway

MOSAICField is like a universal translator and a 3D printer for biology. It takes messy, mismatched, multi-language slices of tissue and stitches them together into a single, high-definition 3D movie of what's happening inside a tumor. This allows doctors and scientists to see the true shape of diseases and how they evolve in three dimensions, which is a massive leap forward for understanding cancer and developing better treatments.

Technical Summary

1. Problem Statement

The field of spatial biology is advancing rapidly with technologies capable of measuring transcriptomics, proteomics, and epigenomics at high spatial resolution. However, a major bottleneck exists in integrating data across different spatial technologies (modalities) and reconstructing 3D tissue architectures from serial 2D slices.

Current challenges include:

• Cross-Modality Misalignment: Existing methods often fail to align slices from different modalities (e.g., spatial transcriptomics vs. H&E histology vs. multiplex protein imaging) because they lack common molecular features.
• Physical vs. Morphological Distinction: Most alignment tools treat all spatial deformations as a single problem. They fail to distinguish between:
  1. Physical Alignment: Reconstructing the original 3D geometry of the tissue by correcting for global distortions (rotation, scaling, translation, and non-linear warping) caused by sample preparation.
  2. Morphological Alignment: Tracking specific anatomical structures (e.g., ducts, blood vessels, tumor boundaries) that traverse the tissue at oblique angles relative to the slicing direction. These structures require local, non-linear tracking that physical alignment alone cannot capture.
• Limitations of Existing Tools: Methods like PASTE, STalign, SANTO, and CAST either rely on shared features (which don't exist across distinct modalities) or assume rigid/linear transformations, failing to capture complex local tissue deformations.

2. Methodology: MOSAICField

The authors introduce MOSAICField (Multimodal Spatial Alignment and Integration with Coordinate neural Field), a unified framework that performs two distinct types of alignment using a deep learning-based approach.

Core Architecture

The method operates in two sequential steps for every pair of adjacent slices:

Step 1: Affine Alignment (Global Initialization)

• Goal: Estimate global translation, rotation, and scaling.
• Mechanism: Uses an extension of Global Invariant Optimal Transport (OT).
• Objective Function: Minimizes a loss function combining:
  1. Wasserstein Term: Measures spatial distance between transformed source coordinates and target coordinates.
  2. Gromov-Wasserstein Term: Measures the structural similarity of features within each slice (e.g., gene-gene distances vs. protein-protein distances). This allows alignment even when slices share zero common features (e.g., RNA vs. Protein).
• Optimization: Solves for a coupling matrix ($\Pi$) and an affine transformation matrix ($T$) simultaneously using block coordinate descent.

Step 2: Nonlinear Deformation via Neural Fields

• Goal: Refine the alignment to capture complex, local non-linear distortions.
• Mechanism: Treats slices as multi-channel images and learns a neural deformation field ($\phi$) using a Multi-Layer Perceptron (MLP). The MLP takes 2D coordinates as input and outputs a displacement vector.
• Loss Function:
  • Similarity Loss ($L_{sim}$): Uses a Generalized Modality Independent Neighborhood Descriptor (MIND). This descriptor encodes local structural patterns (patch differences) rather than raw pixel intensities, making it robust across different modalities (e.g., matching H&E texture to gene expression patterns). It is combined with a Mean Squared Error (MSE) loss on normalized grayscale intensities.
  • Regularization ($L_{reg}$): Includes Jacobian regularization (to prevent folding/inversion) and magnitude regularization (to limit displacement).

Dual-Mode Alignment Strategy

MOSAICField uniquely separates the problem into two scales:

1. Physical Alignment ($\phi_P$): Computed on a global, low-resolution scale. It reconstructs the 3D tissue model by aligning the entire tissue block, correcting for global distortions to place slices in their correct relative 3D positions.
2. Morphological Alignment ($\phi_M$): Computed on a local, full-resolution scale after physical alignment. It focuses on specific regions of interest (ROIs) to track structures (like ducts) that may curve or twist through the tissue. This captures fine-grained, non-linear deformations that global alignment misses.

3. Key Contributions

• First Cross-Modality Framework: MOSAICField is the first method to successfully align spatial slices from completely different modalities (e.g., Xenium transcriptomics, CODEX proteomics, and H&E histology) without requiring shared molecular features.
• Dual-Alignment Formulation: It explicitly distinguishes between physical alignment (restoring 3D geometry) and morphological alignment (tracking oblique anatomical structures), solving a problem previously addressed only via manual annotation or assumed to be identical.
• Neural Implicit Representations: It leverages neural fields to model complex, continuous deformation fields, outperforming methods based on B-splines or rigid graph matching.
• Generalized MIND Loss: Introduces a modality-independent neighborhood descriptor that enables structural comparison between images with different channel counts and physical meanings.

4. Results

The authors evaluated MOSAICField on both simulated data and a real-world dataset from the Human Tumor Atlas Network (HTAN).

A. Simulated Data Evaluation

• Physical Alignment: On simulated mouse embryo slices with global distortions (rotation, scaling, sinusoidal warping) and disjoint feature sets, MOSAICField achieved the lowest Registration Error and highest Label Transfer Adjusted Rand Index (LTARI), significantly outperforming STalign, PASTE, SANTO, and CAST.
• Morphological Alignment: In a simulation where a hollow lumen traversed slices obliquely, MOSAICField achieved the highest Jaccard Similarity in tracking the lumen's shape, whereas other methods failed to align the structure correctly due to reliance on shared features or rigid transformations.

B. Real-World Application: Prostate Cancer (HTAN HT891Z1)

• Dataset: 16 consecutive slices from a prostate tumor, measured using three platforms: 10x Genomics Xenium (475 genes), CODEX (25 proteins), and H&E staining.
• Physical Alignment:
  • MOSAICField constructed a coherent 3D model of the tumor.
  • It achieved near-perfect Jaccard Similarity (0.98–1.00) between adjacent slices, outperforming competitors.
  • It significantly improved feature correlations (Pearson correlation) between aligned locations. For example, protein-protein and gene-protein correlations increased by 10-fold compared to unaligned data and 1.8-fold over affine-only alignment.
• Morphological Alignment:
  • The method successfully tracked the prostatic ductal system across the 3D volume, capturing complex shape changes (shrinking, expanding, twisting) that physical alignment alone missed.
  • Compared to manual annotations, MOSAICField's morphological tracking showed a 35% to 100% relative improvement in Jaccard similarity over the next best method (STalign).
  • Functional Inference: By accurately aligning the 3D space, MOSAICField enabled the discovery of strong correlations between the CD44 gene (Xenium) and CD44 protein (CODEX) within specific ductal regions, demonstrating its ability to infer functional relationships across modalities.

5. Significance and Impact

• Enabling 3D Tumor Atlases: MOSAICField provides the computational infrastructure necessary to build comprehensive 3D atlases of tumors and tissues, integrating diverse "omics" layers into a single spatial context.
• Biological Discovery: By enabling the tracking of morphological structures (like ducts and vasculature) in 3D, it facilitates the study of dynamic biological processes such as tumor invasion, angiogenesis, and tissue regeneration.
• Methodological Advancement: The separation of physical and morphological alignment sets a new standard for spatial data analysis, moving beyond simple registration to true structural tracking.
• Open Source: The code is publicly available, allowing the community to apply this framework to diverse spatial datasets.

In summary, MOSAICField represents a significant leap forward in spatial biology, solving the critical challenge of integrating heterogeneous spatial data into accurate 3D models and enabling the automated tracking of complex tissue architectures.