An explanatory benchmark of spatial domain detection reveals key drivers of method performance

This paper presents a comprehensive benchmark of 26 spatial domain detection methods across diverse real and semi-synthetic datasets, revealing that performance is primarily driven by data resolution and cellular heterogeneity rather than architectural novelty, and introduces a modular framework to guide future tool development and selection.

The Gist

Imagine you are looking at a massive, bustling city from a helicopter. You can see the buildings, the parks, the busy streets, and the quiet neighborhoods. In the world of biology, this "city" is a piece of tissue (like your brain or a tumor), and the "buildings" are individual cells.

For a long time, scientists could only look at the cells one by one, like reading a phone book. They knew what genes were inside each cell, but they lost the map. Spatial Transcriptomics is like finally getting that helicopter view: it tells us not just what the cells are, but where they are sitting in the tissue.

However, just having a map isn't enough. We need to figure out the neighborhoods. Which cells belong to the "downtown" area? Which ones are in the "suburbs"? This is called Spatial Domain Detection.

The problem? There are dozens of different computer programs (algorithms) trying to draw these neighborhood lines, and they all claim to be the best. But until now, nobody had a fair way to test them. Some programs were only tested on one specific city, while others were tested on a different one, making it impossible to know who was actually the best driver.

The Big Experiment: The "City Simulator"

The authors of this paper decided to settle the debate. They didn't just look at a few real cities; they built a giant, flexible simulator.

Think of it like a video game engine for biology. They created over 1,000 fake tissue samples where they could control every single variable:

• Resolution: Could they see individual houses (cells) or just whole city blocks (spots)?
• The "Gene Panel": Did they have a full encyclopedia of every building's purpose, or just a tiny pamphlet with 33 words?
• Noise: Did the city have fog (missing data) or random construction zones (biological noise)?

They ran 26 different computer programs through this simulator and also tested them on 63 real tissue samples from six different technologies.

What They Discovered

Here are the key takeaways, translated into everyday language:

1. The "High-Res" vs. "Low-Res" Divide
Some programs are like sports cars: they zoom beautifully on high-resolution data (where you can see every single cell), but they crash on low-resolution data (where cells are blurred together). Other programs are like trucks: they are sturdy and handle the blurry, low-resolution data well, but they aren't as fast or precise on the high-res stuff.

• The Lesson: There is no "one size fits all." You have to pick the right vehicle for the road you are driving on.

2. The "Neighborhood" Matters
Some programs are great at finding neighborhoods that look very different from each other (like a park vs. a factory). But when the neighborhoods look very similar (like two different types of apartments), many programs get confused and mix them up.

• The Lesson: If your tissue is very uniform, you need a very sensitive tool. If it's very diverse, almost any tool will work.

3. The "Randomness" Problem
Many of these computer programs have a "shuffle" button. If you run the same program twice on the same data, it might give you slightly different results because of random numbers used inside the code. The authors found that some programs are very stable (like a rock), while others are jittery (like a leaf in the wind).

• The Lesson: If you use a jittery program, your results might change just because you ran it at a different time of day.

4. The Secret Sauce isn't the Engine
The authors took apart the most popular programs (the ones using complex Neural Networks, which are like fancy AI engines) and swapped their parts. They found that the "engine" (the complex math) wasn't the most important part.

• The Analogy: It's like building a car. You can have a Ferrari engine, but if you put it on a bicycle frame with bad tires, it won't go fast. The preparation (cleaning the data) and the final step (grouping the results) mattered more than the fancy AI architecture itself.

5. The Power of the "Crowd"
When they combined the results of all the programs into one "consensus" map, it was often better than any single program working alone.

• The Lesson: It's like asking a committee of experts instead of just one person. Even if one expert is wrong, the group usually gets it right.

Why This Matters

This paper is a user manual for the future.

• For Scientists: It tells them exactly which tool to pick based on their specific experiment (e.g., "If you have low-resolution data, use Tool X. If you have high-resolution data, use Tool Y").
• For Developers: It tells them to stop obsessing over making their AI "fancier" and start focusing on cleaning their data and making their software easier to use.

In short, the authors built a massive testing ground that stopped the guessing game. They showed us that in the world of mapping biological cities, the best tool depends entirely on the terrain you are exploring.

Technical Summary

1. Problem Statement

Spatial transcriptomics (ST) enables genome-wide mapping of gene expression within tissue context, making the identification of spatial domains (regions with distinct transcriptional and spatial characteristics) a critical downstream task. However, the field suffers from:

• Conflicting Conclusions: Existing benchmarks often rely on narrow datasets (e.g., a single 10x Visium dataset) and specific metrics, leading to contradictory claims about which methods perform best.
• Lack of Explanatory Power: Current evaluations are largely descriptive ("which method wins?") rather than explanatory ("why does a method fail or succeed under specific conditions?").
• Hidden Variability: Many methods contain stochastic elements (random initialization, data augmentation) that are often fixed or uncontrolled in benchmarks, masking inherent instability.
• Limited Parameter Space: Real-world datasets cover only a small fraction of the possible combinations of technological parameters (resolution, sparsity, gene panel size) and biological factors (domain shape, cellular heterogeneity).

2. Methodology

The authors present a comprehensive, explanatory benchmarking framework that combines real data evaluation with systematic semi-synthetic simulation.

A. Benchmark Scope

• Methods Evaluated: 26 computational methods categorized into four groups:
  1. Clustering-based: (e.g., BANKSY, CellCharter)
  2. Neural Network-based: (e.g., GraphST, STAGATE, SEDR, SpaceFlow)
  3. Statistical Model-based: (e.g., BayesSpace, BASS, PRECAST)
  4. Image Segmentation-based: (e.g., Vesalius)
• Baselines: Non-spatial clustering (Leiden in Scanpy/Seurat) and spatially smoothed non-spatial baselines.
• Real Datasets: 63 tissue sections from 6 different ST technologies (including 10x Visium, MERFISH, Slide-seq, osmFISH, STARmap) covering varying resolutions, gene panel sizes, and tissue types.
• Semi-Synthetic Data: A flexible pipeline generating >1,000 datasets by combining:
  • Ground Truth: Cell coordinates and domain labels generated in silico with varying shapes (laminar, circular, concentric, complex).
  • Expression Profiles: Derived from a single-nucleus RNA-seq dataset of the mouse brain.
  • Systematic Perturbations:
    • Technological: Resolution (spot aggregation), Gene Panel Size (downsampling), and Sparsity (zero-inflation).
    • Biological: Type I (ambient RNA/noise addition) and Type II (cellular infiltration/heterogeneity).

B. Evaluation Metrics

• Accuracy: Adjusted Rand Index (ARI) against expert manual annotations or ground truth.
• Spatial Coherence: Percentage of Abnormal Spots (PAS), measuring how often a spot's label differs from the majority of its neighbors.
• Stability: Quantified by permuting input cell order (without changing data values) to force different random seeds, measuring variance in ARI across 12 trials.
• Scalability: Runtime and memory usage on datasets ranging from 2,000 to 100,000 cells.
• Usability: A weighted checklist scoring availability, maintenance, and documentation.

C. Ablation Study

A modular, plug-and-play framework was developed for six neural network-based methods. The authors decomposed these methods into four independent modules:

1. Preprocessing
2. Adjacency/Graph Construction
3. Neural Network Architecture/Training
4. Clustering
   They systematically swapped these modules between methods to isolate the contribution of each component to overall performance.

3. Key Contributions

1. Explanatory Benchmarking: Shifts the paradigm from descriptive ranking to identifying the drivers of performance (e.g., how resolution or heterogeneity specifically impacts different algorithm classes).
2. Semi-Synthetic Simulation Pipeline: A robust tool for generating ground-truth data with controlled variations in resolution, sparsity, and tissue architecture, overcoming the limitations of real-data ground truth uncertainty.
3. Stochasticity Quantification: A novel strategy to expose hidden variability in methods by permuting input order, revealing that stability is often driven by pipeline choices (preprocessing/postprocessing) rather than algorithmic class.
4. Modular Framework: An open-source Snakemake pipeline that allows for component exchange and consensus analysis, facilitating method refinement and development.

4. Key Results

A. Performance Drivers

• Resolution Dependence: Spatial methods generally outperform non-spatial baselines, but the gain is highly resolution-dependent.
  • High Resolution (MERFISH): Spatial methods yield massive improvements (ARI increase up to 0.48).
  • Low Resolution (Visium): Gains are modest (ARI increase ~0.16).
  • Extreme Low Resolution/High Sparsity (Slide-seq): Performance drops significantly across nearly all methods.
• Gene Panel Size: Performance degrades as the number of profiled genes decreases. Methods like BANKSY and CellCharter show sharp drops with small panels (<1,000 genes).
• Cellular Heterogeneity: This is the dominant factor. Methods that assume high transcriptional distinctness fail when domains contain heterogeneous cell types. Robust methods (e.g., BASS, SpaceFlow, SpaDo) maintain performance even with 50-70% infiltrating cell content.
• Spatial Coherence vs. Accuracy: In high-resolution data, there is a strong negative correlation (Spearman $\rho \approx -0.85$) between ARI and PAS. Methods that enforce strong spatial smoothness perform better on high-resolution data where expression patterns are less smooth, while transcriptional-similarity-focused methods perform better on lower-resolution data.

B. Stochasticity

• Many methods exhibit significant variability across runs due to random initialization or input ordering.
• Stable Methods: CCST, MNMST, and SpatialPCA showed minimal variability.
• Unstable Methods: SpiceMix and STAGATE showed high variability.
• Driver of Stability: Stability is determined more by preprocessing choices (e.g., PCA vs. variable gene selection) and regularization than by the core neural network architecture.

C. Ablation Study Findings

• Architecture is Secondary: Swapping neural network architectures (e.g., autoencoders vs. DGI) had little impact on performance for most methods.
• Preprocessing & Clustering are Critical: The choice of preprocessing (dimensionality reduction) and the final clustering algorithm (e.g., k-means vs. mclust) had the largest impact on performance.
• Consensus Strategy: Aggregating outputs from multiple methods via a consensus approach consistently outperformed individual methods, particularly on Visium datasets.

D. Scalability & Usability

• Scalability: Runtime and memory usage vary by orders of magnitude. BANKSY is fast and memory-efficient, while MERINGUE and GraphPCA can require >200GB RAM for 100k cells.
• Usability: TACCO, PAST, and PRECAST scored highest on documentation and maintenance. Many tools suffer from fragmented documentation and discrepancies between published descriptions and code.

5. Significance and Recommendations

• For Users: Method selection must be tailored to the specific data characteristics.
  • For high-resolution, heterogeneous data: Use robust methods like BASS, SpaDo, or SpaceFlow.
  • For low-resolution data: Spatial methods offer marginal gains; non-spatial baselines with smoothing may suffice.
  • For large datasets: Prioritize scalable tools like TACCO, PAST, or SpaceFlow.
• For Developers:
  • Focus on robustness to cellular heterogeneity rather than just architectural novelty.
  • Prioritize stability through careful preprocessing and regularization.
  • Improve usability (documentation, maintenance) as it is currently a major bottleneck.
  • Consider consensus approaches as a viable strategy to improve robustness without developing new complex architectures.

This study provides a principled foundation for the next generation of spatial domain detection tools, emphasizing that performance is a function of the interplay between data characteristics (resolution, heterogeneity) and method design (preprocessing, smoothing strength), rather than a simple hierarchy of algorithmic complexity.