Class-Support Mismatch Dominates Protocol-Driven Optimism in Spatial Transcriptomics, with a Larger Residual Penalty for Graph Neural Networks

Technical Summary: Class-Support Mismatch Dominates Protocol-Driven Optimism in Spatial Transcriptomics

1. Problem Statement

Spatial transcriptomics (ST) benchmarks frequently report inflated performance metrics when using randomly interleaved cross-validation (CV) compared to spatial-block CV. The prevailing attribution for this "random-versus-block" performance gap is spatial leakage: the claim that message passing in Graph Neural Networks (GNNs) and spatial autocorrelation allow test information to contaminate training.

However, this attribution has never been rigorously tested because switching from random to block CV simultaneously alters five distinct design axes:

1. Training-set size: Block protocols discard buffer zones, significantly reducing the number of training spots.
2. Class support: Contiguous blocks often lack the full diversity of tissue domains present in the whole section, leading to mismatches between training and test class sets.
3. Cross-role message passing: Random partitions create many train-test neighbor pairs; block partitions cut the graph at boundaries.
4. Preprocessing scope: Global preprocessing (e.g., gene selection, scaling) vs. fold-local preprocessing.
5. Spatial extrapolation: The actual scientific challenge of predicting unseen spatial regions.

A single performance gap statistic conflates these factors, making it impossible to isolate "leakage" from other artifacts like class-support mismatch.

2. Methodology

The author conducted a deeply audited study using 10x Visium human breast cancer data (3,798 spots, 11 Leiden domains) and replicated findings on 12 expert-annotated DLPFC sections (47,280 spots).

Graph Pruning and Verification:
To eliminate message-passing leakage, the author constructed a "zero-contamination" graph. For each fold, they pruned the spatial graph to ensure:

• Zero cross-role edges: No edges connect train, validation, or test nodes.
• Infinite hop distance: The minimum hop distance between train and test nodes is infinite.
• Verification: Two independent implementations verified these properties, confirming that buffer zones alone are insufficient as they remove blocks but not the edges connecting them.

The Protocol Ladder:
The study employed an additive decomposition using an 11-arm protocol ladder. By changing one design axis at a time (e.g., matching class support, size, graph masking, or preprocessing scope), the author isolated the contribution of each factor to the total performance gap.

• Base Ladder: Decomposed the total gap (Random vs. Block) into: Training-set size, Class support, Spatial (residual), and Preprocessing scope.
• Masked Arms: Used to de-confound the spatial penalty for GNNs by matching graph masking rules and preprocessing scopes between random and block protocols.

Persistent Homology Audit:
The author re-evaluated the utility of persistent homology (topology) features. They replaced the previous "shared-graph" feature computation (which leaked training information into test features) with an inductive rule where features are computed strictly within the induced subgraph of each fold's role.

Statistical Framework:

• Units: Analysis was based on $n=4$ independent spatial blocks (Folds 1 and 3 were identical partitions).
• Intervals: Hierarchical bootstrap 95% confidence intervals.
• Tests: Exact two-sided sign-permutation tests (with a floor of $p=0.125$ due to $n=4$).

3. Key Results

A. Decomposition of the Random-to-Block Gap
The total closed-set macro-F1 gap of +0.447 decomposes additively as follows:

• Class-support mismatch: +0.276 (The largest component, ~62% of the gap). This arises because contiguous blocks do not contain all tissue domains, forcing models to predict classes unseen in training.
• Residual spatial extrapolation: +0.098 (~22%).
• Preprocessing scope: +0.051 (~11%).
• Training-set size: +0.022 (~5%).

B. The GNN Penalty
After matching size, class support, graph masking, and preprocessing scope:

• Graph Neural Networks (GCN, GAT): Exhibit a substantial residual spatial penalty of +0.214 (CI: +0.160, +0.273).
• Non-graph Classifiers (SVM, XGBoost, kNN, MLP): Exhibit a small, unresolved penalty of +0.042 (CI: -0.039, +0.143), which includes zero.
• Family Gap: The difference between GNN and non-graph penalties is +0.172, statistically resolvable only for GNNs.

C. Message Passing and Leakage

• Cross-role message passing contributes +0.050 (GCN) and +0.016 (GAT) to the gap under block protocols.
• For non-graph methods, this value is exactly 0.000.
• The author concludes that message-passing leakage is a real but minor channel, not the primary driver of the observed optimism.

D. Persistent Homology
The previously reported topological benefit of +0.0154 macro-F1 was entirely an artifact of leakage.

• When features are recomputed inductively within each fold, the benefit disappears.
• The clean estimate is -0.0144 (SD 0.0269, $p=0.50$), indicating no difference between TopoSpatial and a matched GAT without topology.

E. Method Ranking Reversal (DLPFC)
On 12 DLPFC sections, the ranking of methods reversed based on the protocol:

• Random CV: XGBoost-PCA+XY (using raw coordinates) significantly outperformed SVM-PCA.
• Block/Leave-One-Out CV: SVM-PCA outperformed XGBoost-PCA+XY.
• This demonstrates that benchmarks relying solely on random CV may consistently select inferior methods that exploit spatial autocorrelation (via coordinates) rather than learning generalizable biological features.

4. Significance and Claims

The paper claims to correct the field's understanding of "spatial leakage" in ST benchmarks. Its primary contributions are:

1. Refutation of the "Leakage" Monolith: The random-versus-block gap is not a single quantity driven by message-passing leakage. The dominant component is class-support mismatch, an artifact of how spatial blocking reduces class diversity in training/test sets.
2. GNN-Specific Penalty: The genuine spatial-extrapolation penalty (the difficulty of predicting unseen spatial regions) is substantially larger for GNNs than for non-graph classifiers. This suggests GNNs are uniquely sensitive to the distribution of local graph context in unseen regions, not just to label leakage.
3. Methodological Correction:
   • Buffer zones are insufficient: They remove blocks but not edges. Explicit graph pruning and verification are required to eliminate message-passing leakage.
   • Feature Leakage: Persistent homology features computed on the full graph constitute a leakage channel; they must be recomputed inductively per fold.
   • Reporting Standards: Benchmarks must report the decomposition of the gap (size, class support, preprocessing, spatial) rather than a single inflation figure. Class support must be matched before attributing performance drops to spatial extrapolation.
4. Negative Result on Topology: The study provides a clean negative result, showing that the reported benefits of persistent homology in ST were entirely due to data leakage, not model capability.

The author explicitly states they do not claim to have created a leak-free benchmark without qualification (as the target labels are transductive Leiden clusters) but assert that their protocols are leak-free regarding message passing, node features, and preprocessing. They emphasize that the "spatial penalty" is a real phenomenon for GNNs, distinct from the artifacts of protocol design.