🧬 biology

Beyond Accuracy: Benchmarking the Reproducibility and Robustness of Single-Cell Embeddings

This paper introduces BioBench, a framework that challenges the assumption that reproducibility correlates with model class by demonstrating through the Biological Stability Score that algorithmic solvers, rather than whether a method is linear or deep, are the primary determinants of single-cell embedding stability.

Original authors: Advika Jain

Published 2026-09-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Advika Jain

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

In the microscopic world inside our bodies, every cell carries a unique instruction manual written in genes. For decades, scientists could only read the average of these manuals from a crowd of cells, missing the subtle differences that make one cell a fighter against infection and another a builder of tissue. Today, a technology called single-cell sequencing allows researchers to read the manual of individual cells, revealing a hidden landscape of diversity that defines health and disease. To make sense of millions of these individual readings, scientists use computer programs to compress the complex data into simpler maps, known as embeddings. These maps group similar cells together, helping researchers identify new cell types or track how diseases progress. However, a quiet assumption has long guided this field: that simple, old-fashioned math methods are reliable and steady, while newer, complex artificial intelligence models are prone to wobbling and changing their minds.

A new study challenges this assumption by asking a fundamental question: if you run the same analysis twice on the same data, do you get the exact same map? The researcher, Advika Jain, built a testing framework called BioBench to measure this stability. They took five different computer methods—ranging from classic mathematical techniques to modern deep learning models—and ran them on three large collections of human cell data. Instead of just checking if the maps were accurate, they measured how much the maps shifted when the computer was asked to simply start over with a different random starting point, or when the data was slightly tweaked in a realistic way. The goal was to find out if the reliability of a method could be guessed just by knowing whether it was "old" or "new."

The results revealed a surprising reality. The idea that simple methods are always steady and complex ones are always shaky turned out to be false. The study found that reproducibility exists on a wide spectrum that cuts across the divide between old and new. One of the classic, simple methods, called non-negative matrix factorization, was found to be just as unstable as the most complex deep learning model tested. When the researcher ran these two methods multiple times, the maps they produced shifted significantly, sometimes changing the grouping of cells in ways that could alter a biological conclusion. In fact, the deep learning model was not the worst offender; in some cases, it was more stable than the classic method.

The most revealing part of the study came from a direct comparison of two methods that look almost identical on paper: a standard mathematical technique called PCA and a slightly faster version called Truncated SVD. Both methods perform the same basic task of simplifying the data, and both produced maps of nearly identical quality. Yet, when the researcher ran them again and again, the standard method produced the exact same map every single time, while the faster version changed its output significantly with every new run. The only difference between them was the specific computer algorithm, or "solver," used to do the math. One used a precise, step-by-step calculation, while the other used a randomized shortcut to save time. This proved that the stability of a result depends not on whether the method is simple or complex, but on the specific way the computer is instructed to solve the problem.

The researcher also discovered that stability is not a fixed trait of a method; it changes depending on the data being analyzed. A method that was highly stable on one set of blood cells might be quite unstable on a set of cells from different organs. This means that a scientist cannot simply trust a method because it worked well in a previous study; they must measure the stability for their own specific data. The study introduced a new score, the Biological Stability Score, to quantify this. When applied to the results, this score showed that for some methods, the changes caused by simply restarting the computer were so large that they could hide or fake real biological effects. For instance, on one dataset, removing a portion of the cells from the analysis did not change the map for the deep learning model any more than just restarting the model did, meaning the change was indistinguishable from random noise.

Ultimately, the study argues that the scientific community needs to change how it evaluates these tools. Accuracy alone is not enough; a method must also be proven to be reproducible. The author suggests that every future benchmark should report a "noise floor," a measurement of how much a method's results wiggle when run multiple times, alongside its accuracy scores. This would allow researchers to distinguish between a real biological discovery and a fluke caused by the computer's random starting point. By releasing their testing framework as an open tool, the researcher hopes to make this standard practice, ensuring that the maps guiding our understanding of human biology are not just accurate, but also solid and dependable.

Technical Summary: Beyond Accuracy: Benchmarking the Reproducibility and Robustness of Single-Cell Embeddings

Problem Statement

Single-cell RNA sequencing (scRNA-seq) analysis relies heavily on low-dimensional representations (embeddings) for tasks such as clustering, cell-type annotation, and batch integration. Currently, these models are evaluated almost exclusively on predictive accuracy. A critical gap exists in the quantification of reproducibility: the stability of a model's learned representation when incidental details change, such as the random seed or defensible preprocessing perturbations.

The field operates under an implicit assumption that reproducibility can be inferred from a method's family: that classical, linear methods are inherently reproducible, while deep, stochastic methods are not. This study challenges that assumption, arguing that reproducibility is a property of the specific algorithm and its solver implementation rather than the model class (linear vs. deep). Without measuring the "re-run noise floor," benchmarks risk reporting variance as signal, leading to unreliable biological conclusions.

Methodology

The author introduces BioBench, an open-source, provenance-tracked framework designed to measure embedding stability under a standardized pipeline. The study focuses on three axes of reproducibility, with results reported for robustness (Q2) and re-run consistency (Q3).

1. Experimental Design

Datasets: Three publicly available human datasets were used: PBMC 10k, PBMC 68k (subsampled), and Tabula Sapiens (subsampled). All data underwent a single canonical preprocessing pipeline (Scanpy) involving quality control, library-size normalization, log1p transformation, and selection of the top 2,000 highly variable genes (HVGs).
Methods Evaluated: Five CPU-feasible methods were selected to decorrelate model class from reproducibility:
- PCA: Deterministic linear factorization (Exact SVD solver).
- ICA: Stochastic linear method (FastICA with random initialization).
- TruncatedSVD: Stochastic linear method (Randomized solver).
- NMF: Stochastic linear method (Non-negative matrix factorization).
- scVI: Stochastic deep generative model (Variational Autoencoder).
Perturbations: To test robustness, three types of perturbations were applied to raw counts before the canonical pipeline:
- Gene Dropout: Sets a random fraction of non-zero entries to zero (preserves count structure).
- Cell Subsampling: Draws cells without replacement, stratified by cell type (preserves count structure).
- Additive Gaussian Noise: Adds noise to entries (does not preserve count structure; used as a worst-case stressor).
Re-run Protocol: Every method and condition was executed under three independent random seeds (0, 1, 2) to establish a per-method, per-dataset noise floor.

2. Metrics

Biological Stability Score (BSS): A novel metric quantifying representational stability on a model-agnostic basis. It is a convex combination of:
- Neighborhood Preservation (50%): Mean Jaccard overlap of $k=15$ nearest neighbors between baseline and variant embeddings.
- Cluster Preservation (30%): Adjusted Rand Index (ARI) between Leiden partitions.
- Label Stability (20%): Fraction of cells retaining assigned cell-type labels (only for annotated datasets).
- Score Range: [0, 1], where 1 indicates perfect preservation.
Decision Rule: A perturbation is considered to produce a "real effect" only if the resulting BSS is strictly lower than the method's re-run noise floor ( $BSS_{perturbed} < F_{noise}$ ). If the BSS is above the floor, the change is indistinguishable from seed noise.
Correctness Metrics: On the annotated Tabula Sapiens dataset, standard metrics (ARI, NMI, cell-type accuracy, macro-F1, kBET, iLISI, cLISI) were used to ensure stability was not being measured at the expense of accuracy.

Key Results

1. Reproducibility Forms a Continuum, Not a Binary Split

Contrary to the assumption that linear methods are reproducible and deep methods are not, reproducibility spans a wide continuum that is not predicted by model class:

Perfectly Reproducible: PCA (BSS = 1.000 across all datasets).
Near-Deterministic: ICA (BSS 0.874–1.000).
Strongly Dataset-Dependent: TruncatedSVD (BSS 0.380–0.855).
Substantially Irreproducible: NMF (BSS 0.325–0.550) and scVI (BSS 0.258–0.526).
Key Finding: NMF (a classical linear method) is as irreproducible as scVI (a deep generative model).

2. The Solver is the Determinant of Reproducibility

A matched-pair experiment compared PCA (Exact SVD solver) and TruncatedSVD (Randomized solver).

Quality: Both methods produced near-identical representations in terms of downstream accuracy (Tabula Sapiens ARI: 0.587 vs. 0.585; identical cell-type accuracy).
Reproducibility: They differed drastically in re-run stability. On PBMC 10k, PCA achieved 1.000 while TruncatedSVD achieved 0.525 (a gap of 0.475 BSS).
Conclusion: The gap is attributable solely to the solver (deterministic vs. randomized), not the model family or the representational content.

3. The Noise Floor is Dataset-Dependent

The re-run noise floor varies significantly across datasets for the same method. For example, TruncatedSVD's BSS ranged from 0.380 on PBMC 68k to 0.855 on Tabula Sapiens. This demonstrates that noise floors cannot be assumed or transferred between datasets; they must be measured per experiment.

4. Robustness vs. Reproducibility are Uncorrelated

A method's robustness to data perturbations does not predict its reproducibility across re-runs.

ICA was highly reproducible but showed variable robustness depending on the dataset.
scVI was less reproducible than ICA but more robust than NMF on certain datasets.
Accuracy vs. Reproducibility: The ranking of methods by accuracy (e.g., PCA ≈ SVD > ICA > scVI > NMF) did not match the ranking by reproducibility.

5. Application of the Noise Floor

Applying the decision rule revealed that some apparent "perturbation effects" were actually indistinguishable from seed noise. For instance, on Tabula Sapiens, subsampling 30% of cells for scVI did not move the representation further than simply re-training the model (BSS remained above the noise floor). Conversely, the same perturbation on NMF produced a real effect (BSS fell below the floor).

Significance and Claims

The paper argues that reproducibility is a first-class evaluation axis that must be reported alongside accuracy in single-cell benchmarks.

Refutation of the "Family" Heuristic: The study provides direct evidence that reproducibility cannot be inferred from whether a method is linear or deep. A practitioner choosing a method based on the assumption that "linear = reproducible" may inadvertently select a highly unstable method (e.g., NMF) or miss a stable one (e.g., ICA).
Solver Selection Matters: The choice of solver (exact vs. randomized) is a critical design decision that impacts reproducibility more than the choice of model architecture. The trade-off between computational speed (randomized solvers) and exact reproducibility is explicit and measurable.
Actionable Framework: BioBench provides the infrastructure to measure the "re-run noise floor." The author claims that without this measurement, benchmarks cannot distinguish between real biological effects and artifacts of random initialization.
Modest Scope: The author acknowledges limitations, including the use of only three seeds (limiting precision of confidence intervals), the exclusion of GPU-based foundation models (scGPT, Geneformer) due to computational constraints, and the use of Gaussian noise as a non-count-native stressor. The conclusions are strictly bounded to the CPU-feasible methods tested (PCA, ICA, SVD, NMF, scVI).

In summary, the paper concludes that reproducibility is a property of the solver and its use of randomization, not the model class, and that every single-cell benchmark should report a per-dataset re-run noise floor to ensure the reliability of biological interpretations.