💻 bioinformatics

Building computational benchmarks: an Omnibenchmark reimplementation of a single-cell preprocessing pipeline evaluation

This paper presents Omnibenchmark, a reproducible and standardized framework that unifies the evaluation of single-cell RNA sequencing preprocessing pipelines to address inconsistencies in existing benchmarks and facilitate more flexible, extensible method comparisons.

Original authors: Choudhury, A., Kitak, T., Carrillo, B., Busch, P., Emons, M., Gunz, S., Koderman, M., Luo, S., Mallona, I., Meara, A., Wissel, D., Robinson, M. D.

Published 2026-10-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Choudhury, A., Kitak, T., Carrillo, B., Busch, P., Emons, M., Gunz, S., Koderman, M., Luo, S., Mallona, I., Meara, A., Wissel, D., Robinson, M. D.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

In the last decade, scientists have developed a way to read the genetic instructions inside individual cells, rather than just averaging the signals from a million cells mixed together. This shift has transformed our understanding of biology, revealing that tissues are not uniform blocks of tissue but complex mosaics of distinct cell types, each with its own role. To make sense of the massive amounts of data generated by these experiments, researchers rely on computer programs that clean up the raw numbers, organize them, and group similar cells together. These programs, often called pipelines, are essential tools. However, because there are now hundreds of different software packages available, each making slightly different choices about how to process the data, it has become difficult for scientists to know which tool is the best for a specific job. One program might group cells differently than another, not because the biology changed, but because the software handled the math differently. Without a clear way to compare them, researchers risk drawing conclusions based on the quirks of a specific program rather than the reality of the cells they are studying.

A team of researchers at the University of Zürich and the Swiss Institute of Bioinformatics decided to tackle this confusion by building a standardized testing ground. They took an existing study that compared five popular software pipelines for analyzing single-cell data and rebuilt it from the ground up to be more rigorous, transparent, and reproducible. Their goal was not just to see which program was the fastest or most accurate, but to create a framework that could be easily updated and expanded by the wider scientific community. They focused on five major pipelines: Seurat, Scrapper, OSCA, Scanpy, and Rapids. These tools all perform the same general tasks: they filter out low-quality data, select the most informative genes, reduce the complexity of the data into a simpler form, and finally group the cells into clusters. The researchers ran these pipelines on three different datasets containing thousands of cells, testing them across a wide range of settings to see how small changes in the software's configuration affected the final results.

The team discovered that the speed and accuracy of these pipelines varied significantly depending on the specific choices made by the user. For instance, the time it took to run the analysis was heavily influenced by the number of features selected and the complexity of the graph used to connect similar cells. Some pipelines, particularly those designed to run on specialized graphics processors, were much faster than others, but this speed did not always guarantee better accuracy. In fact, the researchers found that no single pipeline was the best performer across all datasets. A tool that excelled at grouping cells in one experiment might struggle with a different dataset. This inconsistency highlighted that the "best" tool is not a fixed answer but depends on the specific data and the parameters chosen. One of the most important findings was that the way different software packages handled a key setting called the "resolution" parameter was not consistent. Because the tools interpreted this setting differently, it was difficult to compare them fairly. To fix this, the researchers adjusted their benchmark to focus on the actual number of cell groups produced, rather than the abstract setting used to create them, ensuring a more direct comparison.

Beyond simply ranking the tools, the study revealed that the process of building benchmarks itself needs to change. The researchers showed that by standardizing how data is fed into these programs and how results are collected, it becomes possible to run hundreds of variations of an experiment automatically. They found that even small differences in the software version or the underlying computer hardware could alter the results, sometimes leading to significant errors. For example, when they tested a specific update to one of the software packages, they noticed a sudden drop in performance that traced back to a subtle change in how the program solved a mathematical problem. This suggests that software tools need to be tested continuously, much like a car model is tested under different conditions, rather than just once when it is released. The researchers also noted that while some tools claimed to use multiple computer cores to speed up processing, they did not always achieve the expected gains, indicating that efficiency is not guaranteed just because a feature exists.

The ultimate value of this work lies in the framework the team created, which they call Omnibenchmark. This system allows other scientists to plug in new datasets, new software, or new ways of measuring success without having to rebuild the entire experiment from scratch. By making the code, the data, and the software environments openly available and easy to modify, the researchers have provided a template for how computational biology can move forward. Instead of isolated studies that are hard to reproduce, the field can now build a shared, living library of benchmarks that evolves as the technology improves. The study concludes that while there is no single perfect pipeline for every situation, having a standardized, transparent way to evaluate them allows the community to make better-informed choices. It turns the chaotic landscape of available tools into a navigable map, where the strengths and weaknesses of each method are clearly visible, helping scientists focus on the biology rather than the software.

Technical Summary: Building Computational Benchmarks: An Omnibenchmark Reimplementation of a Single-Cell Preprocessing Pipeline Evaluation

Problem Statement
The rapid proliferation of single-cell RNA sequencing (scRNA-seq) techniques and datasets has been accompanied by an explosion of computational analysis tools and frameworks. While numerous "neutral" benchmarks exist to evaluate specific computational tasks, they often suffer from a lack of standardization. Critical issues include the use of incompatible datasets, metrics, or parameter sets across different studies, making cross-study comparisons difficult. Furthermore, existing benchmarks often lack reproducibility due to missing artifacts, incomplete provenance, unavailable simulation code, and fragile software environments. The field lacks a unified, extensible framework that allows for the systematic investigation of entire analysis pipelines, including the isolation of specific steps and the exploration of broad parameter spaces.

Methodology
The authors present scrna-bench, a reimplementation and enhancement of a previous "end-to-end" pipeline benchmark [21], built using the Omnibenchmark framework [17]. The goal was to create a reproducible, standardized, and extensible template for evaluating scRNA-seq pipelines.

Framework Structure: The benchmark is organized into four distinct stages: datasets, methods, metrics, and a report (metric collector). The workflow is orchestrated via a self-describing YAML benchmark plan that defines inputs, outputs, software environments, and code repositories for each stage.
Datasets and Pipelines: The study focuses on three datasets (sc-mix, cb, and be1) and five major analysis pipelines: Seurat, Scrapper, OSCA, Scanpy, and Rapids-singlecell.
Standardization and Harmonization: The authors injected code snippets to harmonize inputs and outputs across all stages. They exposed commonly tuned parameters to enable a broader grid search across the parameter space, including:
- Number of PCA components ( $n\_comp$ )
- Number of nearest neighbors ( $n\_neig$ )
- Number of highly variable genes ( $n\_hvg$ )
- Cluster count offset ( $\delta_{cluster}$ ), defined as the signed offset from the true number of clusters.
- Filtering strategies (manual vs. automated).
Reproducibility: To ensure reproducibility and address version-specific instabilities, the authors created three Apptainer (container) images. Two Python-based images were built to test different versions of the rapids-singlecell package (v0.13.5 and v0.14.1), as the original study encountered significant performance drops with the older version due to PCA algorithm instabilities.
Metrics: Performance was evaluated using Adjusted Rand Index (ARI) as the primary metric for clustering accuracy (using ground-truth cell annotations), alongside other label agreement metrics (NMI, Fowlkes-Mallows, etc.) and internal cluster structure metrics (Silhouette, Davies-Bouldin). Computational times for individual pipeline steps (filtering, PCA, UMAP, Leiden clustering) were also recorded.
Statistical Analysis: Linear models (both fixed and random effects) were fitted to determine the influence of pipeline choice and parameter settings on ARI performance.

Key Results

Computational Performance: Runtime varied significantly across pipelines and parameters. Rapids (GPU-accelerated) and Scrapper (C++ with R bindings) were the fastest pipelines overall. OSCA exhibited slower PCA implementation, while Scanpy showed slower UMAP implementation. The study noted that runtime comparisons between CPU and GPU pipelines reflect architectural differences and should not be interpreted as purely algorithmic efficiency measures.
Clustering Accuracy:
- The number of specified clusters was the strongest driver of clustering performance (ARI).
- There was no single "top performer" across all datasets; stability varied greatly depending on parameterization, particularly for the cb dataset.
- A notable finding was that the resolution parameter is implemented differently across pipelines. Consequently, the authors shifted the parameter of interest to the number of clusters rather than the resolution parameter to ensure fair comparison.
- Rapids and Scrapper generally maintained high accuracy, though a reversal in top performance was observed between the be1 and cb datasets. Scanpy, Rapids, and Seurat showed suboptimal performance on the be1 dataset.
Parameter Sensitivity: Expanding the parameter grid revealed that while some configurations improved ARI over the original study's defaults, others performed worse, especially on the cb dataset. Linear modeling indicated that dataset-specific effects were dominant, with the pipeline choice and the number of clusters being the primary drivers of variance in ARI.
Reproducibility Challenges: The reimplementation uncovered a significant instability in rapids-singlecell v0.13.5, where ARI scores dropped across all parameterizations due to underlying PCA algorithm issues. This highlighted the necessity of testing multiple software versions, treating benchmarks as continuous tests for evolving packages.

Significance and Claims
The paper positions scrna-bench not merely as a set of results, but as a template and canvas for future benchmarking efforts. Its primary significance lies in:

Standardization: It unifies the process of benchmark construction, enabling consistent method comparisons and facilitating the extension of benchmarks to broader parameter sweeps, different software versions, and additional pipelines.
Reproducibility and Extensibility: By archiving all artifacts on Zenodo and using containerized environments, the entire benchmark is reproducible, forkable, and modifiable. This addresses the recurring barriers of missing artifacts and fragile environments found in previous studies.
Community Collaboration: The authors view the benchmark as a "draft template" intended to be iteratively improved by the community. They propose that benchmarks should become "continuous entities" that evolve alongside methods and datasets.
Generalizability: While focused on scRNA-seq, the authors argue that this strategy of standardized, extensible benchmarking templates can be applied to other areas of computational biology and computational subfields to enable open-source modules that can be reused and continuously updated.

The authors remain modest about the scope, acknowledging that they currently evaluated only three datasets and five pipelines, and that CPU-based methods were tested on single cores. They identify the expansion of software environments across package versions and the inclusion of integration steps (for multi-sample processing) as necessary future extensions.

Technical Summary: Building Computational Benchmarks: An Omnibenchmark Reimplementation of a Single-Cell Preprocessing Pipeline Evaluation

More like this