scprocess: a pipeline for processing, integrating and visualising atlas-scale single cell data

The paper introduces scprocess, a Snakemake-based pipeline designed to automate, standardize, and ensure the reproducibility of processing, integrating, and visualizing atlas-scale single-cell RNA sequencing data, particularly from 10x Genomics platforms.

The Gist

Imagine you are a librarian who has just received a massive shipment of books. But these aren't normal books; they are millions of tiny, fragile diaries written by individual cells in a human body. Your job is to read them, organize them, and figure out what each cell is doing.

In the past, scientists could only read a few diaries at a time. But now, thanks to new technology, they can read millions of them at once. This is called "atlas-scale" research. The problem? The sheer volume of data is overwhelming. It's like trying to sort a mountain of books by hand while the mountain keeps growing. It's messy, slow, and if you make a mistake in how you sort one pile, the whole library gets messed up.

Enter scprocess. Think of scprocess as a super-smart, automated robot librarian designed specifically to handle this mountain of cellular diaries.

Here is how this robot works, broken down into simple steps:

1. The Delivery (Reading the Raw Data)

When the raw data arrives, it's like a chaotic box of shredded paper. The robot first needs to put the pieces back together.

• The Old Way: Traditionally, this was done by a very slow, heavy-duty machine that ate up a lot of electricity (memory) and took forever.
• The scprocess Way: This robot uses a new, turbo-charged engine called alevin-fry. It's like switching from a slow-moving truck to a high-speed drone. It sorts the shredded papers (reads) and counts them incredibly fast, using much less energy.

2. The Cleanup Crew (Filtering the Noise)

Not every piece of paper in the box is a real diary. Some are just empty wrappers (empty droplets), and some are just smudges of ink from a torn page (ambient RNA).

• The Problem: If you don't clean this up, you might think a smudge is a real cell, or you might miss a real cell hiding in the noise.
• The Solution: The robot has two cleaning modes. One is a heavy-duty vacuum (CellBender) that takes its time to be super precise. The other is a quick, efficient broom (DecontX) for when you need speed. It also has a special trick to spot "doublets"—cases where two cells got stuck together in one wrapper, which would confuse the analysis.

3. The Quality Check (The Bouncer)

Now that we have the papers, the robot acts like a bouncer at a club. It checks the ID of every "cell."

• It looks for signs of damage: Did the cell lose too much ink? Is it full of "mitochondrial" trash (like a cell that is dying)?
• The Smart Twist: The robot knows that sometimes a "damaged" cell is actually just a stressed-out cell, not a bad one. It uses a special metric (looking at how much "spliced" vs. "unspliced" ink there is) to tell the difference between a real nucleus and a cell that accidentally leaked its cytoplasm. It lets the user set the rules so they don't accidentally throw out the VIPs (important cells).

4. The Grouping Party (Integration and Clustering)

Now we have millions of clean diaries. We need to group them. Are these cells all from the brain? Are those from the liver?

• The Challenge: If you try to compare millions of pages at once, your computer will crash.
• The Solution: The robot doesn't read every single word. Instead, it picks out the most interesting sentences (Highly Variable Genes) that tell the story of what makes each cell unique. It ignores the boring, repetitive words that are the same in every cell.
• The Party: It then throws a massive party where similar cells hang out together. It uses a "GPU" (a super-fast graphics card) to make this happen in minutes instead of days, creating a map where cells that look alike stand next to each other.

5. The Translator (Labeling the Cells)

Once the cells are grouped, the robot needs to give them names. "This group is a neuron," "That group is a muscle cell."

• The Old Way: Scientists used to guess based on a few famous keywords.
• The scprocess Way: It uses a statistical detective (pseudobulk analysis). Instead of asking every single cell what it is, it asks the "representative" of the group. This prevents the robot from getting confused by one noisy cell and gives a much more reliable answer. It can even use a pre-trained AI (like CellTypist) that has already read millions of other diaries to help guess the names.

6. The Zoom-In Feature (Subclustering)

Sometimes, the robot finds a group that looks like "Brain Cells," but it knows there's more detail hidden inside.

• The Feature: You can tell the robot, "Zoom in on just the Brain Cells." It then re-runs the whole process on just that small group to find the tiny differences—like finding a specific type of neuron that only wakes up when you are stressed.

Why is this important?

Before scprocess, organizing this data was like trying to build a library by hand, one book at a time, with no instruction manual. Every scientist did it slightly differently, making it impossible to compare their results.

scprocess is the instruction manual and the robot arm combined. It ensures that:

1. Speed: It handles millions of cells without breaking a sweat.
2. Reproducibility: If you run the same data through it twice, you get the exact same result.
3. Transparency: It keeps a detailed log (HTML reports) of every decision it made, so you can see exactly how the library was organized.

In short, scprocess turns a chaotic mountain of cellular data into a beautifully organized, searchable library, allowing scientists to finally understand the complex stories written inside our bodies.

Technical Summary

Based on the provided preprint, here is a detailed technical summary of the scprocess paper.

1. Problem Statement

The field of single-cell RNA sequencing (scRNA-seq) has shifted toward "atlas-scale" research, generating datasets with millions of cells across hundreds of samples. This scale introduces significant challenges:

• Computational Bottlenecks: Traditional tools (e.g., Cell Ranger) are resource-intensive, making alignment and quantification slow and memory-heavy for large cohorts.
• Reproducibility Issues: The analysis pipeline involves dozens of critical decision points and tools. The lack of a unified framework makes it difficult to track parameters, leading to irreproducible results.
• Data Management: Handling large multi-sample datasets often exceeds the memory limits of standard workflows, particularly during feature selection and integration.
• Methodological Flexibility vs. Standardization: While flexible, the current landscape of disparate tools makes it hard to maintain consistent quality control (QC) and batch correction across diverse experimental designs.

2. Methodology

scprocess is a Snakemake-based pipeline designed to automate and standardize the processing of scRNA-seq and single-nucleus RNA-seq (snRNA-seq) data, specifically optimized for 10x Genomics technology. It is built for High-Performance Computing (HPC) environments and utilizes Conda for dependency management.

The pipeline follows a modular workflow with the following key technical components:

• Input & Configuration: Accepts raw FASTQ files, sample metadata, and a YAML configuration file. It supports both single-sample and multi-sample (100+) workflows.
• Alignment & Quantification:
  • Replaces the resource-heavy Cell Ranger with simpleaf (part of the alevin-fry ecosystem).
  • Uses pseudoalignment for speed and memory efficiency.
  • Generates separate spliced and unspliced read counts (crucial for RNA velocity) and includes automatic detection of 10x assay chemistry.
• Cell Calling & Ambient RNA Removal:
  • Offers two pathways: CellBender (GPU-accelerated, high precision, high resource cost) or DecontX (CPU-based, fast).
  • For DecontX, it integrates DropletUtils (barcodeRanks or emptyDrops) for initial cell calling and uses an estimated ambient profile for more accurate noise removal.
• Quality Control (QC):
  • Filters based on library size, feature counts, mitochondrial proportion, and spliced read proportion (a critical metric for snRNA-seq to detect cytoplasmic contamination).
  • Uses user-specified thresholds rather than automated MAD (Median Absolute Deviation) to avoid bias in heterogeneous samples.
  • Employs scDblFinder for doublet detection, validated against benchmarked ground truth.
• Feature Selection (HVGs):
  • Solves the memory bottleneck of loading full count matrices by processing data in gene chunks.
  • Implements a modified Seurat vst algorithm that allows HVG selection per sample or sample group before consolidation, preventing batch effects from driving gene selection.
  • Filters out "ambient genes" (identified via differential expression between empty droplets and cells) to improve signal-to-noise ratio.
• Integration & Clustering:
  • Performs PCA, batch correction (using Harmony), graph construction, Leiden clustering, and UMAP.
  • Offers two backends: standard Scanpy (CPU) and RAPIDS-singlecell (GPU-accelerated) for massive performance gains in clustering and dimensionality reduction.
  • Includes a two-round integration strategy: the first round identifies doublet-enriched clusters for removal before a final clean integration.
• Downstream Analysis:
  • Marker Identification: Uses a pseudobulk approach with edgeR (aggregating counts at the sample level) rather than treating individual cells as replicates, ensuring statistical robustness.
  • Annotation: Integrates CellTypist (logistic regression) and a custom XGBoost model (trained on human brain data) for automated cell type labeling.
  • Multiplexing: Supports HTO (Hashtag Oligo) demultiplexing via Seurat's HTODemux and external tools.
  • Subclustering: Allows iterative analysis on specific cell populations with re-evaluation of ambient contamination and HVGs.

3. Key Contributions

• Unified Workflow: Consolidates disparate tools (simpleaf, CellBender/DecontX, scDblFinder, Harmony, edgeR, CellTypist) into a single, reproducible Snakemake pipeline.
• Scalability: Specifically engineered for atlas-scale datasets (100+ samples) via chunked processing, HPC optimization, and GPU acceleration options (RAPIDS).
• Reproducibility: The YAML configuration file serves as a permanent record of all parameters. The pipeline generates detailed HTML reports for every major step, allowing users to inspect intermediate outputs and refine parameters.
• Statistical Rigor:
  • Adopts a pseudobulk strategy for marker gene identification to correct for sample-level dependencies.
  • Implements a robust strategy for removing ambient RNA-influenced genes during HVG selection.
  • Addresses specific snRNA-seq challenges (e.g., mitochondrial filtering caveats and spliced read metrics).
• Flexibility: Modular design allows users to execute steps independently, rerun specific stages, or swap algorithms (e.g., CPU vs. GPU) without breaking the workflow.

4. Results

• Performance: The pipeline successfully processed a dataset of 149 samples (Pineda et al. 2024), generating comprehensive HTML reports and intermediate outputs.
• Efficiency: By utilizing simpleaf and RAPIDS-singlecell, the pipeline significantly reduces runtime and memory usage compared to standard Cell Ranger-based workflows, making large-scale analysis feasible on standard HPC clusters.
• Accuracy: The integration of doublet detection (scDblFinder) and ambient RNA removal (DecontX/CellBender) ensures high-quality input for downstream clustering. The pseudobulk marker approach reduces false positives common in single-cell differential expression.
• Usability: The pipeline provides a simple CLI interface, lowering the barrier to entry for researchers managing complex, multi-sample projects.

5. Significance

scprocess addresses a critical gap in the single-cell ecosystem: the transition from small-scale experiments to atlas-scale biology.

• It democratizes large-scale analysis by providing a "turnkey" solution that handles the computational heavy lifting (alignment, integration, batch correction) while maintaining transparency.
• It enhances scientific reproducibility by enforcing standardized, documented workflows, reducing the "black box" nature of complex scRNA-seq analyses.
• By supporting both CPU and GPU architectures and offering modular steps, it is adaptable to diverse institutional computing resources.
• The pipeline is open-source (MIT license) and available via GitHub, fostering community adoption and further development.

In summary, scprocess is a robust, scalable, and reproducible framework that enables researchers to efficiently transform raw 10x Genomics sequencing data into high-quality, integrated single-cell atlases suitable for deep biological discovery.