📄 other

Rank or Value? An Empirical Analysis of Single-Cell Transformer Tokenization under Sequencing-Depth Loss

This empirical study demonstrates that in single-cell RNA sequencing, fixed expression value binning outperforms corpus-median rank encoding for cell classification under sequencing-depth loss, revealing that invariance to library-size multiplication does not guarantee robustness to stochastic molecule loss.

Original authors: Liu Chen

Published 2026-07-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Liu Chen

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to understand a crowd by listening to a thousand people shout at once. In the world of biology, scientists use a technique called single-cell RNA sequencing to do just that, but instead of voices, they are listening to the "shouts" of genes inside individual cells. Each cell is a tiny factory, and the genes are the workers. Sometimes a worker shouts loudly (high expression), sometimes quietly (low expression), and sometimes not at all (zero expression). The goal is to figure out what kind of factory each cell is—like a muscle factory or a blood factory—just by listening to this chaotic noise.

To make sense of this noise, computers need to turn the raw numbers into a language they can understand, a process called "tokenization." It's like translating a messy grocery list into a neat, numbered order. For a long time, scientists have debated the best way to do this. One popular idea is to rank the genes: "Who is the loudest? Who is second loudest?" This method is often praised because it seems immune to how many people are in the room; if everyone shouts half as loud, the ranking stays the same. Another idea is to just write down the volume of the shout, perhaps rounding it to simple categories like "quiet," "medium," or "loud." The big question is: if the recording gets fuzzy or we lose some of the sound (like when a microphone is too far away), which translation method keeps the story intact?

This paper, titled "Rank or Value? An Empirical Analysis of Single-Cell Transformer Tokenization under Sequencing-Depth Loss," dives into that exact question. The researchers, led by Liu Chen, set up a controlled experiment to see which method survives when the data gets "thinner." They took a real dataset of 2,638 human blood cells and simulated a scenario where the sequencing machine missed a lot of molecules, effectively thinning the data down to as little as 5% of its original strength. They tested two main approaches: one that ranks genes based on a massive reference library (called "corpus-median rank," similar to how the model Geneformer works) and one that simply bins the expression values into fixed categories (similar to how scBERT works).

The results were surprising and flipped the common intuition on its head. The researchers found that the "ranking" method, which was supposed to be the robust, unshakeable champion, actually fell apart much faster than the "value" method when the data got thin. When they reduced the data to just 25% of its original depth, the value-based method (fixed bins) kept a high accuracy score of 0.857, while the ranking method dropped significantly to 0.776. In fact, the value-based approach held onto about 6 points more of its original performance than the ranking method did.

Why did the ranking method fail? The paper explains that the specific way this ranking was done—adjusting for a gene's typical behavior across a huge library—accidentally promoted genes that were barely whispering. In the thinning simulation, these "whispering" genes were the first to go silent completely, causing the ranking order to shuffle wildly. It's like trying to organize a race where the runners are ranked by how fast they are relative to their usual speed; if a slow runner suddenly stops running because they tripped, the whole order of the race gets messed up. In contrast, the value-based method, which just looked at how loud the shout was right then and there, was more stable. Even when the volume dropped, the relative differences between "loud" and "medium" stayed clear enough for the computer to still recognize the cell type.

The study concludes that while ranking genes might sound like a clever way to ignore technical noise, it isn't actually robust to the random loss of molecules that happens in real experiments. The authors emphasize that this doesn't mean the famous "Geneformer" model is broken, but rather that its specific way of turning data into tokens might be fragile when the data quality drops. They suggest that future models should be tested not just on how well they learn, but on whether their "language" stays stable when the experiment gets a little shallower. In the end, sometimes it's better to just listen to the volume of the shout than to try to guess the rank of the voice in a noisy room.

Technical Summary: "Rank or Value? An Empirical Analysis of Single-Cell Transformer Tokenization under Sequencing-Depth Loss"

Problem Statement
Single-cell RNA sequencing (scRNA-seq) data is characterized by sparsity and strong sensitivity to sequencing depth (library size). Transformer-based foundation models for single-cell data, such as Geneformer and scBERT, require converting sparse, unordered count vectors into token sequences. These models employ fundamentally different tokenization strategies:

Rank-based (e.g., Geneformer): Genes are ordered by expression relative to a corpus-level median. This encoding is theoretically invariant to uniform scaling (multiplying all counts in a cell by a constant).
Value-based (e.g., scBERT): Genes are assigned discrete tokens based on fixed, discretized expression values.

A common intuition suggests that rank encodings are robust to sequencing-depth variations. However, the paper argues that real-world depth loss is not a uniform rescaling but a stochastic removal of molecules (thinning), which creates new zeros and can reorder genes. The central empirical question is whether corpus-median rank tokenization preserves cell-type information better than fixed expression-value bins when transcriptomes are subjected to stochastic UMI loss.

Methodology
The study isolates tokenization from model capacity by using a controlled, linear probe framework rather than comparing pretrained transformers.

Dataset: 2,638 peripheral blood monuclear cells (PBMCs) from the 10x Genomics 3k dataset, aligned to 18,974 genes shared with the Geneformer-104M vocabulary. Cells were annotated into 8 classes (e.g., CD4 T cells, monocytes) using Scanpy tutorial labels.
Encodings Tested:
1. Continuous: Log-normalized expression.
2. Value-bin: Discretized log-expression clipped at 5 (mimicking scBERT).
3. Corpus-rank: Genes ranked by $X_{ig}/m_g$ (where $m_g$ is the corpus median), mapped to sparse percentile scores (mimicking Geneformer).
4. Plain-rank: Genes ranked by raw counts within the cell (unscaled rank control).
5. Binary: Detection-only ( $X_{ig} > 0$ ).
Simulation of Depth Loss: Observed counts were thinned using a binomial process ( $\tilde{X} \sim \text{Binomial}(X, p)$ ) to simulate retention rates of 75%, 50%, 25%, 10%, and 5%. Each depth level was tested with 30 paired stochastic realizations.
Evaluation: Class-balanced linear support-vector classifiers were trained on full-depth data and evaluated on thinned held-out folds. The primary metric was Macro-F1. Performance degradation was measured as the differential retention of performance relative to each encoding's full-depth baseline.

Key Results

Value Bins Outperform Ranks under Thinning: At 25% UMI retention, the value-bin encoding achieved a Macro-F1 of 0.857, significantly outperforming the corpus-median rank encoding (0.776). Relative to their respective full-depth baselines, value bins retained 5.9 Macro-F1 points more than corpus-median ranks (95% bootstrap interval: 5.6–6.3; Holm-adjusted $p < 10^{-8}$ ).
The Role of Corpus Scaling: The "Plain-rank" (unscaled) control improved performance over corpus-median ranks (0.799 vs. 0.776 at 25% retention) but still lagged behind value bins. This indicates that while corpus normalization contributes to the performance gap, the loss of quantitative magnitude (retaining only order) is also detrimental under stochastic molecule loss.
Mechanism of Failure: Token-level analysis revealed that corpus-median scaling disproportionately promoted genes with very low counts. Half of the top 128 tokens in the corpus-rank encoding were supported by a single UMI, compared to less than 1% for plain ranks. Consequently, corpus-ranked tokens were intrinsically more fragile to thinning; the mean analytic probability of a top-128 token remaining detected after 25% thinning was 45.8% for corpus ranks versus 82.0% for plain ranks.
Impact on Rare Classes: The performance gap was most pronounced for rare cell types (e.g., dendritic cells, megakaryocytes). At 10% retention, rank-based encodings failed to distinguish rare classes (F1 near zero), whereas value bins and continuous expressions maintained higher F1 scores.
Stability of Value Bins: Continuous log-expression and clipped value bins tracked each other closely, suggesting that coarse magnitude information is sufficient for linear annotation and more stable than rank order under depth loss.

Significance and Claims
The paper does not claim to rank pretrained foundation models (Geneformer vs. scBERT) but rather evaluates the stability of their underlying tokenization principles under a specific covariate shift (molecule loss).

Refutation of Robustness Intuition: The study demonstrates that invariance to library-size multiplication does not imply robustness to stochastic molecule loss. The "robustness" of rank encoding is conditional on uniform scaling, a condition not met by typical sequencing depth reduction.
Corpus Normalization Risks: The authors show that corpus-level scaling, intended to highlight genes unusually expressed relative to their baseline, can inadvertently elevate low-count, fragile genes to prominent token positions, destabilizing the representation when data is thinned.
Evaluation Recommendations: The authors argue that single-cell foundation model benchmarks should include stress tests for sequencing-depth shifts. They propose that evaluation protocols must report performance under mismatched depths, token stability under thinning, and the specific impact of corpus-level scaling, rather than relying solely on accuracy at matched depths.

The results are presented as a conditional finding on a specific dataset (PBMC 3k) and reference partition, intended to identify a concrete failure mode for future large-scale testing rather than to provide a definitive population-level inference.

Technical Summary: "Rank or Value? An Empirical Analysis of Single-Cell Transformer Tokenization under Sequencing-Depth Loss"

More like this