PathwayBench: a multi-criterion benchmark of pseudobulk pathway activity scoring methods reveals rank-window competition as a mechanism of biological signal loss in single-cell RNA-seq

Technical Summary: PathwayBench

Problem Statement
Pathway activity scoring is a critical step in single-cell RNA-seq (scRNA-seq) analysis, transforming high-dimensional expression data into interpretable scores for comparing conditions and defining cell states. Despite the dominance of the pseudobulk case-control regime in applied disease studies, method selection for pathway scoring is rarely guided by systematic, multi-criterion evidence. Prior benchmarks have focused on cell-level perturbation ground truth or clustering robustness, leaving the pseudobulk setting largely unaddressed. Furthermore, when magnitude-aware methods (e.g., ssGSEA, GSVA, z-score) and rank-based methods (e.g., AUCell, UCell) yield divergent results, the field lacks a mechanistic explanation for this disagreement.

Methodology
The authors present PathwayBench, a comprehensive benchmark evaluating five widely used pathway scoring methods: ssGSEA, GSVA, z-score, AUCell, and UCell.

• Datasets: The evaluation utilizes eight pseudobulked scRNA-seq case-control datasets spanning five tissues (brain, blood, kidney, lung, heart) and 682 donors. These cover diverse conditions including Alzheimer's, Parkinson's, COVID-19, idiopathic pulmonary fibrosis, dilated cardiomyopathy, systemic lupus erythematosus, and chronic kidney disease (CKD).
• Evaluation Criteria: Methods are assessed against five distinct criteria:
  1. Biological Relevance: Direction accuracy (against literature ground truth), AUROC, and effect size (Cohen's d).
  2. Robustness: Aggregation stability, outlier sensitivity, normalization stability, and sample-size stability.
• Performance Classification: Per-criterion performance is discretized into "Good," "Intermediate," or "Poor" based on fixed thresholds to facilitate comparison.
• Mechanistic Investigation: To explain discrepancies between method families, the authors conducted controlled simulations with varying "competitor-gene burdens" (unrelated genes upregulated in the same condition) to test the limits of rank-based scoring windows.
• Real-World Validation: The mechanism was tested in a clinically relevant setting involving extracellular matrix (ECM) remodeling in CKD.

Key Results

• Variable Biological Relevance: All methods detect literature-supported directions above chance, but with a sensitivity gradient. The z-score method achieved the highest direction accuracy (0.773), followed by ssGSEA (0.733), GSVA (0.705), UCell (0.662), and AUCell (0.605). Discrimination (AUROC) was tightly clustered across all methods.
• Rank-Window Competition: The study identifies a mechanism termed "rank-window competition." In rank-based methods (AUCell, UCell), scores are derived from the position of pathway genes within a fixed top-rank window. When non-pathway "competitor" genes saturate this window (common in broadly remodeled transcriptomes like fibrosis or inflammation), the score collapses or, in extreme cases, inverts its sign.
  • Simulation: Under high competitor burden (500 unrelated genes), rank-based methods exhibited sign inversion in 40% of replicates, while magnitude-aware methods (z-score, GSVA) sustained the signal.
  • Real Data: In CKD, where ECM remodeling is a hallmark, rank-based methods produced near-zero or wrong-direction effects, whereas magnitude-aware methods correctly identified upregulation.
• No Single Dominant Method: No method satisfied all five criteria.
  • GSVA achieved "Good" ratings on four criteria (biology, outlier, normalization, sample-size stability) but "Poor" on aggregation stability.
  • UCell also achieved "Good" on four criteria, including the highest aggregation stability, but was "Intermediate" on sample-size stability.
  • Z-score offered high sensitivity but intermediate robustness scores.
  • AUCell and ssGSEA met fewer criteria, with AUCell showing the lowest sensitivity and normalization stability.
• Trade-offs: A clear trade-off exists between sensitivity (direction accuracy) and robustness. Z-score occupies the high-sensitivity/median-robustness quadrant, while UCell occupies the high-robustness/moderate-sensitivity quadrant.

Significance and Contributions
The paper claims three principal contributions:

1. Mechanistic Explanation: It provides a coherent explanation for the systematic divergence between magnitude-aware and rank-based methods, attributing it to rank-window competition. This explains why rank-based methods may fail in fibrotic, inflammatory, or broadly remodeled transcriptomes.
2. Reframed Method Selection: By demonstrating that no single method is universally superior, the work reframes the "best method" debate as a question of analysis priorities. It suggests that method selection should depend on whether a study prioritizes biological signal detection (favoring z-score) or technical robustness (favoring UCell or GSVA).
3. Extensible Infrastructure: PathwayBench is released as a versioned, reproducible scaffold. It includes per-dataset scores, evaluation code, and an interactive "PathwayBench Advisor" application that allows users to input their specific priorities (e.g., maximum sensitivity vs. cross-normalization stability) to receive ranked method recommendations.

Conclusion
The authors conclude that practitioners analyzing transcriptomes with broad remodeling (e.g., fibrosis, inflammation) should prefer magnitude-aware methods or explicitly check for rank-window competition before relying on rank-based scores. The benchmark serves as a tool for evidence-based method selection and community extension, rather than a final verdict on a single "best" algorithm.