scDiagnostics: systematic assessment of cell type annotation in single-cell transcriptomics data

Technical Summary: scDiagnostics – Systematic Assessment of Cell Type Annotation in Single-Cell Transcriptomics

Problem Statement
While cell type annotation has become a standard component of single-cell RNA sequencing (scRNA-seq) workflows, the rigorous assessment of these computational annotations remains a significant challenge. Current workflows frequently rely on "black box" automated tools that transfer labels from annotated reference datasets to new query datasets. A critical limitation of this approach is the assumption of perfect alignment between reference and query data. In reality, technical batch effects, biological differences (e.g., disease states vs. healthy atlases), and the presence of novel or intermediate cell states often lead to misalignments. Consequently, query cells may be forcibly assigned to the most similar available reference category, resulting in misleading annotations that distort downstream biological interpretation. Existing diagnostic tools are often task-specific, require specific neural network architectures, or rely solely on confidence scores that fail to detect systematic biases or dataset incompatibilities.

Methodology
The authors present scDiagnostics, an open-source R/Bioconductor package designed to provide a systematic, method-agnostic framework for evaluating cell type annotations. The package is compatible with all major annotation tools (e.g., SingleR, Azimuth, CellTypist, scArches) and accepts standard SingleCellExperiment objects, supporting memory-efficient processing of large datasets via DelayedArray.

The framework operates through three complementary analytical components:

1. Projection Visualization: Maps query cells onto a low-dimensional latent space (e.g., PCA) derived from the reference dataset. Unlike non-linear techniques that may distort local density, scDiagnostics employs statistically sound projection methods to visualize the alignment of query cells against their assigned reference counterparts.
2. Alignment Assessment: Quantifies the similarity between matched reference and query cell types using global distributional metrics (e.g., Spearman correlation, Wasserstein distance, Bhattacharyya coefficients, Hellinger distance, Cramér's test, Hotelling's T² test) and cell-type-specific marker gene expression profiles.
3. Anomaly Detection: Identifies specific query cells that deviate significantly from their assigned reference cell type. This module utilizes the Isolation Forest algorithm and PCA reconstruction error to flag outliers. The system distinguishes between four potential causes for anomalies: (i) technical artifacts (batch effects, doublets), (ii) reference incompleteness (true out-of-reference populations), (iii) annotation ambiguity (mapping errors), and (iv) biological drift (disease activation or trajectory shifts).

The package integrates these diagnostics to allow users to interrogate flagged cells, separating them from standard categories for further analysis (e.g., differential expression, cell-cell interaction) rather than discarding them or conflating them with reference categories.

Key Results
The authors validated scDiagnostics using simulated data with known ground truth and two real-world case studies:

• Simulated Data: In scenarios where a cell type present in the query was missing from the reference, standard label transfer methods forcibly misclassified these cells. scDiagnostics successfully identified these "out-of-reference" populations as anomalous, showing a bimodal distribution of anomaly scores that clearly separated misclassified cells from correctly annotated ones. Benchmarking on the Zeisel mouse brain atlas, under conditions of label noise, class imbalance, and simulated batch effects, demonstrated high sensitivity and specificity (high AUROC and AUPRC).
• COVID-19 PBMC Analysis: Applied to peripheral blood mononuclear cells (PBMCs) from COVID-19 patients against a healthy reference, scDiagnostics detected a subset of CD14+ monocytes that were forcibly annotated as healthy monocytes by standard tools (Azimuth, SingleR, etc.). These cells exhibited significantly elevated anomaly scores and a distinct shift in principal component space. Crucially, these anomalous cells showed strong upregulation of an Interferon (IFN) gene signature (e.g., IFI6, LY6E, ISG15), a disease-specific state that was obscured when all CD14+ monocytes were treated as a single population. The tool successfully recovered this disease signal, which was diluted in standard differential expression analyses.
• MERFISH Colitis Analysis: In spatial transcriptomics data from a mouse model of colitis, scDiagnostics identified a subpopulation of fibroblasts that deviated from the homeostatic fibroblast manifold. These anomalous cells were spatially localized near inflamed epithelium and neutrophils and exhibited a rewired ligand-receptor interaction network (gaining inflammatory signals like Il1b-Il1r1 while losing homeostatic Wnt signals). This stratification allowed for a more accurate resolution of cell-cell communication networks compared to analyzing all fibroblasts as a single group.

Significance and Claims
The paper claims that scDiagnostics addresses a critical unmet need in single-cell analysis by providing a systematic, transparent, and tool-agnostic method to detect complex or ambiguous annotation cases that standard workflows miss. The authors emphasize that:

• Confidence scores are insufficient: High confidence scores from annotation tools do not preclude high anomaly scores; many mislabeled cells (particularly those representing disease states) retain high confidence, leading to undetected errors.
• Prevention of "Over-correction": The tool helps identify when integration methods or label transfer strategies "over-correct" by merging distinct biological states (e.g., disease-associated states) into healthy reference manifolds.
• Human-in-the-loop integration: scDiagnostics serves as a quantitative "guardrail" for automated workflows, including emerging foundation models and Large Language Models (LLMs). By flagging high-confidence but anomalous assignments, it triggers necessary human review, preventing the propagation of errors into downstream biological conclusions.
• Broad Applicability: While developed for scRNA-seq, the framework is demonstrated to be effective for spatial transcriptomics and is extensible to multi-modal data, offering a path to detect discrepancies between different data modalities (e.g., transcriptomics vs. surface protein expression).

In conclusion, the authors posit that as single-cell datasets grow in scale and complexity, and as automation increases, tools like scDiagnostics are essential for maintaining rigor, ensuring reproducibility, and enabling the discovery of novel or disease-associated cell states that would otherwise remain obscured.