Single-cell foundation models predict durable CAR T response despite imperfect cell annotation

Technical Summary: Single-cell foundation models predict durable CAR T response despite imperfect cell annotation

Problem Statement
CD19-targeted chimeric antigen receptor (CAR) T cell therapy achieves high initial response rates in pediatric B-cell acute lymphoblastic leukemia (B-ALL), yet approximately half of patients relapse within one year. The duration of post-infusion B-cell aplasia (BCA) serves as a validated pharmacodynamic surrogate for CAR T persistence. While single-cell RNA sequencing (scRNA-seq) of pre-infusion CAR T products contains predictive information regarding long-term persistence, extracting this information traditionally requires highly sophisticated, bespoke bioinformatics and expert cell annotation. This reliance on manual annotation limits reproducibility and clinical translation. Furthermore, it remains unclear whether single-cell foundation models (scFMs)—large transformer-based networks pretrained on millions of cells—can generalize to engineered, antigen-stimulated CAR T products, which represent a significant distribution shift from the resting peripheral blood cells typically used in pretraining corpora.

Methodology
The study evaluated four distinct scFMs: scGPT (generative transformer), scFoundation (masked autoencoder), CellPLM (cell language model), and UCE (universal cell embeddings using protein language model-derived gene representations).

1. Datasets:

   • Reference: A healthy peripheral blood mononuclear cell (PBMC) atlas (48,530 cells, 31 annotated populations) used for zero-shot benchmarking and fine-tuning references.
   • Primary Cohort: Pre-infusion CAR T products from 33 pediatric B-ALL patients (231,088 cells) from two clinical trials. Products were profiled under both basal and CD19-stimulated conditions. Patients were stratified into three groups based on BCA duration: BCA1 (short, n=17), BCA2 (intermediate, n=11), and BCA-L (long-duration sustained, n=5).
   • Post-Infusion: A single-patient post-infusion dataset (9,525 cells) to test cross-context generalization.

2. Workflow:

   • Annotation: Models were applied in zero-shot mode and, for scGPT and scFoundation, after fine-tuning on the healthy donor partition. Cell type annotation was performed via k-nearest-neighbor (k-NN) mapping in the learned embedding space using FAISS.
   • Performance Metrics: Annotation accuracy was measured using weighted F1 scores.
   • Clinical Prediction: Patient-level cell type proportions derived from scFM annotations were used as features for a logistic regression classifier to predict the probability of sustained BCA (BCA-L vs. non-sustained).
   • Validation: Leave-one-out cross-validation (LOOCV) was used to assess predictive performance. Statistical significance of biomarkers was determined via permutation testing (1,000 label shuffles).

Key Results

• Annotation Performance and Domain Shift:

  • On healthy PBMCs, all four models achieved weighted F1 scores >0.84 in zero-shot mode, with high accuracy for abundant populations but poor performance on rare subsets.
  • Applying models to engineered CAR T products resulted in a systematic accuracy penalty, with weighted F1 scores dropping by 0.21–0.25 units compared to PBMCs.
  • CD19 stimulation imposed an additional penalty (0.10–0.26 F1 drop), indicating that activation-induced transcriptional states are underrepresented in pretraining data.
  • Fine-tuning improved scGPT's performance but slightly decreased scFoundation's, though neither fully recovered PBMC-level accuracy.
  • Systematic Failures: All models consistently failed to identify CD4+ regulatory T (Treg) cells (F1 ≈ 0.00) and post-infusion exhausted CD8+ subsets, regardless of architecture or fine-tuning.

• Clinical Prediction Despite Imperfect Annotation:

  • Despite reduced per-cell annotation accuracy, scFM-derived cell proportions successfully stratified patients with long-duration BCA (BCA-L) from non-sustained groups.
  • The LOOCV area under the receiver operating characteristic curve (AUC) was 0.879 (95% CI: 0.742–0.986) using CD19-stimulated data.
  • Biomarker Discovery: CD8+ XCL1/2+ cells were the only population consistently identified as a significant predictor of durable persistence across both scGPT and scFoundation under CD19 stimulation.
  • Superiority over Expert Annotation: For several cell types (e.g., CD4+ Th2/Th9/Th17, CD4+ memory), scFM-derived proportions outperformed proportions derived from original expert manual annotations in discriminative power (e.g., ΔAUC = +0.25 for Th2/Th9/Th17). This suggests scFM embeddings capture continuous transcriptional variation obscured by discrete categorical labeling.
  • Stimulation State: Biomarkers differed between basal and stimulated conditions. CD8+ XCL1/2+ and CD8+ naïve cells were predictive in both, while proliferative populations were significant only in basal samples.

• Cross-Context Generalization:

  • On post-infusion in vivo data, models showed architecture-dependent generalization. CellPLM performed best (F1=0.72), while UCE collapsed (F1=0.51), assigning most cells to a single population.
  • Exhausted T-cell states remained systematically unidentifiable across all architectures in the post-infusion setting.

Key Contributions

1. Demonstration of Robustness: The study establishes that scFMs can extract clinically actionable signals from engineered CAR T products despite significant domain shifts and imperfect per-cell annotation.
2. Biomarker Identification: It identifies CD8+ XCL1/2+ cells as a reproducible, cross-model biomarker associated with durable CAR T persistence, linking this to the recruitment of XCR1+ cDC1 dendritic cells.
3. Annotation vs. Prediction Decoupling: The work challenges the assumption that perfect cell annotation is a prerequisite for accurate clinical prediction, showing that cohort-aggregated proportions can average out annotation noise to preserve downstream signal.
4. Clinical Decision-Support Agent: The authors developed and released a locally deployable AI agent that automates the workflow from scRNA-seq data to patient-level risk prediction, demonstrating feasibility for clinical translation.
5. Benchmarking Limitations: The study highlights critical gaps in current scFMs, specifically the inability to identify regulatory (Treg) and exhausted T-cell states, arguing for the deliberate inclusion of these disease-activated states in next-generation pretraining corpora.

Significance and Claims
The paper claims that while current scFMs are not yet perfect for per-cell annotation in engineered cellular therapy contexts, their embeddings carry sufficient phenotypic signal to support cohort-level clinical stratification. The authors posit that the "right test" for clinical translatability is not standard integration benchmarks or per-cell accuracy, but rather the ability of representations to carry phenotype-relevant signal in downstream tasks under domain shift.

The study modestly frames its findings as a proof-of-concept. It acknowledges that the BCA-L cohort is small (n=5), resulting in wide confidence intervals, and that the comparison against expert annotation is a within-dataset benchmark requiring orthogonal validation (e.g., flow cytometry). The authors emphasize that the decision-support agent is currently a prototype requiring prospective validation in larger, multi-center cohorts before routine clinical deployment. They conclude that future scFMs intended for cellular therapy must be trained on data that explicitly includes regulatory and exhausted immune states to overcome current systematic failures.