Predicting Pre-treatment Resistance or Post-treatment Effect? A Systematic Benchmarking of Single-Cell Drug Response Models

This study systematically benchmarks single-cell drug response models across diverse datasets, revealing that while scDEAL demonstrates superior robustness to class imbalance, most current methods struggle to predict intrinsic pre-treatment resistance despite effectively capturing post-treatment transcriptional changes, thereby highlighting the need for next-generation models with greater clinical relevance.

The Gist

Imagine you are a doctor trying to predict which cancer cells will survive a new chemotherapy treatment and which will die. In the past, doctors looked at the tumor as a big, blurry smoothie of millions of cells. They couldn't tell if a tiny, dangerous group of "super-survivor" cells was hiding inside.

Now, thanks to single-cell RNA sequencing, we can look at every single cell individually, like zooming in with a super-magnifying glass. This is great, but it created a new problem: We have a lot of different computer programs (AI models) trying to predict which cells will survive, but nobody knows which one is actually the best.

This paper is like a massive, rigorous "Taste Test" or "Olympics" for these computer programs. The researchers put nine different AI models through a series of tough challenges to see who wins.

Here is the breakdown of their experiment using simple analogies:

1. The Contestants (The AI Models)

The researchers gathered nine different "coaches" (AI models like scDEAL, DrugFormer, Beyondcell, etc.). Each coach has a different strategy for guessing which cells will survive the drug. Some use simple math, while others use complex deep learning (like advanced neural networks).

2. The Training Grounds (The Data)

To test these coaches, the researchers used a massive library of data:

• 26 different datasets containing over 760,000 cells.
• These cells came from 12 different types of cancer (like breast, lung, and leukemia) and were treated with 21 different drugs.
• The Challenge: They tested the models in two scenarios:
  • The "Fair Play" Scenario (Balanced): An equal number of cells that die vs. cells that survive. This is like a 50/50 coin toss.
  • The "Real World" Scenario (Imbalanced): In real patients, the "bad" cells (resistant ones) are often rare, like finding one needle in a haystack of a million. The researchers made the data extremely unbalanced (e.g., 100 safe cells for every 1 bad cell) to see if the models could still find the needle.

3. The "Gold Standard" Test (Lineage Tracing)

This is the most clever part of the paper. Usually, we don't know for sure if a specific cell before treatment was going to be a "survivor." We usually just guess based on whether the patient got better or worse later.

To fix this, the researchers used a technique called Lineage Tracing.

• The Analogy: Imagine you have a set of identical twins. You give one twin a label (a barcode) and split them up. You treat one twin with the drug and leave the other alone. Later, you check the label. If the treated twin survived, you know the untreated twin (who you looked at earlier) was destined to survive, too.
• This gave the researchers 100% true answers (Ground Truth) to see if the AI models could actually predict the future before the drug was even given.

4. The Results: Who Won?

• The "Cell Line" vs. "Real Patient" Gap:
  Almost all the models did great when tested on lab-grown cells (cell lines). It's like a video game character doing well in a practice level. But when they moved to real human tissue, their performance dropped significantly. Real tumors are messy and chaotic; lab cells are neat and uniform. The models struggled with the messiness of real life.

• The Imbalance Problem:
  When the data was unbalanced (the "needle in a haystack" scenario), most models panicked. They either missed the rare bad cells entirely or got confused.

  • The Winner: One model, scDEAL, was the "Olympic Champion." It was the most robust. Even when the data was messy or unbalanced, it kept performing better than the others.

• The Big Limitation (The "Crystal Ball" Problem):
  Here is the most important finding: Most models are actually bad at predicting the future.

  • When the models were asked to predict which cells would survive before the drug was given (using the Lineage Tracing data), almost everyone failed. Their scores dropped to the level of random guessing.
  • Why? The models are really good at spotting the changes that happen after the drug hits (like a car crash). But they are terrible at spotting the hidden weakness that existed before the crash. They can't see the "intrinsic resistance" that makes a cell a survivor before the battle even starts.
  • scDEAL was the only one that managed to peek a little bit into the future, but even it wasn't perfect.

5. The "PDAC" Case Study (The Real-World Proof)

To prove scDEAL wasn't just lucky, the researchers tested it on their own new data from Pancreatic Cancer (PDAC) organoids (tiny 3D tumor balls grown in a lab).

• The Result: scDEAL correctly identified which cells were sensitive and which were resistant.
• The "Why": They dug deeper and found that scDEAL's success wasn't just because of its complex math architecture. It was because of how it was taught (the specific labels used during training). It's like a student who gets an A not just because they are smart, but because their teacher gave them the right study guide.
• Biological Check: The model didn't just guess numbers; it correctly identified the biological pathways (like the cell's "survival switches") that actually happen when cancer cells fight back against drugs.

The Bottom Line

This paper tells us two main things:

1. We have a good runner-up: The model scDEAL is currently the best tool we have for predicting drug responses, especially when data is messy or unbalanced.
2. We still have a long way to go: Current AI models are great at describing what happens after treatment, but they are terrible at predicting who will survive before treatment starts. They are like weather forecasters who can tell you it's raining now, but can't tell you if it will rain tomorrow.

The Future: To truly help doctors, we need to build the next generation of AI that can look at a healthy-looking cell and say, "Hey, you look fine, but you have a hidden superpower that will make you survive this drug." Until then, we have to be careful about trusting these predictions too much in the clinic.

Technical Summary

1. Problem Statement

Intratumoral heterogeneity is a primary driver of variable drug responses in cancer. While single-cell RNA sequencing (scRNA-seq) allows for high-resolution profiling of this heterogeneity, the computational models developed to predict drug response at the single-cell level lack systematic evaluation. Key challenges identified include:

• Lack of Ground Truth: Conventional scRNA-seq datasets often infer drug resistance from treatment status (pre- vs. post-treatment) rather than experimentally validated single-cell outcomes, leading to potential labeling bias.
• Class Imbalance: Real-world clinical data often features highly skewed distributions of resistant vs. sensitive cells, yet most models are evaluated on balanced datasets.
• Generalizability: Models trained on cell lines (e.g., GDSC, CCLE) often fail to generalize to complex patient-derived tumor tissues.
• Prediction Goal Ambiguity: It remains unclear whether current models predict intrinsic resistance (pre-treatment state) or merely capture drug-induced transcriptional changes (post-treatment effect).

2. Methodology

The authors conducted a comprehensive benchmarking study involving nine representative single-cell drug response prediction models.

Datasets:

• Scale: 26 curated datasets comprising >760,000 cells across 12 cancer types and 21 therapeutic agents (targeted therapy, chemotherapy, immunotherapy).
• Scenarios:
  • Balanced: Subsets created with equal numbers of sensitive/resistant cells (500–5,000 cells).
  • Imbalanced: Subsets created with minority-to-majority ratios of 1:3, 1:10, 1:30, and 1:100 to simulate real-world scarcity of resistant cells.
  • Lineage-Tracing (Ground Truth): Used the ReSisTrace dataset (Olaparib-treated ovarian cancer). This technique uses heritable barcodes to track sister cells, allowing pre-treatment cells to be definitively labeled as "pre-resistant" or "pre-sensitive" based on the survival of their sister cells post-treatment.
  • Independent Validation: An in-house Pancreatic Ductal Adenocarcinoma (PDAC) organoid dataset treated with Paclitaxel and Trametinib.

Benchmarked Models:
Nine methods representing diverse strategies (statistical, deep learning, transfer learning) were evaluated:

• Beyondcell, CaDRReS-sc, DREEP, DrugFormer, Precily, SCAD, scDEAL, scDr, scIDUC.

Evaluation Metrics:

• AUROC, Accuracy, F1-score, Precision, Recall.
• Normalized AUPRC: Used specifically for imbalanced datasets to correct for baseline random classifier performance.

Experimental Design:

• Label Substitution: To test the impact of training data labeling, the authors replaced scDEAL's training labels with those generated by the SCAD framework to isolate the effect of label construction vs. model architecture.
• Biological Validation: Pseudotime trajectory analysis (Monocle) and pathway enrichment (ssGSEA, AUCell) were performed on model-predicted populations to verify biological plausibility.

3. Key Contributions

1. First Systematic Benchmark: Established a standardized evaluation framework for single-cell drug response models across balanced, imbalanced, and lineage-tracing scenarios.
2. Ground-Truth Validation: Introduced lineage-tracing data as a rigorous benchmark to distinguish between predicting intrinsic resistance vs. drug-induced changes.
3. Identification of Robust Models: Identified scDEAL as the most robust model across diverse scenarios, particularly under class imbalance.
4. Mechanistic Insight: Demonstrated that high performance in conventional settings often reflects the ability to detect post-treatment transcriptional shifts rather than pre-treatment resistance prediction.

4. Key Results

A. Performance on Balanced Datasets:

• Top Performer: scDEAL achieved the highest median AUROC (0.67–0.82) and Accuracy across all sample sizes.
• Context Dependence: All models performed significantly better on cell lines than on patient tumor tissues (Wilcoxon p = 2.8 × 10⁻¹¹).
• Therapy Type: Models generally performed better on targeted therapies than chemotherapies.
• Immunotherapy: Only DrugFormer could handle immunotherapy data, leveraging a knowledge graph of gene functions, though its overall performance was moderate.

B. Performance on Imbalanced Datasets:

• Robustness: Under extreme imbalance (1:100), most models suffered sharp performance declines. scDEAL maintained the highest stability in AUROC and F1-score.
• Majority Class Bias: Many models (e.g., CaDRReS-sc, DrugFormer) performed significantly better when the majority class was "sensitive" rather than "resistant." scDEAL showed minimal sensitivity to the dominant class.
• Metric Nuance: While scDEAL led in AUROC, DREEP and DrugFormer showed competitive performance in specific imbalance ratios (1:30, 1:100) using normalized AUPRC, though DrugFormer failed to capture minority classes at 1:100.

C. Lineage-Tracing Evaluation (The Critical Finding):

• Conventional Setting (Pre vs. Post): Models performed well (scDEAL AUROC ~0.87), likely because they easily distinguish drug-induced transcriptional states.
• Pre-treatment Prediction (Intrinsic Resistance): When tasked with predicting the fate of pre-treatment cells using only pre-treatment data, performance collapsed for most models (AUROC ~0.50–0.55, near random).
• Exception: scDEAL was the only model to retain moderate predictive power (AUROC ~0.60), suggesting it captures latent intrinsic resistance signals better than others.

D. Biological Interpretability (PDAC Case Study):

• scDEAL predictions aligned with known biological mechanisms.
• Trajectory Analysis: Cells predicted as resistant showed progressive acquisition of resistance markers along pseudotime trajectories.
• Pathway Enrichment:
  • Paclitaxel: Resistant cells showed upregulation of G2/M checkpoints and microtubule stability.
  • Trametinib: Resistant cells showed upregulation of MAPK feedback targets and EMT signatures.
• Label Substitution: Replacing scDEAL's training labels with SCAD's labels caused a drastic performance drop, proving that high-quality training label construction is as critical as the model architecture.

5. Significance and Conclusion

• Clinical Relevance Gap: The study highlights a fundamental limitation in current AI: most models predict drug response (post-treatment effect) rather than drug resistance (pre-treatment state). This limits their utility for pre-treatment patient stratification.
• Robustness of scDEAL: scDEAL emerges as the state-of-the-art for current tasks, particularly due to its robustness to class imbalance and its ability to capture intrinsic resistance signals, likely driven by its specific training label construction.
• Future Directions:
  • Development of models specifically designed to predict intrinsic resistance using lineage-tracing data as ground truth.
  • Integration of foundation models (Large Language Models) and generative AI to better capture complex biological patterns.
  • Expansion of benchmarks to include combination therapies, which are standard in clinical practice.

In summary, this paper provides a critical reality check for the field, demonstrating that while current models are effective at characterizing drug-induced changes, they struggle to predict the intrinsic resistance of cells before treatment occurs. It calls for a shift toward more biologically grounded training strategies and rigorous ground-truth validation.