Transcriptomic Models for Immunotherapy Response Prediction Show Limited Cross-cohort Generalisability

This study systematically benchmarks nine state-of-the-art transcriptomic models for predicting immune checkpoint inhibitor response and finds that they exhibit limited cross-cohort generalisability and biological consistency, performing at or near chance levels across independent datasets.

The Gist

The Big Picture: The "Crystal Ball" Problem

Imagine cancer treatment as a high-stakes game of chess. Immune Checkpoint Inhibitors (ICIs) are a new, powerful type of chess piece that wakes up the body's immune system to fight the cancer. The problem? This new piece works brilliantly for some players (patients) but is completely useless for others. In fact, it only works for about 0% to 40% of people.

Doctors need a "crystal ball" to predict before starting treatment who will win the game and who will lose their time and money on a treatment that won't work.

For years, scientists have been trying to build these crystal balls using Transcriptomics. Think of transcriptomics as taking a massive, detailed photo of every single instruction (gene) inside a tumor cell. By looking at these photos, researchers hope to find a pattern that says, "This patient will respond," or "This patient won't."

The Study: The "Taste Test"

This paper is essentially a blind taste test for nine different "crystal ball" recipes that scientists have created recently.

The researchers gathered nine of the most advanced models (some looking at bulk photos of the whole tumor, others looking at individual cell photos) and asked a simple question: "Do these models work on patients they have never seen before?"

Usually, these models are trained on one specific group of patients (like a chef learning to cook a dish for a specific family). The researchers wanted to see if these chefs could cook the same delicious dish for a completely different family with different tastes.

The Results: A Mixed Bag of "Almosts"

The results were a bit disappointing, but very important. Here is what they found, using some metaphors:

1. The "One-Size-Fits-All" Myth is Broken
Most of the models performed poorly when tested on new groups of patients. It's like a chef who makes a perfect lasagna for their Italian grandmother but serves a burnt, bland version to a customer in a different country.

• The Bulk Models (The Wide-Angle Lens): These models look at the whole tumor as a big blur. They generally performed at the level of a coin flip (50/50 chance). They couldn't reliably tell the difference between a winner and a loser.
• The Single-Cell Models (The Microscope): These models look at individual cells, which is much more detailed. They did slightly better, but they were still unreliable. They worked great on one specific type of tumor but failed miserably on another.

2. The "Overfitting" Trap
Some models looked amazing on the data they were trained on but failed completely on new data. This is called overfitting.

• Analogy: Imagine a student who memorizes the answers to a specific practice test. They get 100% on the practice test. But when they walk into the real exam with slightly different questions, they fail because they didn't learn the concepts, they just memorized the answers. Many of these models just memorized the specific patients they were trained on.

3. The "Language Barrier"
The models also struggled because they speak different "languages."

• Some models were trained on data processed one way (like measuring ingredients in cups), and the new data was measured another way (like measuring in grams). When you mix these up, the recipe fails.
• Some models were built for specific cancers (like Triple-Negative Breast Cancer) and tried to apply those rules to lung cancer. It's like trying to use a map of New York City to drive in London; the streets are too different.

The Good News: What They Did Agree On

Even though the models were bad at predicting the outcome, they were surprisingly good at agreeing on the biology.

When the researchers looked at why the models made their predictions, they found that the best models all pointed to the same biological themes:

• The "Cytotoxic" Team: They all agreed that patients who respond well have a strong army of "killer" T-cells (like Granzyme B) ready to attack.
• The "Allograft" Signal: They all noticed that successful tumors look a bit like a rejected organ transplant (a sign that the immune system is actively fighting).

It's like having five different weather forecasters. They might disagree on exactly what time the rain will start, but they all agree that it is going to rain. This tells us that the biological signals are real; the problem is just that our current models aren't good enough at reading the map to predict the storm accurately.

The Conclusion: We Need Better Maps

The paper concludes that while we have built some very sophisticated "crystal balls," they aren't ready for the real world yet. They are too sensitive to the specific group of patients they were trained on.

What needs to happen next?

1. Better Training: We need to train these models on a much wider variety of patients, not just one specific group.
2. Standardization: We need to make sure all models speak the same "language" (using the same data processing methods) so they can be compared fairly.
3. New Tech: The authors suggest using AI that understands biology (like a chef who understands why ingredients work together, not just the recipe) and combining gene data with other patient info (like blood tests or tumor size) to build a truly reliable crystal ball.

In short: We have the ingredients for a great immune therapy predictor, but right now, the recipe is too finicky. We need to fix the kitchen before we can serve the meal to patients.

Technical Summary

1. Problem Statement

Immune checkpoint inhibitors (ICIs) have revolutionized cancer therapy, but only a minority of patients (0–40%) respond, while others develop resistance. Accurate pre-treatment prediction is a critical unmet need. While transcriptomic biomarkers derived from bulk and single-cell RNA sequencing (scRNA-seq) offer a promising avenue for capturing tumor-immune interactions, the cross-cohort generalisability of existing state-of-the-art prediction models remains unclear. Most models are developed and validated on specific, often small, cohorts, raising concerns about overfitting, domain shift, and their applicability to unseen clinical populations with different tumor types, sequencing technologies, and immune contexts.

2. Methodology

The authors conducted a systematic, rigorous benchmark of nine state-of-the-art transcriptomic ICI response predictors using strictly unseen, independent datasets.

• Models Evaluated:
  • Bulk RNA-seq Models (5): COMPASS (Transformer-based foundation model), IRNet (Graph Neural Network), NetBio (Network-based + Logistic Regression), IKCScore (Fixed statistical signature), and TNBC-ICI (Fixed gene panel for Triple-Negative Breast Cancer).
  • scRNA-seq Models (4): PRECISE (XGBoost + Reinforcement Learning), DeepGeneX (Deep Neural Network), Tres (Statistical T-cell resilience signature), and scCURE (MLP + cell embedding).
• Datasets:
  • Bulk: Three independent cohorts (Cho et al., Ribas et al., Poddubskaya et al.) covering NSCLC and Melanoma.
  • scRNA-seq: Four independent cohorts (Gondal et al., Franken et al., Luoma et al., Reinstein et al.) covering various cancers (Melanoma, HNSCC, etc.).
• Evaluation Protocol:
  • Strict Unseen Testing: Models were tested on data they were not trained on.
  • Unified Preprocessing: Data was harmonized (GENCODE v36, Ensembl IDs) but processed according to each model's specific requirements (e.g., TPM for some, TMM for others, log-CPM for 10x data).
  • Metrics: Accuracy, Macro F1-score (crucial for handling class imbalance), and AUC. Runtime was also recorded.
  • Biological Analysis: Pathway-level enrichment (GSEA) and network clustering (EnrichmentMap) were performed to assess biological consistency and convergence across models.

3. Key Contributions

• First Cross-Modality Benchmark: This is the first study to systematically compare bulk and single-cell transcriptomic models for ICI response on a unified set of independent, unseen datasets.
• Revealing the "Generalisation Gap": The study demonstrates that despite high performance in original publications, most models degrade significantly when applied to new cohorts, often performing near chance levels.
• Biological Convergence Analysis: The authors move beyond accuracy metrics to analyze what biological signals models are capturing, revealing that while some models converge on immune themes, others capture irrelevant metabolic or technical artifacts.
• Runtime and Usability Assessment: The paper highlights the computational trade-offs, showing that many high-performing models require cohort-specific retraining, limiting their clinical utility for single-sample prediction.

4. Key Results

A. Predictive Performance

• Overall Modest Performance: Across all unseen cohorts, predictive performance was generally modest.
  • Bulk Models: Performed at or near chance levels (AUC ~0.5) in most cohorts.
    • COMPASS: Despite pretraining on 10,000+ TCGA samples, it failed to generalize well (AUC 0.31–0.55), suggesting foundation models struggle with domain shift in this context.
    • NetBio: Showed instability, achieving perfect metrics (1.0) on one cohort (likely overfitting) but failing on others.
    • TNBC-ICI: Performed near chance outside its specific breast cancer context, highlighting the lack of portability of tumor-specific panels.
  • scRNA-seq Models: Showed slightly better performance but remained highly dataset-dependent.
    • PRECISE: Achieved the best overall performance (F1 up to 0.81 in Luoma et al.) and showed improvement with feature refinement, suggesting cell-state-aware features are valuable.
    • DeepGeneX: Generalized only in cohorts with similar immune contexts; failed in divergent landscapes.
    • Tres & scCURE: Performed near chance or showed limited ranking capacity (low AUC) despite decent F1 scores.

B. Biological Consistency

• Sparse Overlap: There was little reproducible overlap in specific pathway-level biomarkers across models.
• Convergence on Immune Themes:
  • PRECISE and Tres showed the most biological coherence, converging on immune-related programs like Allograft Rejection, Graft-versus-host disease, and Granzyme B (GZMB) pathways.
  • Fixed Panels (IKCScore, TNBC-ICI): Converged on adaptive immunity and T-cell receptor signaling.
• Divergence and Artifacts:
  • IRNet was an outlier, predominantly capturing metabolic pathways (pyruvate, nitrogen metabolism) with almost no overlap with immune-specific pathways, suggesting a bias toward cellular metabolism rather than ICI biology.
  • Tres identified a vast number of heterogeneous pathways, reducing specificity.

C. Technical and Practical Constraints

• Class Imbalance: High class imbalance ratios (0.25–0.94) in real-world datasets severely impacted performance, with many models failing to predict the minority class (responders) effectively.
• Runtime: scRNA-seq models were computationally expensive (e.g., scCURE took >280,000 seconds per dataset due to LOOCV retraining), whereas bulk models were generally faster.
• Preprocessing Sensitivity: Performance was highly sensitive to normalization methods (TPM vs. CPM vs. TMM), indicating a lack of robustness to technical variations.

5. Significance and Future Directions

• Clinical Reality Check: The study underscores that current transcriptomic models are not yet ready for broad clinical deployment due to poor cross-cohort robustness. They capture biologically meaningful signals in specific contexts but fail to generalize across tumor types and technologies.
• Need for Domain Adaptation: The failure of foundation models (like COMPASS) suggests that simple pretraining on TCGA is insufficient. Future models require stronger domain adaptation strategies and normalization-agnostic architectures.
• Biological Grounding: The divergence in pathway analysis highlights the need for models to be constrained by biologically grounded mechanisms (e.g., immune activation) rather than purely data-driven correlations that may capture batch effects or metabolic noise.
• Proposed Solutions: The authors suggest future work should focus on:
  • Heterogeneous Graph Neural Networks: To model multi-scale interactions (genes, cells, pathways).
  • LLM Integration: To incorporate prior biological knowledge and improve interpretability.
  • Multi-modal Integration: Combining transcriptomics with clinical, spatial, and genomic data to improve generalisability.

In conclusion, while transcriptomic models offer theoretical promise, this benchmark reveals a critical gap between model development and real-world clinical applicability, necessitating a shift toward more robust, biologically constrained, and domain-adaptive frameworks.