Benchmarking zero-shot single-cell foundation model embeddings for cellular dynamics reconstruction

This study demonstrates that current zero-shot single-cell foundation models underperform traditional highly variable gene baselines in reconstructing cellular dynamics due to an architectural tendency to over-compress temporal signals and artificially linearize branched biological trajectories.

The Gist

The Big Picture: Trying to Reconstruct a Movie from Stills

Imagine you are a detective trying to figure out how a movie was made, but you only have a few scattered snapshots (photos) taken at different times. You don't have the video; you just have the stills. Your job is to figure out the story: How did the characters move? Where did they go? Did they split into two different groups?

In biology, scientists face this exact problem with cells. They can take "snapshots" of cells at different times (like a developing embryo or a tumor growing), but they can't watch a single cell change in real-time because the process destroys the cell. They need to use math to connect the dots and reconstruct the "movie" of how cells change, grow, and decide their fate.

The Contenders: The "Old School" vs. The "AI Superstars"

To solve this puzzle, the researchers tested two different ways to organize the data:

1. The Old School Method (HVG-PCA): This is like a traditional detective who looks at the most obvious, changing clues. They pick the genes that change the most (Highly Variable Genes) and use a simple map to plot them. It's a straightforward, reliable tool.
2. The AI Superstars (Foundation Models): These are the new, massive Artificial Intelligence models (like Geneformer, scGPT, etc.). They have been "trained" on millions of cell snapshots from all over the internet. The hope was that these AIs, having seen so much data, would be like a genius detective who understands the deep, hidden rules of biology and could reconstruct the movie perfectly without needing to be taught the specific story first.

The Experiment: The "Zero-Shot" Challenge

The researchers set up a rigorous test. They took real biological datasets (like cells turning into skin cells, or stem cells turning into blood cells) and hid parts of the timeline.

• The Test: They gave the AI models and the Old School method the beginning and middle of the story, then asked them to guess the end (Extrapolation). Or, they gave them the end and asked them to guess the beginning (Backtracking). Or they asked them to fill in the missing middle scenes (Interpolation).
• The Goal: See which method could draw the most accurate "movie" of the cell's journey.

The Shocking Result: The AI Got Lost

The researchers expected the AI models to win because they are so powerful. They didn't.

In fact, the simple, "Old School" method (HVG-PCA) consistently outperformed the fancy AI models. The AI models struggled to predict the future or the past accurately.

Why did the AI fail?
The paper suggests the AI models are "over-smart" in a bad way. Because they were trained on millions of different cells to find common patterns, they learned to ignore the "noise" and focus on the "stable" parts of a cell's identity.

• The Analogy: Imagine the AI is a translator who is so good at summarizing a book that it turns every exciting, fast-paced action scene into a boring, flat summary. It sees a cell changing from a baby to an adult and thinks, "Oh, they are both just 'human cells,' so I'll make them look the same."
• The "Linearization" Problem: Real biology is messy and branching (like a tree with many forks). The AI models tended to squash these complex branches into a straight, boring line. They smoothed out the critical moments where a cell makes a big decision (like "become a heart cell" vs. "become a brain cell"), making it look like all cells just follow one straight path.

The "Temporal Compression" Bottleneck

The authors call this a "Temporal Compression" bottleneck.

Think of time as a river with rapids, waterfalls, and calm pools.

• The Old School method sees the rapids and the waterfalls clearly. It knows where the water speeds up and where it splits.
• The AI models act like a filter that smooths the river into a calm, straight canal. They treat the "changes over time" as if they were just background noise (like a dirty camera lens) and tried to clean them away. In doing so, they accidentally erased the very thing they were supposed to study: the movement and change.

The Takeaway

This paper is a reality check for the field of biology.

• The Good News: We have powerful AI tools that are great at identifying what type of cell we are looking at (static tasks).
• The Bad News: These same tools are currently terrible at predicting how cells move and change over time (dynamic tasks). They are too busy trying to find the "average" cell that they miss the unique, fleeting moments of transformation.

The Conclusion: If you want to understand the story of how cells change, don't just rely on the giant AI models yet. The simpler, more traditional tools are currently better at keeping the plot twists and the branching paths of life intact. To fix the AI, scientists need to teach it to value "change" and "time" just as much as it values "stability."

Technical Summary

1. Problem Statement

Reconstructing cellular trajectories from time-resolved single-cell transcriptomics is essential for understanding biological processes like development and disease progression. However, single-cell assays are destructive, yielding only static snapshots rather than continuous longitudinal data. Computational methods must therefore infer dynamics from these snapshots.

While Single-Cell Foundation Models (scFMs) (e.g., Geneformer, scGPT) have shown promise in static tasks (clustering, annotation) via zero-shot transfer, their ability to capture non-linear temporal dynamics and branching lineages remains uncharacterized. It is unclear whether the universal representations learned by these large-scale pre-trained models are superior to traditional Highly Variable Gene (HVG) baselines for inferring cellular flows, particularly in challenging scenarios like backtracking progenitor states or extrapolating future fates.

2. Methodology

The authors established a systematic benchmark framework to isolate the quality of embeddings from downstream modeling choices.

• Datasets: Five diverse time-series scRNA-seq datasets were used, covering hematopoietic lineage branching, embryoid body development, epithelial-to-mesenchymal transition (EMT), and pancreatic differentiation. Cell counts ranged from ~3,000 to ~49,000.
• Embedding Pipelines: Six embedding strategies were compared:
  1. Baseline: HVG selection + Principal Component Analysis (HVG-PCA).
  2. Foundation Models: Zero-shot embeddings from five models: Geneformer, Genecompass, scGPT, UCE, and scFoundation.
• Trajectory Inference: Four Optimal Transport (OT)-based methods were applied to the embeddings to reconstruct dynamics:
  • Dynamical Optimal Transport (DOT)
  • Unbalanced Dynamical Optimal Transport (UOT)
  • Dynamical Schrödinger Bridge
  • Regularized Unbalanced Optimal Transport (RUOT)
• Evaluation Tasks: The framework tested three dynamical scenarios by partitioning time points:
  1. Backtracking: Reconstructing initial states from later time points.
  2. Interpolation: Recovering transient intermediates between observed time points.
  3. Extrapolation: Predicting future states beyond the observed window.
• Alignment & Metrics: To ensure fair comparison, embeddings were aligned into a shared latent space using Generalized Procrustes Analysis (GPA). Performance was quantified using three metrics:
  1. Distributional Recovery: Wasserstein-1 distance (EMD) between predicted and observed cell state distributions.
  2. Temporal Ordering: Spearman correlation between inferred pseudotime and reference pseudotime.
  3. Directional Consistency: Local velocity coherence (cosine similarity of velocity vectors among neighbors).

3. Key Results

The study found that HVG-PCA embeddings consistently outperformed zero-shot scFM embeddings across most tasks and metrics.

• Superiority of HVG Baseline: The HVG-PCA baseline achieved the lowest Wasserstein-1 distances (best distributional recovery) and highest velocity coherence across diverse biological systems. This was particularly evident in backtracking and extrapolation tasks, where foundation models struggled to recover unobserved states.
• Foundation Model Limitations:
  • Geneformer and scGPT were the most competitive foundation models but still lagged behind HVG.
  • scFoundation (which encodes raw expression values) performed the worst.
  • While some foundation models showed comparable velocity coherence in specific regimes, they generally failed to preserve the distributional complexity of unobserved cells.
• Mechanistic Analysis (The "Temporal-Compression" Bottleneck):
  • Temporal Compression: Foundation models exhibited a significantly lower Time Variance Ratio (TVR) compared to HVG. They compressed time-scale differences, effectively treating temporal variation as noise (similar to batch effect correction).
  • Branch Blurring: In bifurcating systems (e.g., pancreatic differentiation, hematopoiesis), foundation model embeddings reduced the separation between distinct cell fates (e.g., SC-β vs. SC-EC, Neutrophils vs. Monocytes). This "linearization" of branched structures obscured critical divergence points.
  • Inductive Bias: The authors argue that current self-supervised training objectives (e.g., masked token prediction) prioritize stable, persistent cell identity signals over transient, process-specific temporal signals, leading to an over-smoothing of dynamic trajectories.

4. Key Contributions

1. First Systematic Benchmark of scFMs for Dynamics: This work is the first to rigorously evaluate zero-shot scFM embeddings specifically for trajectory inference and dynamical reconstruction, moving beyond static tasks like clustering.
2. Identification of a Fundamental Bottleneck: The paper identifies a "temporal-compression" bottleneck where current foundation models inadvertently suppress the very temporal and branching signals required for accurate trajectory inference.
3. Robust Sensitivity Analysis: The authors demonstrated that the superiority of HVG embeddings is robust to variations in alignment strategies, reference spaces, and latent dimensionality, confirming the finding is not an artifact of specific hyperparameters.
4. Quantitative Framework: The study provides a comprehensive evaluation pipeline (embedding × inference × task) with standardized metrics (EMD, Pseudotime, Velocity Coherence) for future dynamical model development.

5. Significance and Implications

• Current State of scFMs: While scFMs provide unified views of cell states, they are currently not superior to simple HVG baselines for reconstructing cellular dynamics. In fact, their tendency to over-correct variation may degrade performance in dynamical tasks.
• Guidance for Future Development: The findings suggest that next-generation foundation models must explicitly balance robustness to technical noise with the retention of biologically meaningful temporal structure. Future pretraining objectives should likely incorporate temporal dynamics or branching constraints rather than relying solely on static reconstruction losses.
• Practical Recommendation: For researchers aiming to reconstruct cellular trajectories from snapshot data, traditional HVG-based embeddings remain the more reliable and robust choice over zero-shot foundation model embeddings.