📄 other

Do Single-Cell Models Learn Real Biology? An Empirical Analysis of Physical Interaction Signal in Pretrained scGPT Gene Embeddings

This empirical study demonstrates that the pretrained gene-token embeddings of the scGPT model inherently encode physical protein-protein interaction structures, as evidenced by significantly higher cosine similarity between interacting gene pairs compared to matched non-interacting controls across multiple databases.

Original authors: Liu Chen

Published 2026-07-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Liu Chen

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the secret language of life. For years, scientists have been building massive libraries of "single-cell" data. Think of a single cell as a tiny, bustling city where thousands of genes are the workers. Usually, we look at the whole city at once, but single-cell technology lets us peek at just one worker at a time to see what they are doing. Recently, scientists started using "foundation models"—super-smart AI systems originally designed to read human books—to read these genetic libraries instead. These models, like the one called scGPT, are trained on millions of these genetic snapshots without being told the answers to specific questions. They just soak up the patterns.

But here is the big, nagging question: Are these AI models actually learning real biology, or are they just really good at guessing? It's like asking if a student who memorized a dictionary actually understands how to write a novel, or if they just know which words often appear next to each other. To find out, we need to test if the AI has learned the hidden rules of how things fit together in the body. Specifically, we want to know if the AI "knows" that certain proteins (the workers built by genes) physically hug or shake hands with each other inside our cells, even though the AI was never shown a list of those handshakes. If the AI can figure this out just by reading the genetic data, it means it has captured a piece of real biological truth in its brain.

This paper, titled "Do Single-Cell Models Learn Real Biology?", sets out to answer that exact question using a clever detective game. The researchers took a pre-trained AI model called scGPT and asked it to look at pairs of genes. They wanted to see if the AI's internal "map" placed genes that physically interact closer together than genes that don't. They didn't teach the AI anything new; they just looked at the map it had already built.

The team used two giant databases of known protein interactions (BioGRID and STRING) as their "answer key." They took 89,187 pairs of genes that are known to physically interact and compared them to pairs that were carefully matched to look similar but had no recorded interaction. It's like taking a group of people who are known to be best friends and comparing them to a group of strangers who happen to have the same job and age, to see if the AI can tell the friends apart just by looking at their "genetic fingerprints."

The results were a mix of "yes, but..." The AI did show a real, measurable signal. When the researchers measured how close the AI placed interacting genes, they found that the known interacting pairs were indeed closer together than the strangers. In the world of statistics, this is a modest but real victory. For the BioGRID database, the AI got a score of 0.581 (where 0.5 is a pure guess and 1.0 is perfect). For the STRING database, the score got better as the confidence in the interaction data got higher, reaching 0.686 for the most trusted interactions. This suggests that the AI's internal geometry does contain a "shadow" of real physical relationships, likely because interacting proteins often work together in the same cellular neighborhoods, and the AI picked up on those shared patterns.

However, the paper is very careful not to overhype this. The authors explicitly state that this signal is not strong enough to be used as a standalone tool to predict new interactions. If you tried to use this AI to find a new protein handshake, it would make too many mistakes. The model contains the biology, but it doesn't fully understand the causes or the direction of the interactions. It's like the AI has a vague sense of who hangs out together, but it doesn't know why they are friends or who started the friendship. The study proves that these massive models aren't just random noise generators; they have absorbed some real structural knowledge of how life works, but they are still far from being perfect biologists. The takeaway is that the biology is there, buried in the math, waiting to be unlocked, but we can't just trust the model to do the heavy lifting on its own yet.

Technical Summary: An Empirical Analysis of Physical Interaction Signal in Pretrained scGPT Gene Embeddings

Problem Statement
While single-cell foundation models (scFMs) like scGPT are trained on millions of transcriptomes to learn reusable structures, it remains unclear whether their internal representations contain biological relationships that were not explicitly supplied as training labels. Existing benchmarks often test whether a model can use its representations for downstream tasks (e.g., cell-type annotation) or outperform simple baselines. However, these tasks do not definitively prove that the model's static weights encode specific biological structures, such as physical protein-protein interactions (PPIs). This paper addresses a narrow, controlled question: Do the static gene-token embeddings of a pretrained scGPT model rank independently curated physical protein interactions higher than carefully matched non-interacting gene pairs?

Methodology
The study employs a rigorous, label-free evaluation of the official whole-human scGPT checkpoint without any fine-tuning or additional training.

Data Sources:
- Model: The 512-dimensional gene-token embedding matrix was extracted from the pretrained scGPT checkpoint.
- Positive Pairs (Ground Truth):
  - BioGRID 5.0.259: 89,187 multi-validated human physical interactions (supported by multiple experimental systems or publications).
  - STRING v12: Physical links used for replication at three confidence thresholds (400, 700, and 900), yielding 207,943, 85,452, and 44,484 pairs respectively.
- Negative Pairs (Control): For every positive pair, a "non-edge" was sampled. Crucially, these negative pairs were degree-matched: one endpoint was retained, and the other was replaced by a gene from the same degree stratum (20 strata total) to prevent hub status from acting as a shortcut.
Representation Processing:
- Primary analysis used the token vectors ( $w_g$ ) transformed by the model's learned LayerNorm parameters and normalized to unit length ( $z_g$ ).
- Sensitivity analyses tested raw tokens, centered tokens, and random 512-dimensional Gaussian vectors as controls.
Evaluation Metrics:
- Similarity Score: Cosine similarity between gene pair embeddings.
- Discrimination: Area Under the Receiver Operating Characteristic Curve (ROC-AUC) and Average Precision (AP).
- Uncertainty Estimation: To account for gene-gene dependencies, a "gene-block jackknife" approach was used (deleting pairs incident to specific gene blocks) to calculate 95% confidence intervals, rather than standard pair-level bootstrapping.

Key Results
The study found that the static geometry of the pretrained scGPT embeddings contains a reproducible, albeit modest, signal of physical protein interactions.

BioGRID Performance:
- Interacting pairs showed higher cosine similarity than degree-matched non-edges (Mean: 0.090 vs. 0.055).
- ROC-AUC: 0.581 (95% CI: 0.573–0.589).
- Average Precision: 0.604.
- Enrichment was graded: the fraction of observed interactions in the top 5% of scores was 80.1%, rising monotonically from the lowest to the highest similarity deciles.
STRING Confidence Gradient:
- Discrimination improved monotonically as the confidence threshold of the STRING database increased, supporting the hypothesis that higher-confidence biological evidence correlates with stronger embedding signals.
- ROC-AUC: 0.612 (threshold 400) $\rightarrow$ 0.657 (threshold 700) $\rightarrow$ 0.686 (threshold 900).
- The mean cosine similarity for positive pairs increased from 0.097 to 0.148 as confidence rose, while negative pair similarity remained stable (~0.05–0.06).
Robustness and Controls:
- Representation Sensitivity: Results were nearly identical using raw token vectors (ROC-AUC 0.582 for BioGRID) versus LayerNorm-transformed vectors. Centering reduced performance slightly but retained the signal.
- Random Control: Random 512-dimensional vectors produced chance-level performance (ROC-AUC $\approx$ 0.50), confirming the signal is specific to the learned scGPT geometry.
- Sampling Stability: Results remained stable across ten independent negative sampling draws.
- Degree Stratification: The signal persisted across all gene-degree quartiles, though BioGRID performance was strongest in lower-degree quartiles and weaker in the highest-degree quartile.

Significance and Claims
The paper concludes with a precise, modest claim regarding biological containment:

Containment vs. Causality: The results provide direct evidence that a pretrained single-cell model's static token geometry contains physical-interaction structure. This supports the claim that the model "contains" biological relationships. However, the authors explicitly state this does not prove the model possesses causal understanding, regulatory logic, or that it will outperform supervised models in downstream interaction prediction tasks.
Utility: The effect size (ROC-AUC ~0.58–0.69) is insufficient for a standalone interaction predictor. The authors suggest the geometry is more useful for prioritization (enrichment in the upper tail) than for exhaustive classification.
Mechanism: The likely mechanism is that interacting proteins participate in shared complexes and pathways, leading to co-expression patterns across cell states. The self-supervised transcriptome prediction task compresses these recurring dependencies into the token geometry without ever seeing an interaction database.
Evaluation Framework: The paper advocates for a three-step evaluation sequence: (1) test if the representation contains a specific biological relation; (2) test if the downstream readout utilizes it; and (3) compare the full system against task-specific baselines. This prevents overstating modest representational results as general biological intelligence.

In summary, the study demonstrates that "some real biology is present in the representation," but its strength and utility must be measured task by task, distinguishing between the presence of biological structure and the model's ability to leverage it for superior downstream performance.

Technical Summary: An Empirical Analysis of Physical Interaction Signal in Pretrained scGPT Gene Embeddings

More like this