Multi-Dimensional Spectral Geometry of Biological Knowledge in Single-Cell Transformer Representations

This paper demonstrates that single-cell foundation models like scGPT organize gene representations into an interpretable, multi-dimensional geometric structure that systematically encodes biological knowledge, including subcellular localization, protein interaction networks, and regulatory hierarchies.

The Gist

Imagine you have a giant, super-smart robot that has read every biology textbook ever written and studied millions of human cells. This robot, called scGPT, is designed to understand how cells work. But there's a problem: inside the robot's brain, the information is stored in a massive, messy cloud of numbers that looks like static on an old TV. Scientists call this a "black box" because we couldn't see how the robot was thinking, only what it predicted.

This paper is like a team of detectives who finally found a way to peek inside the robot's brain and map out its internal logic. They discovered that the robot isn't just memorizing facts; it has built a 3D mental map of biology that is surprisingly organized, almost like a city plan.

Here is the breakdown of their discovery using simple analogies:

1. The Robot's Brain is a "Biological City"

The researchers found that the robot organizes genes (the instructions for building proteins) into a structured coordinate system, much like a city is organized by neighborhoods.

• The Main Street (The Secretory Pathway): The most important line in the robot's map separates genes based on where they live in the cell.

  • On one end of the street, you have genes for secreted proteins (like messengers sent outside the cell).
  • On the other end, you have cytosolic proteins (the workers staying inside).
  • The Magic: As the robot processes information, it doesn't just stop at "inside vs. outside." It recreates the actual journey a protein takes: first the Mitochondria (the power plant), then the ER (the factory), and finally the Extracellular Space (the delivery zone). The robot has learned the story of how a protein is made and shipped, not just the destination.

• The Social Network (Who Hangs Out With Whom): Another part of the map groups genes based on who physically touches whom.

  • If two proteins are known to shake hands (interact) in real life, the robot places them right next to each other in its mental map.
  • The Cool Part: The robot is so smart that it can tell how strongly they shake hands. The stronger the bond, the closer they sit together. It's like a high school cafeteria where the best friends sit at the same table, and the robot knows exactly who is the "popular kid" and who is just an acquaintance.

• The Bosses and the Workers (Regulation): The robot also maps out who is in charge.

  • It separates the Transcription Factors (the bosses who give orders) from the Target Genes (the workers who follow orders).
  • The Twist: The robot processes this in stages. In the early layers of its brain, it remembers specific details (e.g., "Boss A tells Worker B to stop"). In the deeper layers, it gets the big picture (e.g., "Bosses are different from Workers"). It's like a manager who first checks the specific tasks on an employee's to-do list, then later just remembers "John is a manager."

2. The "Germinal Center" Dance

One of the most beautiful discoveries involved B-cells (a type of immune cell). The researchers watched how the robot handles the genes that control B-cell development.

• The Anchor: There is one gene, PAX5, that acts as the "home base" for B-cells. It stays in the same spot in the robot's map the whole time.
• The Journey: Other genes, like BATF and BACH2, start far away from home base when the robot first looks at them. But as the robot thinks deeper, these genes slowly "walk" toward PAX5, getting closer and closer.
• The Meaning: This mirrors real life! In a human body, B-cells start as generic cells and only become specialized "B-cell experts" after a specific process (the germinal center reaction). The robot has learned this timeline. It knows that these genes become B-cell leaders later in the process, not from the start. It's like watching a movie in the robot's brain rather than just looking at a photo.

3. What the Robot Doesn't Know

The scientists were also honest about what the robot failed to learn.

• It didn't learn some complex topological shapes (like donut shapes in data) that some hoped it would.
• It didn't learn the same things as a different robot model (Geneformer), proving that this specific robot learned its own unique way of seeing biology.
• This is actually good news! It means the robot isn't just copying a textbook; it's building its own understanding, and we now know exactly where its strengths and weaknesses are.

Why Does This Matter?

Before this, using AI in biology was like driving a car with a blindfold on—you could get to the destination, but you didn't know the road.

Now, because we have this map:

1. We can trust the robot: We can check if its internal map matches real biology before we let it make medical decisions.
2. We can find new drugs: If we know the robot groups proteins by how strongly they interact, we can use that map to guess which drugs might work, even if we haven't tested them yet.
3. We can fix broken models: If we train a new robot and its map looks messy (no clear "Main Street" or "Social Network"), we know it hasn't learned biology correctly and needs to be retrained.

In short: The authors proved that this AI isn't just a fancy calculator. It has built a geometric, 3D mental model of how life works, organizing genes by where they live, who they touch, and who they listen to. It's a giant leap toward making AI a true partner in understanding life.

Technical Summary

1. Problem Statement

Single-cell foundation models (e.g., scGPT, Geneformer) have achieved state-of-the-art performance in tasks like cell-type annotation and gene perturbation prediction. However, a fundamental gap remains in understanding what biological knowledge these models actually encode within their internal representations.

• The Question: Do these models merely memorize statistical correlations in gene expression, or do they learn an interpretable, structured internal model of cellular organization?
• The Gap: Previous interpretability work focused on attention patterns, which were found to encode some biological structure but often failed to provide incremental value over trivial baselines and were heavily confounded by co-expression. The residual stream geometry (the hidden states passed between layers) remained an unexplored frontier for biological interpretability.

2. Methodology

The authors performed a systematic, automated geometric audit of the scGPT model (12 transformer layers, 512-dimensional hidden states) using immune-lineage data from the Tabula Sapiens dataset.

• Automated Hypothesis Screening: Instead of testing a single hypothesis, the authors employed an automated two-agent loop (Executor and Brainstormer) that iterated 63 times to test 183 hypotheses across 13 families. This loop utilized explicit permutation controls, confound checks, and cross-seed replication to ensure rigor.
• Spectral Analysis (SVD): The core technique involved applying Singular Value Decomposition (SVD) to the gene embedding matrices at each of the 12 transformer layers. This allowed the authors to decompose the high-dimensional representation space into orthogonal spectral axes (singular vectors) and analyze the variance and biological meaning of each axis.
• Geometric Metrics:
  • Effective Rank & Intrinsic Dimensionality: To measure how the model compresses information across layers.
  • Co-pole Enrichment: Testing if known biological pairs (e.g., Protein-Protein Interactions, TF-Target pairs) cluster at the same poles (top/bottom) of specific singular vectors.
  • Residualization: Regressing out co-expression signals to isolate geometric structure independent of expression correlation.
  • Trajectory Analysis: Tracking the geometric distance and rank of specific genes (e.g., B-cell regulators) across transformer depth.

3. Key Contributions

1. Discovery of a Biological Coordinate System: The paper demonstrates that scGPT does not organize genes in an opaque feature space but rather in a structured, multi-dimensional biological coordinate system.
2. Decoupling Attention from Residual Geometry: The study establishes that the residual stream contains biological information (specifically regulatory relationships) that is invisible to attention-based analysis and independent of co-expression.
3. Layer-Specific Information Processing: It reveals a "division of labor" across transformer layers, where early layers preserve fine-grained regulatory edges and deeper layers compress this into categorical distinctions.
4. Rigorous Negative Findings: By systematically retiring hypotheses that failed under strict controls (e.g., topological features, feed-forward loops), the paper defines the boundaries of what the model does not encode, increasing confidence in the positive findings.

4. Key Results

A. Progressive Spectral Compression

• The model progressively concentrates gene representations onto fewer directions. The effective rank drops 14.4-fold from Layer 0 (23.6) to Layer 11 (1.6).
• By the final layer, a single direction (SV1) accounts for 93.4% of the variance in gene embeddings. This is a learned compression that distills biological signal rather than discarding it.

B. The Three Orthogonal Biological Axes

The dominant spectral axes encode three fundamental aspects of cell biology:

1. SV1 (Localization): Encodes the secretory pathway.
   • Separates secreted/extracellular proteins (one pole) from cytosolic proteins (other pole).
   • Intermediate layers transiently encode mitochondrial and ER compartments, mirroring the biological sequence of protein synthesis and secretion (Mitochondria $\to$ ER $\to$ Extracellular).
2. SV2–SV4 (Interaction Networks): Encodes Protein-Protein Interactions (PPI).
   • Interacting proteins are geometrically co-localized.
   • Quantitative Grading: The geometric proximity correlates monotonically with experimental interaction strength (STRING confidence scores), with a perfect Spearman $\rho = 1.000$ across quintiles.
   • This encoding is driven by physical binding, not just shared functional annotations (GO terms).
3. SV5–SV7 (Regulatory Logic): Encodes Transcriptional Regulation.
   • A compact 6D subspace (SV2–SV7) reliably distinguishes Transcription Factors (TFs) from target genes (AUROC = 0.744).
   • Depth-Dependent Encoding:
     • Early Layers (L0–L3): Encode specific regulatory edges (e.g., "STAT3 regulates BCL2") independent of co-expression.
     • Deep Layers (L4+): Compress this into coarse categorical distinctions (e.g., "TF vs. Target"), losing specific edge details but gaining robustness.
   • Repression vs. Activation: Repression edges are geometrically more prominent and separable than activation edges, likely due to more stereotyped molecular mechanisms.

C. Dynamic Trajectories: The B-Cell Attractor

• The model captures the temporal logic of B-cell differentiation (Germinal Center reaction).
• Convergence: Master regulators recruited during differentiation (BATF, BACH2) start far from the B-cell identity anchor (PAX5) at Layer 0 but converge geometrically toward PAX5 as depth increases.
• Orthogonality: Germinal Center programs and Plasma Cell programs diverge into nearly perpendicular directions by the final layer, reflecting alternative cell fate outcomes.
• Metabolic Isolation: The repressor BCL6 remains geometrically isolated in a metabolic compartment, reflecting its dual role in metabolism and immunity.

D. Negative Findings (What the model does not encode)

• Topological Features: Persistent homology signals disappeared under degree-preserving graph rewiring nulls.
• Cross-Model Alignment: While PPI encoding is consistent between scGPT and Geneformer, the dynamic "B-cell attractor" trajectory is absent in Geneformer's pre-contextual embeddings, suggesting it requires contextual processing.
• Functional Programs: SV2 poles do not encode specific Gene Ontology Biological Process terms, only compartment identity and network co-membership.

5. Significance and Implications

• Interpretability: This work proves that biological transformers learn interpretable internal models of cellular organization, moving beyond "black box" predictions.
• Regulatory Network Inference: The discovery that early-layer residual geometry encodes co-expression-independent regulatory edges provides a new method for inferring gene regulatory networks (GRNs) that outperforms attention-based methods.
• Drug Target Prioritization: The monotonic relationship between geometric proximity and PPI strength suggests that spectral axes can be used to prioritize drug targets and predict novel protein interactions without explicit database lookups.
• Model Auditing: The spectral axes serve as "biological readouts" to audit model quality. If a model fails to encode the secretory pathway on SV1 or PPIs on SV2, its internal biological logic is flawed.
• Layer-Specific Engineering: Downstream applications should not default to final-layer embeddings. Early layers are optimal for regulatory edge inference, while deeper layers are better for cell-type classification.

In summary, the paper establishes that scGPT organizes biological knowledge into a multi-dimensional coordinate system where geometry directly maps to subcellular localization, physical interactions, and regulatory logic, offering a new paradigm for extracting biological insights from foundation models.