🧬 biology

Rotation, rescaling, or replacement? Decomposing organ adaptation in single-cell transformer representations

By leveraging three scGPT checkpoints with identical architectures but different pre-training corpora, this study demonstrates that organ-specific adaptation in single-cell transformers is minimal and concentrated in specific attention layers rather than global geometric transformations, suggesting that models trained on fewer than ten million cells largely retain their initialization noise and that general checkpoints are superior for extracting biological algorithms.

Original authors: Liu Chen

Published 2026-08-06

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Liu Chen

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are a detective trying to figure out how a super-smart robot learns to understand the world. In the field of biology, scientists are building these robots—called "transformers"—to read the instruction manuals inside our cells. These manuals are written in a language of genes, and the robots are trained on massive libraries of cell data to understand how different parts of the body work. The big question is: when we teach a robot about a specific organ, like a brain or a kidney, does it simply learn to look at the same old facts from a new angle, or does it completely rewrite its understanding? This matters because if the robot just "rotates" its view, we can easily translate what it learns about brains to help us understand kidneys. But if it "replaces" its entire worldview, we have to start over every time we switch organs.

This paper investigates that exact mystery using a special set of three robot brains created by the scGPT project. The researchers had a unique advantage: they had three versions of the same robot, built with the exact same blueprints and starting from the exact same random "seed" of numbers. The only difference was their training library: one read 33 million cells from the whole human body, one read 13.2 million brain cells, and one read 814,000 kidney cells. By comparing these three, the author could see exactly what the data taught the robots versus what was just random noise left over from their birth.

The Great "What Did You Learn?" Experiment

Think of these three AI models as three students who started school on the same day, sitting in the same seats, with the same blank notebooks and the same random scribbles in the margins (the "initialization").

Student A read a massive encyclopedia of 33 million pages covering every part of the human body.
Student B read a thick textbook of 13.2 million pages, but only about the brain.
Student C read a thin pamphlet of 814,000 pages, but only about the kidney.

The researchers wanted to know: When Student B and Student C finished their reading, did they just tilt their heads to look at the same facts differently (a rotation), or did they change the size of the words they cared about (a rescaling)? Or, did they throw out the old facts and write entirely new ones (a replacement)?

The "Ghost" in the Machine

Before answering the big question, the researchers found a spooky clue. They noticed that thousands of specific lines in the notebooks of all three students were bit-for-bit identical. These were lines that no student had ever actually read or learned from. Because the notebooks started with random scribbles, and these lines never changed, the researchers realized: "Aha! All three students started with the exact same random scribbles!"

This was a game-changer. It meant the researchers knew exactly what "noise" looked like. They could draw a line in the sand: anything below that line was just the random starting scribbles; anything above it was real learning.

The Shocking Results: It's Not a Rotation, It's a Rewrite

Once they filtered out the random scribbles, the answer to their main question was clear and surprising.

1. The "Kidney" Student Didn't Really Learn Much
The student who read only the kidney pamphlet (814,000 cells) barely changed their notebook at all. Out of 512 possible directions they could learn, only 12 were actually new. The rest was just the original random scribbles. It's like trying to learn a new language by reading a single comic book; you don't really know the language yet. The researchers found that for this student, the "learning" was so weak that it was mostly just noise.

2. The "Brain" Student Changed the Map, Not Just the Angle
The student who read the brain textbook (13.2 million cells) did learn more—about 104 new directions. But here is the twist: they didn't just rotate the map or make the existing roads bigger. They replaced the roads.

The researchers tried to fit the brain student's map onto the whole-body student's map using a "rotation" (turning the map) or a "rescaling" (stretching the map). It didn't work well.

A simple rotation explained less than 30% of the changes.
Even adding a "stretch" for every single road only explained a tiny bit more.
To make the maps match, they had to use a complex, messy linear map that explained only about 57% of the changes.

This means that when the robot learns about a specific organ, it doesn't just look at the same biological facts from a different angle. It discards most of the old "axes" (the main ways it organizes information) and writes new ones. The brain student and the kidney student are speaking different dialects of the same language, not just the same dialect with a different accent.

3. The "Middle" is Where the Magic Happens
The researchers also looked at where in the robot's brain the learning happened. They found that the learning was concentrated in the middle layers of the robot's network, specifically in the parts that pay attention to connections (attention) and the parts that process information (feed-forward). The "normalization" parts (which keep the robot steady) barely changed at all. It's as if the student rewrote the middle chapters of their textbook but left the introduction and conclusion exactly as they were when they started.

4. Direction Matters More Than Size
When the robot learned about the brain versus the kidney, the size of the change didn't tell them which organ it was. A gene that changed a lot could be from anywhere. But the direction of the change was a dead giveaway. If a gene moved in a specific direction, it was almost certainly a brain gene. If it moved the opposite way, it was a kidney gene. The robot learned to recognize organs by which way the information shifted, not by how much it shifted.

The Takeaway for Future Explorers

The paper offers a few very practical rules for anyone trying to use these AI robots to understand biology:

Stick to the Generalist: If you want to understand biology, use the model trained on the whole human body (the 33 million cell model). It has the most "learned" directions and the clearest structure.
Don't Assume Transferability: You cannot simply take a discovery made in the "brain" model and apply it to the "kidney" model by just turning a knob. The internal maps are too different. If you want to use the brain model's findings for kidneys, you have to re-learn them from scratch.
Watch Out for Small Datasets: If a model is trained on fewer than about 10 million cells (like the kidney model in this study), it might not have learned anything real at all. It might just be echoing its own random starting scribbles. Before trusting a small organ-specific model, you should check if it has actually learned anything beyond the noise.

In short, the robot doesn't just "turn" its head to see a new organ; it has to grow a new head. And if it doesn't have enough time to read (enough data), it might not grow a new head at all—it might just be staring blankly at the random scribbles it started with.

Technical Summary: Decomposing Organ Adaptation in Single-Cell Transformer Representations

Problem Statement
Single-cell foundation models, such as scGPT, are increasingly used to extract biological insights from their internal representations. However, a critical assumption in this field remains untested: that the geometry of a model's internal state reflects the biology of its training corpus rather than the constraints of the architecture, initialization scheme, or optimizer. In natural language processing, separating these factors is difficult because public checkpoints rarely share identical architectures and objectives while differing only in corpus. The single-cell field offers a unique controlled experiment: the scGPT release includes three checkpoints (whole-human, brain, and kidney) that share identical architecture (12 layers, 8 heads, 512 dimensions), objectives, vocabulary, and optimizer settings, differing only in their pre-training corpora (33M whole-human cells, 13.2M brain cells, and 814K kidney cells). This paper investigates how organ adaptation manifests in these representations: does it rotate existing biological axes, rescale them, or replace them entirely?

Methodology
The author employs a "common-origin divergence" approach, leveraging the fact that the three checkpoints were initialized from the same random draw.

Initialization Noise Calibration: By identifying 4,437 gene rows that are bit-identical across all three checkpoints (genes never seen in any training corpus), the author establishes that these rows retain the exact initialization values. These values follow a uniform distribution on $[-0.1, 0.1]$ , creating a known isotropic noise floor. Using Random Matrix Theory (Marchenko–Pastur distribution), they define a precise detection threshold: singular values falling within the predicted band are indistinguishable from initialization noise, while those above represent learned signal.
Nested Decomposition of Drift: The author compares gene-embedding matrices ( $X$ and $Y$ ) using a hierarchy of linear maps to determine how well one checkpoint's geometry can be transformed into another's. The models tested include identity, translation, isotropic scaling, orthogonal rotation (Procrustes), per-axis scaling, and full linear maps. The variance explained by each step isolates the contribution of rotation, rescaling, and gene-specific displacement.
Spectral and Layer-wise Analysis: The study analyzes the conservation of spectral directions (singular vectors) between models and maps the location of adaptation across the 163 weight tensors of the transformer stack, distinguishing between learned displacements and retained initialization energy.
Biological Validation: Tissue specificity is assessed using the Human Protein Atlas (HPA) and Tabula Sapiens. The author correlates geometric drift (direction and magnitude) with tissue-specific gene expression and pathway coherence (Gene Ontology, Reactome, Hallmark).

Key Contributions

Noise-Floor Calibration: The paper provides the first exact noise model for single-cell transformer representations by exploiting shared initialization across checkpoints, allowing for a rigorous separation of learned signal from initialization artifacts.
Geometric Characterization of Adaptation: It demonstrates that organ adaptation is not a simple rotation or rescaling of a shared biological subspace but rather a replacement of axes, with only a minority of directions surviving the transition between corpora.
Quantification of Learning: It quantifies the "learned dimensionality" of different corpora, revealing that smaller datasets (e.g., 814K cells) result in models where the vast majority of the embedding space remains unlearned initialization noise.

Results

Dominance of Initialization: Approximately 80% of the initialization variance persists in all three gene-embedding matrices. The number of "learned directions" (singular values exceeding the noise floor) is 119 for whole-human, 104 for brain, and only 12 for kidney.
Replacement, Not Rotation: A similarity transform (rotation + isotropic scale) explains only 48% of the raw drift between whole-human and brain models. When restricted to the learned subspaces (removing initialization noise), this drops to 29% for brain and 8% for kidney. Adding per-axis scaling contributes negligibly (0.3 percentage points). The primary driver of drift is gene-specific displacement that no global linear map can absorb.
Spectral Conservation: Conservation of learned directions decays monotonically with spectral index. The top 10 learned directions of the whole-human model retain ~48% of their mass in the brain model's learned subspace, but this drops to <20% for directions beyond index 50.
Direction vs. Magnitude: At the gene level, the direction of displacement encodes tissue identity, not the magnitude. The leading brain–kidney contrast axis correlates strongly with brain-versus-kidney expression enrichment ( $\rho = 0.246$ ), whereas displacement magnitude shows negligible correlation with tissue specificity.
Layer-wise Adaptation: Adaptation concentrates in the attention and feed-forward projection matrices of the middle layers. LayerNorm gain vectors remain effectively untouched, retaining 99% of their initialization energy.
Data Volume Threshold: Only the brain checkpoint (13.2M cells) acquires a genuinely organ-specific axis (over-encoding neural gene sets). The kidney checkpoint (814K cells) shows no renal enrichment and fails to learn organ-specific structure, behaving largely as unlearned initialization.

Significance and Claims
The paper argues that for researchers extracting interpretable algorithms from foundation models, the choice of checkpoint is a first-order methodological decision.

Use General Checkpoints: The general (whole-human) checkpoint is the superior substrate for extraction, offering the most learned directions and strongest pathway structure.
Do Not Map Across Checkpoints: Findings from one organ checkpoint do not transfer to another via simple basis changes (rotation/scaling). Probes and circuits must be re-fitted for each specific checkpoint.
Focus on the Spectrum: Biological questions likely to transfer across organs are those aligned with the top of the spectral index (the leading learned directions).
Beware of Small Corpora: Checkpoints trained on fewer than roughly 10 million cells (like the kidney model) are "mostly unlearned." Analyses relying on more than a dozen embedding directions in such models are likely analyzing random initialization noise rather than biological signal.

The author concludes that organ adaptation replaces biological axes rather than refining them, and that the "noise floor" of initialization must be explicitly accounted for to avoid misinterpreting random structure as biological conservation.

The Great "What Did You Learn?" Experiment

The "Ghost" in the Machine

The Shocking Results: It's Not a Rotation, It's a Rewrite

The Takeaway for Future Explorers

Technical Summary: Decomposing Organ Adaptation in Single-Cell Transformer Representations

More like this