🧬 biology

In Search of Manifolds Inside Protein Language Models: A Largely Negative Report

This paper systematically investigates the manifold hypothesis in protein language models (pLMs) using a comprehensive battery of geometric and topological diagnostics, concluding that unlike single-cell foundation models, pLM representations are predominantly high-dimensional and linear rather than organized around low-dimensional manifolds.

Original authors: Olivia Denvis

Published 2026-07-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Olivia Denvis

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are trying to understand a massive, chaotic library. In this library, every single book is a protein, the tiny molecular machines that build and run all living things. For a long time, scientists have been using super-smart computer programs, called "protein language models," to read these books. These programs are like digital detectives that have read billions of protein sequences and learned to predict how they fold, what they do, and how they might change.

But here is the big question: How does the computer actually "think" about these proteins? When the computer looks at a protein, does it see a messy pile of data, or does it see a hidden, smooth shape? Scientists have a favorite idea called the "manifold hypothesis." Think of it like this: imagine a giant, crumpled piece of paper floating in a huge 3D room. Even though the paper is floating in 3D space, if you were an ant walking on it, the world would feel flat and two-dimensional. The "manifold hypothesis" suggests that even though the computer's brain is huge and complex (high-dimensional), the important information about proteins might actually be hiding on a smooth, low-dimensional surface, like that flat piece of paper. This idea has been a hit in other fields, like studying how cells grow and change, where scientists found these smooth, low-dimensional "roads" that cells travel along. So, the big question is: Do these protein-detecting computers also have these smooth, hidden roads inside their brains?

A researcher named Olivia Denvis decided to go on a treasure hunt to find these hidden roads inside protein language models. She didn't just guess; she built a massive "detection battery," which is like a super-advanced metal detector and map-maker rolled into one. She tested this tool on several different protein models, including the famous ESM-2 family (which comes in sizes from tiny 8 million parameters to a giant 3 billion) and others like ProtBERT and ProtT5. She used a variety of clever tricks to see if the data was actually sitting on a smooth, low-dimensional surface. She checked if the data had a specific "intrinsic dimension" (a fancy way of asking: how many directions do we really need to describe this?), she tried to flatten the data onto 2D maps to see if patterns emerged, and she even looked for topological loops (like the hole in a donut) that would prove the data was shaped like a specific object.

The result of this hunt? A largely negative report. The treasure hunt came up empty.

Denvis found that the idea of a smooth, low-dimensional road inside these protein models is probably wrong. Instead of finding a flat sheet of paper floating in a 3D room, she found that the data is actually spread out across a vast, high-dimensional space. When she measured the "intrinsic dimension" (the number of directions needed to describe the data), it wasn't a small, manageable number like 2 or 3. For the medium-sized model (ESM-2 650M), the dimension was around 63, and for the giant 3-billion-parameter model, it jumped to 88. Even more telling, as she made the models bigger and smarter, the dimension got bigger, not smaller. If there were a single, smooth road, the dimension should have stayed the same or gotten simpler as the models improved. Instead, the models seemed to be using more and more directions to store information, suggesting the structure is complex and "high-dimensional" rather than a simple curve.

She also tried to use the computer's "map-making" tools to flatten the data down to 2D, just like we do with single-cell data. But when she did this, the useful information got destroyed. It was like trying to flatten a complex, multi-layered cake into a pancake; the flavor (the biological information) got lost. When she tested these flattened maps to see if they could predict things like protein stability or structure, they performed terribly. In fact, the simplest method—just looking at the raw, high-dimensional data with a straight line (a linear probe)—worked the best. The computer didn't need a curved, low-dimensional road; it needed a wide, flat highway with dozens of lanes.

Furthermore, she looked for "loops" or holes in the data, which would indicate a specific shape like a donut or a circle. She found none. The data was topologically "trivial," meaning it was just a big, solid blob without any interesting shapes. She also discovered that the clusters of proteins that looked like they were forming neat groups on 2D maps were actually just an illusion. When she removed the influence of simple things like how long the protein was or what basic ingredients (amino acids) it was made of, the groups fell apart. The "manifold" people thought they saw was just a reflection of these basic, boring facts, not a deep, hidden structure.

To make sure her tools weren't broken, she tested them on things she knew had these smooth roads, like a synthetic "Swiss roll" shape and real data from single cells. Her tools worked perfectly there, finding the low dimensions and the loops. This proved that her methods were good; the problem wasn't the tools, it was the protein data itself.

So, what does this mean? It turns out that protein language models are not like the smooth, continuous maps of cell development. Instead, they are more like a vast, high-dimensional library where every book is organized by a complex mix of length, composition, and family history. The information is there, and it's incredibly useful, but it's not hiding on a simple, low-dimensional surface. It's spread out across a high-dimensional space, and trying to force it into a simple 2D map actually throws away the most important details. The paper concludes that for protein models, we should stop looking for smooth, curved roads and start appreciating the wide, high-dimensional highways they actually travel on.

Technical Summary: In Search of Manifolds Inside Protein Language Models

Problem Statement
The manifold hypothesis posits that high-dimensional data concentrate near a low-dimensional, smoothly varying manifold. This framework has successfully organized the interpretation of neural representations in single-cell foundation models, where continuous biological processes (e.g., differentiation, cell cycles) manifest as recoverable low-dimensional structures and topological loops. However, it remains an open question whether protein language models (pLMs), which are trained on discrete, combinatorial sequence spaces, internalize analogous low-dimensional manifolds. The authors investigate whether the high-dimensional activation spaces of pLMs admit a global, exploitable low-dimensional structure or if the "manifold" metaphor is inappropriate for protein sequence data.

Methodology
The authors assembled a comprehensive "manifold-detection battery" and applied it uniformly to seven protein language models: the ESM-2 family (ranging from 8M to 3B parameters), ProtBERT, and ProtT5. The analysis utilized per-residue and mean-pooled sequence-level embeddings from 40,000 sequences drawn from UniRef50, with homology controlled via MMseqs2 clustering.

The detection battery comprised four diagnostic families:

Intrinsic Dimension (ID) Estimation: Using TwoNN, Maximum Likelihood Estimators (MLE), and PCA participation ratios to determine if dimensionality is stable and significantly lower than ambient width.
Nonlinear Manifold Learning: Applying Isomap, LLE, diffusion maps, and UMAP to assess whether projecting data onto low-dimensional coordinates preserves local neighborhood structure (measured by trustworthiness and geodesic connectivity).
Topological Data Analysis: Computing persistent homology (Vietoris–Rips) to detect robust topological features (e.g., loops) beyond the noise floor, validated against bootstrap null models.
Confound Controls: Regressing out sequence length, amino-acid composition, hydrophobicity, and homology to determine if apparent clustering is driven by trivial covariates.

Crucially, the battery was validated on positive controls with known manifold structures: a Swiss roll (2-manifold), a single-cell cell-cycle dataset (topological loop), and a synthetic torus. Downstream performance was evaluated by comparing linear probes on raw embeddings against probes trained on nonlinear manifold coordinates across four tasks: secondary structure prediction, remote homology classification, stability regression, and subcellular localization.

Key Results
The study yields a largely negative conclusion regarding the existence of low-dimensional manifolds in pLMs:

High and Unstable Intrinsic Dimension: The intrinsic dimension of pLM representations is high in absolute terms (e.g., TwoNN $\approx$ 63 for ESM-2 650M) and grows monotonically with model size (from 22 for 8M to 88 for 3B). Unlike a true manifold, the estimated dimension does not stabilize across neighborhood scales; it drifts upward as the neighborhood size ( $k$ ) increases.
Failure of Nonlinear Coordinates: Nonlinear manifold coordinates never outperform linear probes on raw embeddings. In fact, forcing embeddings into low-dimensional nonlinear spaces (e.g., 2D UMAP) catastrophically degrades performance on downstream tasks (e.g., remote homology accuracy drops from 0.30 to 0.15). Performance saturates only when using 64–128 linear dimensions, suggesting the useful structure is moderately high-dimensional and linear.
Trivial Topology: Persistent homology recovers no robust loops ( $H_1$ features) in pLM embeddings; all features fall within the noise floor. This contrasts sharply with positive controls, where the battery successfully recovered the expected topological loops.
Confound-Driven Structure: Apparent clustering of protein families in 2D visualizations (e.g., UMAP) is largely an artifact of confounding variables. After regressing out sequence length, amino-acid composition, and homology, the family silhouette score collapses from 0.31 to 0.05 (near random chance).
Validation on Controls: The detection battery successfully identified the correct dimensions, high trustworthiness, and expected topology in the Swiss roll, cell-cycle, and torus controls, confirming that the negative results for pLMs are due to the data properties, not methodological failure.

Key Contributions

Systematic Battery: The authors provide a rigorously validated, open manifold-detection battery for transformer representations, tested on both synthetic and biological positive controls.
Negative Findings: They report a coherent set of negative results indicating that pLM representations are high-dimensional, size-dependent, and lack scale-stable manifold dimensions or robust topology.
Confound Analysis: The study demonstrates that the apparent low-dimensional organization of protein families in pLM space is primarily driven by sequence length, composition, and homology, rather than a curved geometric manifold.
Theoretical Distinction: The paper contrasts pLMs with single-cell foundation models, arguing that the discrete, combinatorial, and homology-structured nature of protein sequence space fundamentally differs from the continuous developmental processes that generate low-dimensional manifolds in single-cell data.

Significance and Claims
The authors conclude that protein language model representations are better described as high-dimensional and predominantly linear rather than as low-dimensional manifolds. They argue that the negative result is not a deficiency of the models but a faithful reflection of the underlying domain: protein sequence space is discrete and scattered across clusters defined by homology, lacking the continuous axes that would necessitate compression onto a low-dimensional manifold.

The practical implication is a caution against interpreting 2D UMAP or t-SNE plots of pLM embeddings as revealing biological manifolds; such plots largely reflect trivial covariates. For quantitative analysis, the authors suggest that high-dimensional linear tools (probes, PCA, sparse autoencoders) are more appropriate for the geometry of protein representations than nonlinear manifold learning techniques. The paper does not claim that pLMs are geometrically uninteresting, but rather that the specific, strong form of the manifold hypothesis does not hold for them.

Technical Summary: In Search of Manifolds Inside Protein Language Models

More like this