Technical Summary: A Geometric Atlas of ESM3 Modality Organization

Problem Statement

Protein language models (PLMs) have advanced the prediction of protein structure and function by learning general-purpose representations from large datasets. Multimodal models like ESM3 ingest multiple input channels simultaneously—including amino acid sequence, 3D structure, secondary structure (SS8), solvent accessibility (SASA), and discrete functional annotations—by summing their embeddings into a single residual stream. While previous interpretability work has examined attention heads and learned features in sequence-only models, little is known about the mechanistic organization of these distinct input modalities within a multimodal architecture. Specifically, it remains unclear whether these modalities occupy separate subspaces, at what depth they fuse, and whether this fusion is an architectural artifact or a learned property.

Methodology

The authors employ a "geometry-first" descriptive approach to create an atlas of modality organization across the depth of ESM3 (specifically the esm3-sm-open-v1 model: 1.4 billion parameters, 48 transformer layers, 1536-dimensional embedding).

Experimental Design:

• Isolation and Aggregation: For each protein in the dataset, the model is run in isolation for each of the five single-modality conditions (sequence, structure, SS8, SASA, function) and one "all-modalities" condition.
• Data Harvesting: The residual stream is read out at every layer (1–48). Representations are analyzed both as mean-pooled protein-level vectors and, for a subset, as individual residue-level activations.
• Datasets: Analysis covers an 892-protein human set for primary results and a diverse set of 5,555 proteins from 12 organisms (spanning Eukaryota, Bacteria, and Archaea) to test universality. A control set of 177 proteins with experimental (X-ray/cryo-EM) structures validates findings against AlphaFold-predicted coordinates.
• Metrics:
  • Silhouette Score: Measures the separation between modality conditions at each layer.
  • Centered Kernel Alignment (CKA): Tracks alignment between pairs of conditions to determine fusion timing.
  • Integration Index: The mean pairwise CKA across all physical modalities.
  • Effective Rank: Measures the dimensionality of the variance within the representation cloud.
  • Linear Probes: Logistic probes trained to decode modality identity from the stream.
• Controls: The study includes a randomly initialized model (trained weights reset in the trunk and embeddings, preserving the structure encoder) to distinguish learned fusion from architectural summation artifacts.

Key Results

1. Sharp, Ordered Fusion of Physical Modalities

The four physical modalities (sequence, structure, SS8, SASA) begin in distinct subspaces at the input. They remain maximally separated through the first half of the network, reaching peak separation (Silhouette score ~0.42) around layer 24. A sharp transition occurs between layers 25 and 35, where these modalities collapse into a shared, low-dimensional subspace.

• Order of Fusion: The fusion is ordered. Structure-derived modalities (structure, SS8, SASA) are mutually aligned from the input (Layer 0). Sequence, which carries information not derivable from structure, joins the fusion last, integrating significantly after layer 28.

2. Persistent Orthogonality of Functional Annotations

Unlike the physical modalities, the functional-annotation modality never fuses with the physical subspace.

• Orthogonality: Pairwise CKA between the function channel and any physical modality remains near 0.01–0.07 across all 48 layers.
• Granularity Independence: This separation holds even when using per-residue functional annotations (derived from InterPro) rather than whole-protein annotations, indicating the separation is content-driven rather than an artifact of tokenization granularity.
• Information Retention: Despite orthogonality, functional information is not lost; linear probes can decode enzyme classes and fold classes from the fused physical subspace with high accuracy, suggesting the network extracts functional content into the physical subspace while keeping the explicit function channel on a separate, thin axis.

3. Fusion is a Learned Property

A randomly initialized model with the same architecture shows no depth-dependent fusion. Condition separation remains high and constant, and the integration index does not change with depth. This confirms that the observed fusion is a result of training, not merely the mathematical consequence of summing embeddings.

4. Variance Reorganization Without Isotropy

Fusion does not result in a collapse to a single point or a diffusion into isotropy.

• Effective Rank: The effective rank of the representation cloud collapses to ~3 at the peak separation (layer 24), expands to ~85 dimensions in the fusion zone (layers 25–35), and then contracts.
• Variance Shift: Fusion shifts variance from between conditions to within conditions. Modality identity remains decodable (accuracy ~1.0 in full space) even after fusion, indicating a controlled reorganization rather than a loss of information.

5. Universality and Structural Dependencies

• Universality: The fusion pattern is universal across diverse organisms. Every superkingdom (Eukaryota, Bacteria, Archaea) and every individual organism in the study reaches peak fusion (minimum separation) at exactly layer 35.
• Structural Dependence: Fusion depth is independent of protein length but is delayed by structural disorder. Proteins with higher coil fractions fuse later, while well-folded helical proteins fuse earlier. The effect is not correlated with AlphaFold confidence scores (pLDDT).

Significance and Claims

The paper claims to provide the first comprehensive geometric atlas of how multimodal information is organized in a protein language model. The primary significance lies in revealing that ESM3 organizes its inputs into two distinct regimes:

1. A unified physical regime where sequence and structural features merge into a shared, low-dimensional subspace at mid-depth, suggesting the network builds modality-specific features before committing to a unified physical representation once secondary structure is resolved.
2. A separate functional regime where discrete functional labels are held on an orthogonal axis throughout the network, regardless of whether that information is redundant with the physical modalities.

The authors conclude that this depth-locked organization is intrinsic to the trained model rather than an artifact of specific proteomes or prediction coordinates. The findings suggest that the network's "reasoning" involves a specific temporal ordering of integration, where structural consistency is established before sequence information is fully integrated, while functional labels remain distinct. This work establishes a baseline for mechanistic interpretability in multimodal biological models, demonstrating that fusion is a learned, universal, and ordered phenomenon.