From Words to Amino Acids: Does the Curse of Depth Persist?

1. Problem Statement

Recent analyses of Large Language Models (LLMs) have identified the "Curse of Depth," a phenomenon where deeper layers in autoregressive transformers contribute minimally to final predictions, with most task-relevant computations occurring in earlier or middle layers. Later layers primarily serve to incrementally refine the output distribution rather than generate new information.

The central question of this work is whether this depth inefficiency also exists in Protein Language Models (PLMs). In contrast to LLMs, PLMs frequently utilize:

• Non-autoregressive objectives: Such as Masked Language Modeling (MLM) and Discrete Diffusion.
• Multimodal inputs: Incorporating protein structure and function alongside sequence.
• Different scaling paradigms: Including Mixture-of-Experts (MoE) architectures.

The authors investigate whether the "Curse of Depth" is a universal property of deep transformer-based models or whether the unique constraints of protein modeling (biological structure, evolutionary constraints) influence how depth is utilized.

2. Methodology

The authors conducted a systematic depth analysis across 7 PLM families comprising 26 model variants, covering a broad spectrum of sizes (8M to 15B parameters) and training objectives (Autoregressive, Masked, Diffusion, and Multimodal).

They employed a unified framework combining three types of measurements:

A. Intervention Analysis (Layer Skipping)

Adapting the "skiplayer" method from LLM research, the authors measured the propagated effect of skipping specific layers:

• Autoregressive models: Skipped layers for early tokens and measured the impact on future token predictions.
• Masked/Diffusion models: Skipped layers on a subset of masked tokens and measured the impact on remaining non-intervened masked tokens.
• Metric: The relative change in downstream hidden states and output norms.

B. Representation Similarity Analysis

• Layer-to-Layer Similarity: Calculation of Linear CKA (Centered Kernel Alignment) between representations of different layers to quantify how strongly information changes as it propagates through the network.
• Cross-Modal Similarity: For multimodal models (ESM3, DPLM2), they calculated the Cosine Similarity between sequence-based and structure-based representations at corresponding layers to understand how modalities align and diverge.

C. Evolution of Output Distribution

Using LogitLens and TunedLens, the authors extracted the implied token prediction distribution at each layer.

• Metrics: Measurement of KL-Divergence between distributions of middle layers and the final output, as well as Top-1 Overlap (agreement on the most probable token).

D. Downstream Utility Evaluation

To link intrinsic measurements with practical performance, they evaluated layer-wise performance on ProteinGym, a large-scale benchmark for zero-shot prediction of mutation effects (217 Deep Mutational Scanning assays). They measured the Spearman rank correlation between predictions made by Early-Exit layers and experimental data.

3. Key Contributions

1. First systematic depth study in PLMs: The paper provides the first comprehensive analysis of depth utilization across diverse PLM families, objectives, and modalities.
2. Unified framework: Development of a consistent methodology for comparing autoregressive, masked, diffusion-based, and multimodal models, proving that depth inefficiency is not limited to next-token prediction.
3. Discovery of generalized depth inefficiency: Demonstrated that the "Curse of Depth" is a common feature of modern PLMs, where a subset of layers (often middle) captures most downstream-relevant signals, while later layers offer diminishing marginal returns.
4. Modality-specific insights: Uncovered different depth profiles in multimodal models (particularly ESM3) and explained their behavior through cross-modal alignment dynamics.

4. Key Results

A. The Curse of Depth Persists in PLMs

Across all model families (ESM2, DPLM, ProGen, etc.), the authors observed a consistent pattern:

• Early to middle layers: Skipping these layers causes significant disruption in downstream computations and drastically reduces downstream performance.
• Late layers: Skipping a large proportion of the final layers (e.g., the last 30–50%) leads to minimal changes in downstream representations and only marginal losses in ProteinGym performance.
• Scaling effect: As model size increases, this inefficiency becomes more pronounced. In large models (e.g., ESM2-15B), the "knee" where performance saturates occurs earlier in relative depth.

B. Refinement of Output Distribution

• TunedLens analysis: The KL-Divergence between distributions of middle layers and the final output steadily decreases with depth.
• Stability: The Top-1 prediction often stabilizes well before the final layer. Later layers primarily refine the confidence (probability mass) of the prediction rather than changing the predicted token itself.

C. Multimodal and Diffusion Models

• Generalization: The pattern of depth inefficiency holds for diffusion models (DPLM, DPLM2) and multimodal models (ESM3, DPLM2) when processing purely structure-based or combined sequence-structure inputs.
• ESM3 Anomaly: ESM3 exhibits a distinct depth profile. Unlike other models where early layers are critical, ESM3 shows weak propagated effects in early layers, with stronger contributions occurring in middle to late layers.
  • Explanation: Cross-modal similarity analysis revealed that the first ~60% of ESM3 layers serve the alignment of sequence and structure representations in a shared space. Once aligned, the later layers perform task-specific computation.

D. Sparse Architectures (MoE)

• ProGen3: As a sparse Mixture-of-Experts model, ProGen3 showed a less pronounced "Curse of Depth" compared to dense models. The distinction between high- and low-effect layers did not sharpen as strongly with scaling, suggesting that sparsity can mitigate depth inefficiency.

5. Significance and Implications

• Architectural Efficiency: The results indicate that current deep PLMs are computationally inefficient. A significant portion of inference costs (FLOPs) is spent on layers that contribute little to the final output.
• Future Directions: The paper motivates the development of:
  • Adaptive Computation: Early-exit mechanisms or dynamic depth policies that skip redundant layers during inference.
  • Growing/Looping Architectures: Models that expand depth during training or reuse layers (looping) to improve utilization.
  • Sparse Models: The success of ProGen3 suggests that MoE architectures could be a more efficient path for scaling protein models.
• Multimodal Design: The distinct behavior of ESM3 underscores that in multimodal settings, early depth may be consumed by modal alignment rather than task computation, suggesting future models could benefit from specialized "alignment" modules followed by task-specific "computation" stacks.

In summary, the "Curse of Depth" is not unique to autoregressive LLMs but is a fundamental feature of modern deep transformer-based protein models, regardless of their training objective or modality. This insight calls for a paradigm shift towards depth-efficient protein AI architectures.