Technical Summary: Dominant-Layer ZO

Problem Statement

Zeroth-order (ZO) optimization has emerged as a memory-efficient alternative for fine-tuning Large Language Models (LLMs), enabling adaptation using only forward passes and avoiding the memory overhead of backpropagation. While methods like MeZO have demonstrated that LLMs can be fine-tuned with near-inference-level memory, the internal mechanics of how useful adaptation is distributed across the model's architecture remain unclear. Existing ZO approaches largely treat fine-tuning as a full-model process, leaving a fundamental question unanswered: under forward-only updates, where does useful optimization actually occur within LLM architectures?

Methodology

The authors propose a systematic investigation into the layer-wise behavior of ZO fine-tuning, leading to the discovery and exploitation of a "Dominant Layer."

1. Empirical Discovery of the Dominant Layer

Through a layer-wise isolation study on LLaMA2-7B and Qwen3-8B across multiple downstream tasks (classification, multiple-choice, and generation), the authors observed that:

• Uneven Adaptation: Most layers provide little to no improvement over a no-fine-tuning baseline when updated individually under ZO.
• Single-Layer Dominance: A single specific layer consistently achieves performance comparable to, or exceeding, full-model ZO fine-tuning.
• Task-Agnostic but Model-Specific: For a given model, the same layer dominates across different tasks. However, the index of this layer varies between model families (e.g., Layer 1 for LLaMA2-7B vs. Layer 6 for Qwen3-8B).
• Contrast with First-Order (FO): In matched FO fine-tuning, improvements are evenly distributed, and no single layer dominates, indicating this phenomenon is unique to ZO optimization.

2. Identification via Activation Outliers

To identify the dominant layer without expensive layer-wise fine-tuning, the authors leverage the phenomenon of activation outliers—specific dimensions with extremely large magnitudes that appear consistently across inputs.

• Observation: The dominant layer consistently aligns with the first layer in the residual stream where activation outliers emerge.
• Selection Method: A simple inference-only analysis is proposed. Given a small calibration set, the pre-trained model is run forward to measure layer-wise activation statistics. The first layer exhibiting a clear outlier pattern (abnormally large maximum activation magnitude) is selected as the dominant layer for fine-tuning.

3. Theoretical Explanation

The paper explains why this layer is critical for ZO optimization:

• Perturbation Sensitivity: ZO estimates updates based on the difference in final loss caused by random parameter perturbations. The dominant layer induces the largest loss changes under perturbation.
• Residual Stream Propagation: Because the dominant layer appears early in the model and aligns with the first activation-outlier layer, perturbations injected here enter the residual stream and propagate through all subsequent decoding layers.
• Signal Accumulation: These perturbations are repeatedly transformed and accumulated as they pass through the network, resulting in a disproportionately large and stable change in the final loss. This provides a strong optimization signal for forward-only updates, unlike later layers where perturbations may induce loss changes but lack the propagation depth to form stable, useful updates.

Key Contributions

1. Discovery: Identification of a "dominant-layer" phenomenon where tuning a single layer recovers or exceeds the performance of full-model ZO fine-tuning.
2. Efficient Identification: Demonstration that the dominant layer is model-specific and can be identified prior to training using a lightweight, inference-only analysis of the first activation-outlier layer.
3. Mechanistic Insight: Explanation that the dominance arises from the combination of high perturbation sensitivity and early placement in the residual stream, allowing perturbation effects to accumulate across the network.
4. Validation: Extensive experiments on LLaMA2-7B and Qwen3-8B across nine benchmarks.

Experimental Results

Experiments were conducted on LLaMA2-7B and Qwen3-8B using datasets including SST-2, RTE, CB, BoolQ, WSC, MultiRC, COPA, SQuAD, and DROP.

• Performance:

  • LLaMA2-7B: Dominant-layer ZO fine-tuning improved the average score by 0.57% over full-model MeZO and 0.09% over MeZO LoRA. Notable gains were observed on RTE (+2.17%) and SQuAD (+2.32% for LoRA variant).
  • Qwen3-8B: Dominant-layer ZO improved the average score by 1.15% over full-model MeZO and 1.12% over MeZO LoRA, with significant gains on CB (+3.57%) and SST-2 (+2.04%).
  • Comparison: The method outperformed Sparse-MeZO (which uses element-wise sparse masking) by 0.13% on average, suggesting layer-level selection is as critical as parameter-level selection.

• Training Efficiency:

  • By restricting updates to a single layer (e.g., 1 out of 32 in LLaMA2-7B), the method achieved significant speedups in parameter-side operations.
  • Perturbation time was reduced by 27.34×–31.28×, and update time by 31.56×–32.71×.
  • End-to-End Speedup: Ranged from 1.12× (on long-input tasks like DROP) to 4.52× (on short-input tasks like COPA), depending on the proportion of time spent on forward passes versus parameter operations.

• Ablation Studies:

  • Adding a second layer (e.g., a later sensitive layer) did not consistently improve performance, suggesting the bottleneck is not the number of updated layers but the specific propagation properties of the dominant layer.
  • Within the dominant layer, fine-tuning only the MLP channels associated with activation outliers recovered most of the performance gain, confirming the critical role of these specific channels.

Significance and Claims

The paper claims that full-model ZO fine-tuning does not simply update too many parameters; rather, it allocates optimization effort unevenly, with the majority of useful adaptation concentrated in a single layer.

• Practical Implication: ZO fine-tuning can significantly reduce training costs (memory and time) by focusing on this single dominant layer while preserving, and sometimes improving, full-model performance.
• Conceptual Implication: The findings provide insight for future ZO method design, suggesting that explicitly considering where useful updates arise (layer identity) and within layers (outlier channels) is more effective than uniform full-model updates.
• Limitations: The authors acknowledge a remaining performance gap between Dominant-layer ZO and First-order fine-tuning. They also note that the study was limited to specific models and optimizers, and future work could explore combining this approach with other ZO optimizers (e.g., ZO-AdaMM, FZOO) or more model families.