🤖 machine learning

ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces

ZO-Act is an efficient zeroth-order fine-tuning method that improves upon existing approaches by restricting perturbations to a fixed, activation-informed low-rank subspace, thereby reducing estimator variance and enabling momentum-based optimization while supporting quantized large language models.

Original authors: Xun Dong, Yibo Xu, Naigang Wang, Xin Li, Penghang Yin, Zi Yang

Published 2026-07-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xun Dong, Yibo Xu, Naigang Wang, Xin Li, Penghang Yin, Zi Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly complex library (a Large Language Model) that you want to teach a new skill, like answering questions about a specific topic. Usually, to teach this library, you'd need to send a team of editors through every single book, page by page, making notes on how to change the text. This is called "backpropagation," and it requires a huge amount of memory and energy. If the library is too big, your team runs out of space to write their notes, and the process crashes.

The Problem: The "Blind" Approach
Some researchers tried a different method called Zeroth-Order (ZO) optimization. Instead of reading every page to find the exact mistake, they just guess. They make a tiny, random change to a book, see if the story gets better, then make a different random change and see again. It's like trying to find the right key on a giant keychain by randomly jiggling keys until one works.

The problem with this "blind" guessing is that with billions of keys (parameters), the guesses are incredibly noisy and inefficient. You might jiggle a key that doesn't matter at all, wasting time and energy.

The Solution: ZO-Act (The "Smart Map" Approach)
The paper introduces ZO-Act, a smarter way to do this guessing game. Instead of jiggling any random key, ZO-Act first looks at how the library naturally reacts to a few sample questions (called "activations").

Think of it like this:

The One-Shot Map: Before you start teaching, you ask the library a few practice questions. You notice that when people ask about "history," the library's internal gears mostly spin in a specific, predictable pattern. You draw a map of just those spinning gears.
The Fixed Subspace: You decide that from now on, you will only adjust the gears that are on your map. You freeze the rest of the library so it doesn't get messed up.
The Lightweight Coach: Instead of trying to move the heavy, frozen gears directly, you attach a small, lightweight lever (a "coefficient matrix") to your map. You only need to wiggle this small lever to move the heavy gears in the right direction.

Why This is a Game-Changer

Less Noise: Because you are only wiggling a small lever instead of the whole machine, your guesses are much more accurate. It's like trying to tune a radio by adjusting just the volume knob instead of trying to rebuild the whole antenna.
Faster Learning: Since the "lever" is small, you can use standard, powerful tools (like the Adam optimizer) to wiggle it efficiently.
Works on Tiny Devices: Because the main library stays frozen and you only touch the small lever, you don't need a super-computer to do this. You can even do it on a computer with very limited memory (quantized models), which is like trying to teach the library using a tiny notebook instead of a massive encyclopedia.

What the Paper Found
The researchers tested this on huge models (like Llama-3 and OPT) and found that:

It works better than random guessing: ZO-Act consistently outperformed other "blind" methods across tasks like understanding language, answering questions, and solving logic puzzles.
It works on small computers: Even when the library was compressed to fit on a small device (INT4 quantization), ZO-Act still performed very well.
The Map is Stable: They checked if the "map" they drew at the beginning stayed useful. It did! The most important gears kept spinning in the same way throughout the whole training process, proving that the initial "one-shot" map was a good choice.

The Catch
The paper admits that while ZO-Act is great for saving memory and avoiding the need for backpropagation, it's still not quite as fast or perfect as the traditional "full editor" method (First-Order) if you have unlimited memory. It's a trade-off: you get a very efficient, low-memory solution that is "good enough" and often better than other low-memory tricks, but it still relies on guessing rather than seeing the exact answer.

In Summary
ZO-Act is like giving a blindfolded student a smart map of the most important rooms in a giant building. Instead of stumbling around randomly, they only adjust the furniture in those specific rooms using a small, easy-to-move tool. This saves energy, reduces mistakes, and lets them learn new skills even in a tiny apartment.

Technical Summary: ZO-Act

Problem Statement

Fine-tuning Large Language Models (LLMs) typically relies on first-order optimization (backpropagation), which becomes memory-prohibitive as models scale to billions of parameters due to the storage requirements for activations, optimizer states, and gradient computations. Zeroth-order (ZO) optimization offers a forward-only alternative by estimating update directions using only loss evaluations, thereby avoiding backpropagation. However, existing ZO methods face significant challenges:

High Variance: Perturbing the full parameter space (as in MeZO) results in high-variance gradient estimates, leading to slow convergence and instability.
Suboptimal Subspaces: While some methods restrict perturbations to low-dimensional subspaces (e.g., LOZO, SubZero), they often rely on randomly constructed subspaces or random perturbations that may not align with the actual gradient flow.
Optimizer Compatibility: Many ZO methods struggle to utilize standard momentum-based optimizers (like Adam) effectively because the perturbation structure does not expose explicit, lightweight trainable variables.
Quantization Constraints: Fine-tuning quantized LLMs (e.g., INT4) is difficult with methods that require materializing full-weight perturbations.

Methodology: ZO-Act

The authors propose ZO-Act, a one-shot, activation-informed ZO fine-tuning method. The core insight is that the dominant components of weight gradients in linear layers lie within the subspace spanned by the top right singular vectors of the input activation matrix.

Core Mechanism

One-Shot Initialization: For each linear layer, ZO-Act performs a single forward pass on a calibration batch to compute the input activation matrix $X$ . It then computes the top- $r$ right singular vectors ( $V_r$ ) of $X$ via a thin Singular Value Decomposition (SVD).
Fixed Subspace Parameterization: The weight update is restricted to a low-rank subspace defined by these frozen activation vectors. The effective weight update is parameterized as:
$\Delta W = V_r B$
where $V_r \in \mathbb{R}^{m \times r}$ is the frozen activation basis, and $B \in \mathbb{R}^{r \times n}$ is a lightweight, trainable coefficient matrix. The original pretrained weights $W$ and the basis $V_r$ remain frozen throughout training.
Optimization in Coefficient Space: Instead of perturbing the full weight matrix $W$ , ZO-Act samples random perturbations $Z$ in the low-dimensional coefficient space ( $Z \sim \mathcal{N}(0, I)$ ). The perturbed effective weight becomes $W_{eff} = W + V_r(B + \mu Z)$ .
Forward-Only Estimation: The gradient for $B$ is estimated using forward-difference loss evaluations:
$\hat{g}_B = \frac{L(B + \mu Z) - L(B)}{\mu} Z$
This estimated gradient is then used to update $B$ using standard first-order optimizers like Adam.

Theoretical Analysis

The paper analyzes ZO-Act as zeroth-order optimization over a restricted coefficient space $\beta$ (dimension $k$ ) rather than the full weight space (dimension $d$ ).

Variance Reduction: The variance of the ZO estimator scales linearly with the perturbation dimension. By reducing the dimension from $d$ (full weights) to $k = \sum r n_\ell$ (coefficients, where $r \ll m$ ), ZO-Act significantly reduces the variance-dependent convergence term and the finite-difference error.
Bias-Variance Trade-off: Restricting updates to the activation-informed subspace introduces a subspace approximation bias. However, the authors argue that LLM activations and gradients exhibit low-rank structures, meaning the dominant update directions are well-captured by the top singular vectors of the input activations. Thus, the bias is controlled and mitigated by the data-informed nature of the subspace.

Key Contributions

Activation-Informed Subspace: Unlike prior subspace methods that use random or static low-rank matrices, ZO-Act derives the perturbation subspace directly from input activations, ensuring the subspace aligns with the layer's dominant activation directions.
Explicit Coefficient Optimization: By freezing the basis and optimizing only the coefficient matrix $B$ , ZO-Act transforms ZO fine-tuning into an explicit optimization problem over a low-dimensional space. This enables the direct application of momentum-based optimizers (e.g., Adam) and avoids the materialization of full-weight perturbations.
Quantization Compatibility: The method naturally supports quantized LLMs (e.g., INT4) because the low-bit weights remain frozen, and only the low-bit coefficient matrices are updated.
Theoretical Guarantees: The paper provides a convergence analysis showing that ZO-Act reduces both the variance of the gradient estimator and the finite-difference error compared to full-weight ZO, at the cost of a manageable subspace bias.

Experimental Results

The authors evaluated ZO-Act on Llama-3-8B, OPT-13B, and an INT4-quantized Llama-3-8B across language understanding, question answering, and commonsense reasoning tasks.

Performance: ZO-Act consistently outperformed strong ZO baselines (including MeZO, LOZO, SubZero, AGZO, and ZO-Muon).
- On Llama-3-8B, it achieved the best full-precision ZO results on SST-2, RTE, BoolQ, and WiC. Notably, it improved RTE performance from 81.2 (ZO-Muon) to 87.0 and CB from 69.6 to 89.3.
- On OPT-13B, it similarly achieved top results on SST-2, RTE, CB, and BoolQ.
- On INT4 Quantized Models, ZO-Act remained competitive with full-precision ZO baselines, demonstrating its efficacy for memory-constrained fine-tuning.
Efficiency: ZO-Act matched the lowest memory usage among compared methods (approx. 16.1 GB for Llama-3-8B, dropping to 5.8 GB for INT4) and achieved the fastest runtime (42.0 minutes vs. 44.5+ minutes for others) under a fixed forward-query budget.
Rank Sensitivity: Experiments showed that a rank of $r=1$ often sufficed for full-precision models, yielding the best accuracy while minimizing perturbation dimension. Higher ranks ( $r=32$ ) were beneficial for quantized models to compensate for reduced capacity.
Stability: Diagnostic studies confirmed that the activation-informed subspace remains stable during fine-tuning and captures a significant portion of the full gradient energy, validating the "one-shot" initialization design.

Significance and Claims

The paper claims that ZO-Act represents a significant step forward in making zeroth-order fine-tuning a practical and high-performance alternative for large and quantized LLMs. Its significance lies in:

Bridging the Gap: It narrows the performance gap between ZO and first-order fine-tuning by reducing estimator variance through data-informed subspaces.
Enabling Optimizers: It resolves the difficulty of applying standard optimizers like Adam to ZO methods by exposing explicit, low-dimensional trainable variables.
Scalability: It provides a memory-efficient solution for fine-tuning quantized models, which is critical for deployment on resource-constrained hardware.

The authors acknowledge limitations, noting that ZO-Act still does not fully close the performance gap with first-order methods due to the inherent noise of ZO estimation and the subspace bias. Additionally, ZO-Act remains slower in wall-clock time than backpropagation-based methods due to the multiple forward passes required per update. However, the method is positioned as a superior choice for scenarios where backpropagation is memory-prohibitive or unavailable.