Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theory

Technical Summary: Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theory

Problem Statement
Continual learning requires models to adapt to new tasks without erasing capabilities acquired from previous ones. While recent empirical work (e.g., ZeroFlow, ZO-FC) suggests that zeroth-order (ZO) optimization and forward-pass adaptation can mitigate forgetting more effectively than first-order (FO) descent, the prevailing view treats ZO merely as a noisy estimator of the FO gradient. This perspective fails to explain why ZO might retain better, as estimation accuracy concerns the incoming gradient, whereas forgetting is governed by the induced parameter displacement and the retention curvature it exposes. The paper seeks to isolate the local mechanism of forgetting after the incoming signal and adaptation scale are matched, moving beyond the "noisy FO" narrative to understand the geometric properties of the adaptation shape.

Methodology: Randomized Gradient Shaping
The authors propose a framework where forgetting is analyzed as "shaped adaptation." Instead of comparing optimization trajectories, they compare the quadratic damage functional $Q(\Delta) = \frac{1}{2}\Delta^\top H \Delta$ induced by different displacement operators $\Delta$, where $H$ represents the retention curvature (e.g., the Hessian of historical loss).

1. Calibration of ZO: The paper decomposes practical finite-difference ZO estimators into three components:
   • Raw Mean Alignment: The raw ZO shape preserves the FO signal in expectation.
   • Norm Inflation: Raw ZO inflates the expected squared adaptation norm.
   • Norm Matching: To ensure a fair comparison, the authors introduce a norm-matched ZO shape $P = \kappa^{-1/2}Z$, where $Z$ is the raw shape matrix and $\kappa$ is an inflation factor. This ensures $E[\|Pg\|^2] = \|g\|^2$ while explicitly inducing a mean shrinkage factor of $\sqrt{q/(q+d+1)}$.
2. Theoretical Analysis: Under this norm-matched comparison, the authors derive an exact identity for the expected shaped retention curvature $\bar{H}_q = E[P^\top H P]$. They prove that norm-matched ZO does not uniformly shrink curvature but rather preserves the isotropic component (the average curvature $\bar{\lambda}I$) while contracting the anisotropic component (the deviation from the mean) by a factor of $(1-\tau)$, where $\tau = d/(q+d+1)$.
3. Forgetting Gap Identity: By projecting this curvature identity onto the incoming gradient $g$, the paper derives a precise FO–ZO quadratic forgetting gap. The reduction in forgetting occurs if and only if the FO direction points through a region of above-average retention curvature ($\lambda > \bar{\lambda}$). In this regime, ZO reduces the "curvature excess" damage by a query-dependent fraction. Conversely, if the FO direction is in a low-curvature region, ZO may increase forgetting, framing the result as a geometry-dependent risk control rather than a universal dominance.

Key Contributions

1. Formulation of Local Forgetting as Shaped Adaptation: The paper reframes the comparison between FO and ZO as a question of how different shape operators expose retention curvature under a matched expected squared adaptation norm, removing confounders like step size and trajectory.
2. Calibration of Finite-Difference ZO: It rigorously separates raw mean alignment, finite-smoothing residuals, raw norm inflation, and norm-matched mean shrinkage, establishing a fair comparator for theoretical analysis.
3. Exact Curvature Identity and Forgetting Gap: The authors prove that norm-matched ZO preserves the isotropic retention floor while contracting the anisotropic component. This yields an exact identity showing that ZO reduces mean quadratic forgetting precisely when the FO direction crosses above-average retention curvature.
4. Algorithmic Transfer (RISE): The paper introduces RISE (Retention-Aware Isotropic Shaping for Exact-Gradient Adaptation). RISE transfers the calibrated ZO shape mechanism to exact-gradient adaptation. It computes the exact FO gradient via backpropagation (removing finite-difference smoothing bias) but applies the norm-matched, randomized shaping operator blockwise. This allows the method to import the retention-curvature mixing benefits of ZO while retaining the plasticity signal of exact gradients.

Results

• Theoretical Validation: In a quadratic sandbox, the empirical operator residuals and damage gaps align with the theoretical predictions ($R^2 = 0.9999$), confirming that the mechanism is geometry-dependent and that the sign of the forgetting gap changes as predicted based on the directional curvature excess.
• Vision Continual Learning: On CIFAR100, ImageNet-R, and DomainNet using a ViT-B/16 backbone, RISE applied to the ZO-FC pipeline (replacing the representation-side finite-difference ZO with exact-gradient shaping) improved "Last" accuracy (plasticity) by 0.39–1.39 points and reduced "Fgt" (forgetting) by 0.28–1.72 points compared to standard ZO-FC. This demonstrates that exact gradients can recover plasticity lost to gradient estimation while maintaining the stability benefits of the ZO shape.
• Language Model Continual Learning: On T5 models across various task orders (SeqLoRA and O-LoRA), RISE significantly reduced forgetting (e.g., from 35.57% to 7.01% on one SeqLoRA order) while maintaining "Last" accuracy close to the FO baseline. This contrasts with raw ZO, which often reduces forgetting at the cost of poor plasticity (low "Last" scores).

Significance and Claims
The paper claims to provide the first scale-controlled local mechanism explaining why ZO may forget less, moving beyond the "noisy FO" explanation. It establishes that the benefit is not a uniform dominance of ZO over FO, but a conditional improvement: ZO acts as a "blind-optimal" isotropic exposure that protects against high-curvature directions when the specific retention geometry is unknown.

The significance of RISE lies in its ability to decouple the stability mechanism (randomized shaping) from the plasticity mechanism (exact gradients). By applying the calibrated ZO shape to exact gradients, RISE offers a stability–plasticity tradeoff where the model avoids the finite-smoothing bias of practical ZO while still benefiting from the curvature-mixing properties that reduce retention exposure in high-curvature regimes. The paper concludes that this mechanism is most visible when the FO direction has above-average retention curvature and when blockwise sampling allows for stable realization of the shaping effect without the variance associated with global high-dimensional random directions.