Technical Summary: FIBER – A Differentially Private Optimizer with Filter-Aware Innovation Bias Correction

1. Problem Statement

Differentially Private (DP) training protects individual data by adding noise to gradients. However, this injected noise interacts nontrivially with adaptive optimizers like AdamW. In standard DP-Adam, the squared-gradient accumulator ($v_t$) aggregates privatized gradients. Because the DP noise has zero mean but positive variance, the non-linear squaring operation converts this noise into a persistent positive bias in the second-moment estimate. This "preconditioner inflation" shrinks effective step sizes and degrades adaptivity over time.

Recent approaches attempt to mitigate this by applying temporal filtering (e.g., low-pass filters) to privatized gradients to reduce noise variance. However, the authors identify a critical failure mode: filtering alters the statistics of the DP noise seen by the optimizer's internal state. Specifically, temporal filtering attenuates the variance of the DP noise before it reaches the second-moment accumulator. Consequently, standard bias corrections (which subtract the full noise variance $\sigma_w^2$) become miscalibrated when filtering is present, leading to under-correction or over-correction. Existing methods that apply filtering do not model or correct for this attenuation within the adaptive optimizer's state.

2. Methodology: FIBER

The authors propose FIBER (Filter-aware Innovation Bias-corrected optimizER), a DP optimizer designed specifically for temporally filtered privatized gradients. FIBER integrates three primary components:

A. Innovation-Space Filtering

Unlike previous methods (e.g., DiSK) that apply Exponential Moving Average (EMA) smoothing directly to the gradient state, FIBER filters the residual (innovation) stream.

1. Residual Definition: The innovation is defined as the difference between the current privatized gradient and the previous filtered estimate: $\nu_t = g_t - \tilde{g}_{t-1}$.
2. Recursion: The residual stream is smoothed using a gain parameter $\omega$: $r_t = (1-\omega)r_{t-1} + \omega \nu_t$.
3. Integration: The denoised gradient estimate is formed by integrating the smoothed residual: $\tilde{g}_t = \tilde{g}_{t-1} + r_t$.
   This approach corresponds to a second-order recursion (analogous to a constant-velocity Kalman filter) rather than a first-order EMA. It allows the optimizer to track non-stationary dynamics (drift) in the latent gradient more effectively than simple state smoothing, particularly under high noise regimes.

B. Decoupled Hyperparameter Control

FIBER explicitly separates the two-point observation geometry from the temporal denoising strength:

• Observation Geometry: Determined by parameters $(\kappa, \gamma)$ used in the two-point gradient construction ($u_t(\xi)$), which defines the mixing of current and lookahead gradients.
• Denoising Gain: Determined by $\omega$, which controls the smoothing of the residual stream.
  In prior single-gain formulations, changing the smoothing parameter inherently altered the gradient estimation geometry. FIBER's decoupling allows for independent tuning of the estimator's extrapolation behavior and its temporal smoothing strength.

C. Filter-Aware Second-Moment Calibration

The core theoretical contribution is the derivation of the precise attenuation factor for DP noise variance under the innovation filter.

• Attenuation Factor: For a stable linear filter with gain $\omega$, the variance of the filtered DP noise is reduced by a factor $A(\omega)$. The authors derive this in closed form for the innovation filter:
  $$A(\omega) = \frac{2 - \omega}{4 - 3\omega}$$
  where $A(\omega) \in [0.5, 1]$.
• Correction: Instead of subtracting the full noise variance $\sigma_w^2$ (as in standard DP-Adam) or the unmodeled variance, FIBER subtracts the attenuated contribution:
  $$\bar{v}_t = \max(\hat{v}_t - A(\omega)\sigma_w^2, \epsilon_v)$$
  This ensures the preconditioner is correctly calibrated to the post-filter noise statistics, preventing the inflation or deflation of step sizes.

3. Key Contributions

1. Principled Innovation-Space Filtering: A denoising approach in residual space that uses a lightweight, theoretically grounded second-order recursion to track non-stationary dynamics under DP noise.
2. Decoupled Hyperparameter Control: The first explicit separation of observation geometry parameters $(\kappa, \gamma)$ from the innovation gain $\omega$, enabling independent tuning for practical optimization.
3. First Filter-Aware DP-AdamW Calibration: A rigorous analysis and derivation of the precise attenuation factor $A(\omega)$ for stable linear filters, leading to a bias correction that accounts for filter-induced variance reduction.
4. Extensive Empirical Validation: Demonstration of consistent improvements across vision and language benchmarks under tight privacy budgets.

4. Experimental Results

The authors evaluated FIBER on a diverse set of tasks including MNIST, CIFAR-10/100, ImageNet-1k, and GLUE/E2E NLP tasks.

• Performance: FIBER consistently outperforms state-of-the-art baselines, including DP-AdamW, DiSK (a temporal filtering baseline), and MF-DP-FTRL (a correlated-noise baseline).
• Privacy Regimes: The improvements are most substantial under tight privacy budgets (low $\epsilon$), where DP noise dominates. For example, on CIFAR-100 fine-tuning with ViT-small, FIBER improved accuracy by +2.0 to +5.1 percentage points over DiSK as $\epsilon$ ranged from 1 to 8.
• Ablation Studies:
  • Innovation vs. EMA: Innovation filtering significantly outperforms gradient-state EMA smoothing in high-noise regimes (e.g., $\epsilon=0.5$), but the gap narrows as noise decreases.
  • Decoupling: Independent tuning of $\omega$ and $(\kappa, \gamma)$ yields optimal performance, confirming the value of separating geometry from denoising.
  • Correction Necessity: Removing the filter-aware correction (using standard bias correction) degrades performance, validating the need to subtract the attenuated variance $A(\omega)\sigma_w^2$.
• Compute Efficiency: While FIBER requires two gradient evaluations per step (like DiSK), resulting in approximately 1.79x wall-clock overhead compared to single-step DP-AdamW, it achieves significantly higher accuracy per unit of computational overhead, especially in strict privacy settings.

5. Significance and Claims

The paper claims that FIBER addresses a critical gap in differentially private optimization: the mismatch between filtered gradient statistics and the bias corrections used in adaptive optimizers. By rigorously modeling how temporal filtering attenuates DP noise variance, FIBER provides a "filter-aware" calibration that restores the stability and adaptivity of AdamW under privacy constraints.

The authors position FIBER as a practical solution for long-horizon training and fine-tuning of large models where DP noise typically degrades utility. The work demonstrates that combining innovation-space denoising with precise statistical calibration yields state-of-the-art performance across both vision and language tasks, particularly when privacy budgets are tight. The paper concludes that while the computational cost of two-point gradients is non-negligible, the substantial gains in utility and stability justify the overhead, especially in scenarios where standard DP-Adam fails or converges poorly.