Technical Summary: Adaptive Regularization for Sparsity Control in Bregman-Based Optimizers

Problem Statement

Sparse training offers a pathway to reduce the memory and computational costs of deep neural networks (DNNs). However, a significant challenge in sparse optimization is the indirect control of the final sparsity rate. Methods relying on explicit $\ell_1$ penalties or Bregman-based optimizers (such as LinBreg and AdaBreg) typically control sparsity via a regularization parameter $\lambda$. The mapping between a specific $\lambda$ value and the resulting sparsity rate is non-trivial and highly sensitive.

The authors observe that this sensitivity is particularly pronounced in Bregman-based optimizers. For instance, LinBreg and AdaBreg may require $\lambda$ values differing by up to two orders of magnitude to achieve the same final sparsity. This necessitates expensive, manual trial-and-error sweeps to find an "oracle" $\lambda$ that meets a user-specified sparsity target, hindering the practical deployment of these methods. Furthermore, existing adaptive approaches often rely on $\ell_1$-regularized losses or require re-training, making them computationally expensive or incompatible with the Mirror Descent (MD) setup used in modern sparse training.

Methodology

The paper proposes an adaptive regularization scheme integrated into the Bregman learning framework (specifically LinBreg and AdaBreg) to automatically adjust $\lambda$ during training.

Core Framework

The method builds upon the inverse scale space approach introduced by Bungert et al. [7]. The optimization problem is framed as minimizing a loss function $L(\theta)$ subject to a sparsity constraint $|\theta|_0 \leq a^*$. The algorithm iteratively updates parameters $\theta$ using a linearized loss and a regularization functional $R$ based on Bregman divergence.

To promote sparsity, the authors employ the Elastic Net functional:
$$EN_\lambda(\theta) := \frac{1}{2}|\theta|_2^2 + \lambda|\theta|_1$$
This choice ensures strong convexity and allows for the use of proximal operators. The update rule involves a subgradient step followed by a proximal step:

1. Subgradient Step: $p^{(k+1)} = p^{(k)} - \tau \nabla L(\theta^{(k)})$
2. Proximal Step: $\theta^{(k+1)} = \nabla EN^*_{\lambda(k)}(p^{(k+1)})$

Adaptive Strategy

The key contribution is the dynamic adaptation of $\lambda$ based on the sparsity defect $\epsilon^{(k)} = s^* - s(\theta^{(k)})$, where $s^*$ is the target sparsity and $s(\theta)$ is the current sparsity.

The update rule for $\lambda$ is applied every $f$ iterations:
$$\lambda^{(k+1)} = \begin{cases} \lambda^{(k)} (1 + \alpha |\epsilon^{(k)}|)^{\text{sgn}(\epsilon^{(k)})} & \text{if } k \pmod f = 0 \\ \lambda^{(k)} & \text{otherwise} \end{cases}$$
Where $\alpha$ is an acceleration factor.

Stability Mechanisms:

• Damping: As the sparsity defect $|\epsilon^{(k)}|$ approaches a small tolerance threshold $\zeta_d$, the update frequency $f$ and the acceleration factor $\alpha$ are reduced (damped) to prevent high-frequency oscillations around the target sparsity.
• Theoretical Analysis: The authors provide a lemma showing that under $L$-smoothness assumptions, the loss decays with an error term controlled by the magnitude of the change in $\lambda$. They also analyze the bounded relative change of $\lambda$ to ensure stability.

Key Contributions

1. Automatic Sparsity Control: The paper extends the Bregman framework [7] with an adaptive $\lambda$ mechanism that reliably steers models to user-specified sparsity targets (ranging from 75% to 99%) without manual hyperparameter tuning.
2. Performance Parity with Oracle Tuning: Empirical results demonstrate that models trained with the automatic scheme converge to solutions performing on par with, or better than, those trained with an oracle-tuned fixed $\lambda$.
3. Robustness Preservation: The study investigates the out-of-distribution (OOD) robustness of sparse models. It finds that sparsity induces robustness and that the adaptive scheme preserves this property, often outperforming dense baselines on OOD data (e.g., CNCeleb).
4. Analysis of Sparsity Allocation: The authors identify a sub-optimal sparsity allocation issue where Bregman optimizers tend to keep the classifier layer denser than the rest of the network. At extreme sparsity levels (e.g., 99%), this forces intermediate layers into near-complete sparsity, degrading performance. The paper suggests that per-layer rescaling of $\lambda$ can mitigate this.

Experimental Results

The method was evaluated on Automatic Speaker Verification (ASV) tasks using the VoxCeleb and CNCeleb datasets with ECAPA-TDNN and ResNet34 architectures.

• Sparsity Control: The adaptive method successfully achieved target sparsity levels between 75% and 99% for both optimizers and architectures.
• Convergence: Adaptive models converged faster than non-adaptive (fixed $\lambda$) baselines during early training.
• Performance (EER):
  • On in-domain data (VoxCeleb), adaptive sparse models matched or surpassed the Equal Error Rate (EER) of oracle-tuned non-adaptive baselines.
  • On OOD data (CNCeleb-E), sparse models generally exhibited better generalization than dense baselines, confirming the robustness hypothesis.
  • Architecture Sensitivity: ECAPA-TDNN performed well up to 95% sparsity but degraded sharply at 99% due to the classifier absorbing too much of the sparsity budget. ResNet34 showed similar trends but was more sensitive to the classifier's parameter share.
• Comparison with Pruning: Gradual pruning baselines performed similarly to dense models, whereas Bregman optimizers (especially AdaBreg) achieved superior performance on ECAPA-TDNN at high sparsity levels.

Significance and Claims

The paper claims that its adaptive scheme significantly reduces the engineering cost of producing sparse models by eliminating the need for per-target $\lambda$ searches. This lowers the energy footprint associated with model selection and facilitates deployment on resource-constrained devices.

The authors modestly note that while the method successfully controls sparsity and preserves robustness, it does not fully resolve the issue of sub-optimal sparsity allocation in the classifier layer at extreme sparsity levels. They identify the derivation of theoretical bounds for sparsity oscillations and the development of per-layer sparsity allocation strategies as important directions for future work. The paper does not claim to alter the inherent fairness or biometric misuse risks of speaker verification, noting these remain the responsibility of the deployment stage.