An Improved Adaptive PID Optimizer with Enhanced Convergence and Stability for Deep Learning

Technical Summary: An Improved Adaptive PID Optimizer with Enhanced Convergence and Stability for Deep Learning

Problem Statement
Deep learning training relies heavily on optimization algorithms, primarily built upon momentum-based Stochastic Gradient Descent (SGD). While momentum accelerates training, it suffers from two fundamental drawbacks: (1) noisy and varying gradients across parameters, which a single global learning rate cannot effectively manage, and (2) an "overshoot" phenomenon where accumulated momentum causes parameters to oscillate around the optimum.

To address these, Adaptive PID (AdaPID) optimizers were proposed, combining the adaptive learning rate mechanism of Adam with the control-theory-based Proportional-Integral-Derivative (PID) structure to mitigate oscillations. However, the authors identify that AdaPID inherits two critical limitations from its Adam foundation:

1. Convergence Issues: Like Adam, AdaPID may fail to guarantee convergence due to the nature of its exponential moving average of squared gradients.
2. Stability Issues: AdaPID can still exhibit instability during training, leading to fluctuating performance.

Methodology
The paper proposes a novel optimizer, IAdaPID-ADG (Improved AdaPID AMSDiffGrad), which integrates specific enhancements from two existing Adam extensions—AMSGrad and DiffGrad—into the AdaPID framework.

1. Addressing Convergence (AMSGrad Integration):
   To ensure a non-increasing effective learning rate and guarantee convergence, the authors integrate the core idea of AMSGrad. Instead of using the standard exponential moving average of squared gradients ($v_t$) and squared gradient differences ($d_t$), the optimizer maintains a running maximum of these values ($v^{\max}_t$ and $d^{\max}_t$). These maximum values are used in the denominator of the update rule, preventing the effective learning rate from increasing unexpectedly.

2. Addressing Stability (DiffGrad Integration):
   To mitigate instability and oscillations, the authors incorporate a modulation factor inspired by DiffGrad. This factor, $\mu_t$, is computed based on the absolute difference between the current and previous gradients ($\Delta g_t$) using a sigmoid function: $\mu_t = \frac{1}{1 + e^{-|\Delta g_t|}}$. This factor scales the learning rate dynamically; when the gradient difference is large, the modulation factor adjusts the step size to stabilize the update.

3. The Combined Update Rule:
   The final IAdaPID-ADG update rule combines the PID integral ($I_t$) and derivative ($D_t$) terms with the adaptive scaling from AMSGrad (using $v^{\max}$ and $d^{\max}$) and the stability modulation from DiffGrad ($\mu_t$). The weight update is defined as:
   $$w_t = w_{t-1} - \eta \mu_t \left( \frac{K_i I_t}{\sqrt{\hat{v}^{\max}_t} + \sigma} + \frac{K_d D_t}{\sqrt{\hat{d}^{\max}_t} + \sigma} \right)$$
   where $\eta$ is the learning rate, $K_i$ and $K_d$ are gain coefficients, and $\sigma$ is a stability constant.

Key Contributions

• Novel Optimizer Design: The introduction of IAdaPID-ADG, which uniquely merges the convergence guarantees of AMSGrad and the stability mechanisms of DiffGrad into the PID control structure.
• Comprehensive Evaluation: Extensive testing on four datasets:
  • Benchmark Datasets: MNIST (handwritten digits) and CIFAR-10 (color images).
  • Real-World Medical Datasets: IARC and AnnoCerv (cervical cancer image classification).
• Ablation Study: A systematic analysis on the MNIST dataset demonstrating that both the AMSGrad component (convergence) and the DiffGrad component (stability) contribute positively to the final performance when added to the base AdaPID.

Experimental Results
The proposed optimizer was evaluated against AMSGrad, DiffGrad, and standard AdaPID using training loss, training accuracy, and testing accuracy.

• Benchmark Datasets:
  • MNIST: IAdaPID-ADG achieved a training loss of $2 \times 10^{-5}$ (significantly lower than competitors' $4 \times 10^{-3}$), with 100% training accuracy and 98.68% testing accuracy.
  • CIFAR-10: The optimizer reached a training loss of $3 \times 10^{-6}$ and 80.09% testing accuracy, outperforming all baselines.
• Medical Datasets (IARC & AnnoCerv):
  • Tested on ResNet50, ResNet101, and ResNet152 architectures.
  • On IARC, IAdaPID-ADG achieved training losses around $4 \times 10^{-5}$ to $7 \times 10^{-4}$, compared to DiffGrad's $2 \times 10^{-2}$ to $5 \times 10^{-2}$. Testing accuracy reached up to 97.35%.
  • On AnnoCerv, the optimizer achieved testing accuracies between 94.96% and 95.39%, with training losses orders of magnitude lower than DiffGrad.
• General Observations: Across all datasets, IAdaPID-ADG demonstrated training losses that were 2–3 orders of magnitude lower than competing optimizers. The learning curves showed faster convergence and significantly reduced oscillation compared to baselines.

Significance and Claims
The paper claims that IAdaPID-ADG substantially outperforms existing optimizers by simultaneously resolving the convergence and stability issues inherent in adaptive PID methods. The authors assert that the integration of a non-increasing effective learning rate and a gradient-difference-based modulation factor results in an optimizer that is not only more accurate but also more stable during the training process.

The study concludes that the proposed method offers improved generalization and optimization efficiency across both standard benchmarks and complex real-world medical imaging tasks. As future work, the authors suggest applying the optimizer to complex information systems like recommendation engines and further analyzing its convergence properties through iterative numerical methods such as Krylov subspace approaches.