A Step-Size-Based Spectral Three-Term Conjugate Gradient Method for Unconstrained Optimization with Applications to Robotic Motion Control and Image Denoising

Technical Summary: A Step-Size-Based Spectral Three-Term Conjugate Gradient Method for Unconstrained Optimization

Problem Statement
The paper addresses the problem of solving large-scale unconstrained optimization problems of the form $\min_{x \in \mathbb{R}^n} f(x)$, where $f$ is continuously differentiable and its gradient satisfies Lipschitz continuity. While the Conjugate Gradient (CG) method is favored for its low memory usage and simplicity in handling large-scale problems, existing variants often struggle to balance theoretical convergence guarantees with numerical performance. Specifically, the authors note that while classical RMIL (Rivaie, Mustafa, Ismail, Leong) parameters and their three-term modifications satisfy sufficient descent conditions, they frequently fail to account for the Dai–Liao extended conjugacy condition, which incorporates curvature information to improve robustness, particularly for ill-conditioned problems. Furthermore, under inexact line searches (such as the Strong Wolfe strategy), certain RMIL variants lose equivalence with their exact line search counterparts, leading to distinct and sometimes suboptimal theoretical properties.

Methodology
The authors propose a new algorithm termed STTDL (Spectral Three-Term Dai–Liao). The method constructs a search direction $\delta_n$ using a three-term structure that integrates spectral scaling with a modified conjugacy condition.

1. Modified Conjugacy Condition: The core theoretical innovation is a modification of the Dai–Liao conjugacy condition ($\delta_n^T \gamma_{n-1} = -t g_n^T s_{n-1}$). The authors introduce a "safeguarded" step vector $\hat{s}_{n-1}$ (denoted as $bs_{n-1}$ in the text) which replaces the standard step vector $s_{n-1}$. This safeguard ensures the norm of the step vector is bounded away from zero, preserving curvature information while maintaining stability.
2. Three-Term Search Direction: The search direction is defined piecewise based on the relationship between the current gradient $g_n$, the previous gradient $g_{n-1}$, and the safeguarded step. The direction generally takes the form:
   $$\delta_n = -\phi_n g_n + \beta_n b s_{n-1} + \theta_n \gamma_{n-1}$$
   (or a variant involving $g_{n-1}$), where $\phi_n$ and $\hat{\phi}_n$ are spectral parameters.
3. Spectral Parameter Derivation: The spectral parameters $\phi_n$ and $\hat{\phi}_n$ are derived algebraically to ensure the untruncated search direction satisfies the modified conjugacy condition exactly. To guarantee the sufficient descent condition ($g_n^T \delta_n \leq -\lambda \|g_n\|^2$) regardless of the line search, these parameters are truncated such that $\phi_n, \hat{\phi}_n \geq 1$.
4. Algorithm Execution: The algorithm employs the Strong Wolfe line search conditions to determine the step size $\zeta_n$. It initializes with a steepest descent direction and iteratively updates the solution until the gradient norm falls below a specified tolerance.

Key Contributions

• New Spectral Three-Term Scheme: The paper introduces a novel iterative scheme that combines the RMIL framework with a spectral approach and a modified Dai–Liao conjugacy condition.
• Theoretical Guarantees: The authors prove that the proposed method satisfies the sufficient descent condition unconditionally for all iterations. Furthermore, under standard assumptions (bounded level sets and Lipschitz continuous gradients) and the Strong Wolfe line search, the global convergence of the algorithm is established ($\liminf_{n \to \infty} \|g_n\| = 0$).
• Modified Conjugacy Condition: A new modification of the generalized conjugacy condition is introduced, utilizing a safeguarded step vector to ensure stability and the retention of curvature information.
• Practical Applications: The method is applied to two distinct real-world problems:
  • Robotic Motion Control: Trajectory tracking for a 3-link planar robotic manipulator, formulated as a nonlinear least-squares problem.
  • Image Denoising: Restoration of images corrupted by salt-and-pepper noise, formulated as a smooth unconstrained optimization problem using a specific $\ell_1$-approximation model.

Experimental Results
The performance of STTDL was evaluated against two competing algorithms: IRMIL and TGRM.

• Benchmark Tests: Using 100 unconstrained test problems from Andrei (2008) with dimensions ranging from 2 to 100,000 variables, the authors utilized Dolan–Moré performance profiles.
  • Iterations (NI): STTDL achieved the best iteration count on approximately 75% of the test problems, outperforming IRMIL (60%) and TGRM (37%).
  • Function Evaluations (FE): STTDL required the fewest evaluations on about 51% of the problems.
  • CPU Time: While TGRM was fastest on roughly 44% of problems, STTDL maintained competitive performance, being the fastest on about 34% of cases.
• Application Results:
  • Robotics: In trajectory tracking, STTDL successfully guided the robot's end-effector along a Lissajous curve with residual errors decreasing rapidly to the range of $10^{-6}$ to $10^{-8}$. The joint motions remained smooth without instability.
  • Image Denoising: For images corrupted by 30% and 70% salt-and-pepper noise, STTDL demonstrated competitive Peak Signal-to-Noise Ratio (PSNR) values compared to IRMIL and TGRM, often requiring fewer iterations or comparable CPU time to achieve similar restoration quality.

Significance and Claims
The paper claims that the STTDL algorithm effectively bridges the gap between theoretical convergence properties and numerical efficiency. By incorporating a modified Dai–Liao conjugacy condition into a spectral three-term framework, the method ensures sufficient descent and global convergence while leveraging curvature information to enhance performance on large-scale and ill-conditioned problems. The authors assert that the algorithm is a reliable and effective tool for large-scale unconstrained optimization, validated not only by standard benchmark tests but also by its successful application in robotic control and image processing tasks. The study concludes that STTDL offers an excellent balance between robustness and computational efficiency.