🤖 machine learning

From Non-Convex Self-Concordant Regularization to Scalable Quasi-Newton Training of PINNs

This paper introduces SCORE, a novel self-concordance-inspired quasi-Newton method that utilizes a decrement-coupled shifted secant geometry to stabilize training and achieve lower errors on complex partial differential equations without requiring explicit Hessian construction.

Original authors: Chenhao Si, Kang An, Shiqian Ma, Ming Yan

Published 2026-08-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chenhao Si, Kang An, Shiqian Ma, Ming Yan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Art of Teaching Computers to Solve Nature's Riddles

Imagine you are trying to teach a computer to predict how a drop of ink spreads in water, or how a shockwave ripples through a gas. In the world of science, these are called Partial Differential Equations (PDEs), and they are the mathematical rulebooks nature uses to run the universe. For decades, scientists have used "Physics-Informed Neural Networks" (PINNs) to solve these puzzles. Think of a PINN as a super-smart student who hasn't just memorized the answers but has been forced to read the rulebook (the physics equations) while trying to guess the solution. The student gets graded on how well their guess fits the rules.

However, there's a catch. As the student gets closer to the perfect answer, the grading becomes incredibly tricky. The "rules" become so sensitive that tiny mistakes in the student's guess cause huge, confusing swings in the score. It's like trying to balance a pencil on its tip while standing on a shaking boat; standard study methods (optimizers) often get stuck or give up, unable to find that final, perfect balance. To get a truly accurate solution, the computer needs a way to understand the "shape" of the problem's difficulty, not just the direction to move. This is where the concept of "curvature" comes in—imagining the problem landscape as a bumpy terrain where some hills are steep and others are flat, and the computer needs to know exactly how to step without falling off.

The Paper's Story: A New Way to Walk the Tightrope

In this paper, the authors introduce a new training method called SCORE (Self-Concordant Quasi-Newton method with Shifted Secant Geometry). Their main finding is that by changing how the computer measures the "bumpiness" of the problem landscape, they can guide the neural network to much higher accuracy than before, especially when the solution is already very close to perfect.

The authors argue that standard methods often fail in the final stages of training because they try to measure the terrain using raw, unfiltered data that can be "indefinite" (confusingly flat or even upside-down) or "nearly singular" (so flat it looks like a cliff). They explicitly rule out the idea that simply making the steps smaller or using standard "second-order" tricks is enough to fix this. Instead, they suggest that the computer needs a "shifted" perspective.

Here is how SCORE works, using a playful analogy:

Imagine the computer is a hiker trying to find the very bottom of a foggy, twisting valley (the perfect solution).

The Old Way (BFGS/SSBroyden): The hiker looks at the ground immediately under their feet and the spot they just left to guess the slope. But sometimes, the ground is so weirdly shaped (due to the complex physics) that this guess is wrong, or the ground is so flat the hiker doesn't know which way is down. The hiker might take a tiny, safe step or get confused and stop.
The SCORE Way: The authors propose a clever trick. Before the hiker takes a step, they imagine the ground is slightly "lifted" or "shifted" up by a small, adjustable amount. This is like putting a thin, invisible mattress under the hiker's feet. This "shift" guarantees that the ground always looks like a gentle, downward slope, even if the real ground is confusingly flat.
The Magic Link: The size of this "mattress" isn't random. It is automatically adjusted based on a specific "decrement" number the computer calculates. This number tells the computer how confident it is in its current map.
- If the map is shaky (high decrement), the mattress gets thicker, making the path safer and the steps smaller.
- If the map is clear (low decrement), the mattress gets thinner, allowing the hiker to take bigger, more confident strides.

The paper shows that this "shifted" view allows the computer to keep refining its answer even when the standard methods get stuck. They tested this on four different "nature puzzles":

Viscous Burgers Equation: A problem about how fluids flow and mix.
Kuramoto–Sivashinsky Equation: A chaotic system that models how patterns form and break down, like flames or chemical reactions.
Korteweg–de Vries (KdV) Equation: A problem about how waves travel and interact, like tsunamis or water ripples.
Complex Ginzburg–Landau Equation: A 2D problem involving complex, oscillating patterns, often used to model superconductors or fluid turbulence.

In all these tests, SCORE consistently reached a lower error rate than the standard methods (BFGS and self-scaled Broyden). For example, on the Burgers equation, the standard methods made errors around $1.40 \times 10^{-8}$ , while SCORE got it down to $2.25 \times 10^{-9}$ —a significant improvement in precision. The authors note that this improvement happens without slowing the computer down; the "mattress" trick adds almost no extra time to the calculation.

The paper suggests that the secret to this success is that SCORE doesn't just look at the raw data; it uses a "curvature-relative" view. It adapts its understanding of the problem's shape to the specific moment in the training process. By linking the "shift" (the safety mattress) directly to the "decrement" (the confidence meter), the method creates a self-correcting loop. The authors demonstrate through these simulations that this approach allows the neural network to squeeze out the final bits of accuracy that other methods miss, turning a "good enough" solution into a highly precise one.

In short, the paper finds that when teaching a computer to solve nature's hardest math problems, you don't just need to tell it to walk carefully; you need to give it a way to see the ground clearly, even when the ground itself is trying to trick it. SCORE provides that clear view, ensuring the computer doesn't just stop at "almost right," but keeps going until it hits "perfect."

Technical Summary: From Non-Convex Self-Concordant Regularization to Scalable Quasi-Newton Training of PINNs

1. Problem Statement

Physics-informed neural networks (PINNs) are a mesh-free framework for solving partial differential equations (PDEs) by minimizing a residual objective composed of PDE, boundary, and initial condition terms. While first-order methods like Adam are robust for early training stages, they often stagnate when high-accuracy solutions are required. At this stage, the residual objective typically exhibits indefinite, nearly singular, and poorly scaled local curvature.

This ill-conditioning arises because the PINN Hessian is a sum of a nonnegative Gauss–Newton term and a residual-dependent term that can have either sign. Standard quasi-Newton methods (e.g., BFGS, self-scaled Broyden) infer curvature from successive gradient differences. When the observed secant curvature is weak or indefinite, the learned metric becomes unreliable, forcing optimizers to rely on conservative line searches, damping, or update skipping. This limits the attainable accuracy in the "small-residual" regime where further progress depends on extracting stable curvature information rather than coarse descent.

2. Methodology: The SCORE Algorithm

The authors propose SCORE (Self-Concordance-inspired Quasi-Newton method with Decrement-coupled Shifted secant geometry), a Hessian-free optimizer designed to stabilize late-stage PINN refinement. The method integrates three core concepts:

2.1. Weak Self-Concordance and Shifted Metrics

To address the indefiniteness of the PINN Hessian, the method adopts a weak self-concordance perspective. Instead of relying on the raw Hessian, it utilizes a shifted local metric $M_\mu(\theta) = \nabla^2 f(\theta) + \mu I$ , where $\mu \geq 0$ acts as a positive curvature floor. This shift ensures the local metric is positive definite without altering the underlying optimization objective.

2.2. Shifted Secant Geometry

SCORE represents this positive shifted metric through the secant update mechanism of Self-Scaled Broyden (SSBroyden) methods.

Let $s_k = \theta_{k+1} - \theta_k$ and $y_k = \nabla f(\theta_{k+1}) - \nabla f(\theta_k)$ be the standard step and gradient displacement.
The method introduces a shifted gradient displacement: $\tilde{y}_k = y_k + \mu_k s_k$ .
The pair $(s_k, \tilde{y}_k)$ represents the action of the averaged shifted metric along the accepted step.
The inverse Hessian approximation $H_{k+1}$ is updated using $\tilde{y}_k$ instead of $y_k$ . This allows the optimizer to learn a positive local geometry without explicitly constructing the Hessian or Hessian-vector products.

2.3. Decrement-Coupled Control

A distinguishing feature of SCORE is that a single quasi-Newton decrement ( $\lambda_k$ ) jointly controls both the step selection and the curvature shift:

Decrement Calculation: $\lambda_k = \sqrt{\max\{g_k^\top H_k g_k, 0\}}$ , measuring the proposed step size in the learned metric.
Adaptive Shift: The shift magnitude is determined by $\mu_k = \text{clip}(\eta \lambda_k, [\mu_{\min}, \mu_{\max}])$ . A large decrement (indicating a large step in the current metric) triggers a stronger curvature shift to stabilize the next update.
Candidate Step: A candidate step length is proposed as $\alpha_k^{sc} = (1 + \lambda_k)^{-1}$ , inspired by the self-concordant damped step rule.
Acceptance Logic: The candidate is accepted if it satisfies the strong Wolfe conditions. If rejected, the method falls back to a standard Wolfe line search.

This coupling ensures that the stabilization used to construct the next metric is directly linked to the step taken with the current metric.

3. Key Contributions

Geometric Interpretation: The paper establishes a link between the residual least-squares geometry of PINNs and the curvature learned during self-scaled Broyden refinement, identifying the need to stabilize the local metric represented by the secant update.
Weak Self-Concordance Framework: It introduces a weak-self-concordance-based local-metric perspective for high-accuracy PINN refinement, providing a unified basis for curvature stabilization and decrement-dependent step control in nonconvex objectives.
SCORE Algorithm: It develops a scalable, Hessian-free realization of this perspective within SSBroyden. The method uses shifted secant displacements to represent positive local geometry and couples the shift magnitude to the quasi-Newton decrement.
Empirical Validation: The method is evaluated on four nonlinear PDE benchmarks (viscous Burgers, Kuramoto–Sivashinsky, Korteweg–de Vries, and complex Ginzburg–Landau equations).

4. Experimental Results

The authors evaluated SCORE against standard BFGS and self-scaled Broyden (SSBroyden) baselines across four PDE benchmarks, using identical network architectures, initialization, and training protocols (Adam warm-start followed by quasi-Newton refinement).

Viscous Burgers Equation: SCORE achieved a relative $L_2$ error of $2.25 \times 10^{-9}$ , significantly outperforming SSBroyden ( $1.40 \times 10^{-8}$ ) and BFGS ( $1.56 \times 10^{-8}$ ). An ablation study confirmed that both the shifted secant displacement and the self-concordant candidate-step test contribute complementarily to this improvement.
Kuramoto–Sivashinsky Equation: In this chaotic, time-dependent problem, SCORE sustained error reduction in later time windows where baselines stagnated, achieving a relative $L_2$ error of $1.84 \times 10^{-4}$ compared to $5.59 \times 10^{-4}$ (BFGS) and $7.95 \times 10^{-4}$ (SSBroyden).
Korteweg–de Vries (KdV) Equation: SCORE demonstrated superior control over residual errors in regions of complex wave interactions, reaching a final error of $3.57 \times 10^{-7}$ versus $1.49 \times 10^{-6}$ for BFGS.
Complex Ginzburg–Landau Equation: In a 2D complex-valued setting, SCORE achieved the lowest errors for both real and imaginary components, with a relative $L_2$ error of $8.42 \times 10^{-4}$ for the real part, compared to over $2.1 \times 10^{-3}$ for baselines.

Across all benchmarks, SCORE consistently attained lower final errors while maintaining comparable computational overhead (wall-clock time per refinement block) to the baselines.

5. Significance and Claims

The paper claims that SCORE improves the final accuracy of PINN training by addressing the specific challenge of optimizing in a small-residual, highly ill-conditioned regime.

Curvature-Relative Optimization: The authors argue that in the late stages of PINN optimization, progress is limited not by coarse descent but by the ability to extract reliable curvature information from a delicate local loss geometry. SCORE addresses this by using a curvature-relative view (weak self-concordance) rather than worst-case Lipschitz assumptions.
Stabilization without Objective Modification: The method stabilizes the learned curvature model through shifted secant geometry without modifying the underlying PINN objective or introducing external penalty terms.
Scalability: By avoiding explicit Hessian construction and using a single decrement to coordinate step selection and metric stabilization, SCORE provides a scalable path to high-accuracy refinement for scientific machine learning.

The authors conclude that this work points toward a broader class of second-order methods designed specifically for physics-informed objectives, where step sizes and metric updates are governed by the geometry of the PDE loss itself.