🤖 machine learning

L-SR1: Learned Symmetric-Rank-One Preconditioning

The paper proposes L-SR1, a lightweight learned second-order optimizer that enhances the classical Symmetric-Rank-One algorithm with a trainable preconditioning unit to achieve superior generalization and performance without requiring annotated data or fine-tuning.

Original authors: Gal Lifshitz, Shahar Zuler, Ori Fouks, Dan Raviv

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Gal Lifshitz, Shahar Zuler, Ori Fouks, Dan Raviv

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find the lowest point in a vast, foggy valley (this represents solving a complex math problem). You have two main ways to do this:

The "Deep Learning" Hiker: This hiker has a massive map and a GPS that learned from millions of other hikers. They can run very fast, but they need a huge battery (computing power) and a massive library of maps (labeled data) to work. If they encounter a new type of terrain they've never seen, they might get lost.
The "Classical" Hiker: This hiker uses a simple compass and feels the slope under their feet. They are very efficient with their battery and don't need a massive library of maps. However, they take tiny, cautious steps. It often takes them a very long time to reach the bottom because they don't know the shape of the valley well.

The Problem: We want the speed of the Deep Learning hiker but the efficiency and adaptability of the Classical hiker.

The Solution: L-SR1
The paper introduces a new method called L-SR1 (Learned Symmetric-Rank-One). Think of it as a "Smart Compass" that learns how to feel the terrain better over time, without needing a massive battery or a library of maps.

Here is how it works, broken down into simple concepts:

1. The "Smart Compass" (The Preconditioner)

In math, finding the bottom of the valley requires knowing not just which way is "down" (the slope), but also how the ground curves (the curvature).

Old methods either ignore the curves (taking slow, straight steps) or try to calculate the exact curve every time (which is too slow and heavy).
L-SR1 uses a "Smart Compass" that guesses the shape of the curve. It doesn't calculate the whole map; instead, it remembers a few recent steps it took and uses those to guess the shape of the ground ahead. This is called a "rank-one update."

2. The "Guardian" (PGSM)

There is a risk with guessing the curve: you might guess wrong and start walking up the hill instead of down.

The paper introduces a special rule called PGSM (Projection-Guided Secant Mechanism).
Think of PGSM as a guardian angel attached to the compass. Its only job is to make sure the compass never points in a direction that would make you walk uphill. It forces the compass to stay "positive" (pointing down) and ensures the guess about the curve matches the reality of the last step.
Crucially, this guardian doesn't slow you down. It does its checking while the compass is learning, so when you actually use it, it's fast and light.

3. Learning Without a Teacher

Most "learning" systems need a teacher to show them the right answer (labeled data).

L-SR1 is self-taught. It learns by trying to solve many different types of valleys (math problems) on its own. It doesn't need a teacher to say, "That was the right step." It just figures out, "If I took this step, did I get closer to the bottom?"
Because it learns the rules of how to navigate, it can handle valleys of different sizes (dimensions) without needing to re-learn everything.

4. The Real-World Test: 3D Human Mesh Recovery

To prove this works, the authors tested it on a tricky real-world problem: Monocular Human Mesh Recovery (HMR).

The Task: Imagine taking a single 2D photo of a person and trying to figure out their 3D body shape. It's like trying to guess the shape of a sculpture just by looking at its shadow. It's a very confusing, "ill-posed" problem because depth information is lost.
The Result: L-SR1 was able to reconstruct the 3D body shape faster and more accurately than older, heavy methods, and it didn't need a massive amount of training data or a huge computer model to do it.

Summary of the Paper's Claims

It bridges the gap: It combines the speed of modern AI with the efficiency of classical math.
It's lightweight: It uses a small, compact model that doesn't need to be retrained for every new problem size.
It's stable: The "Guardian" (PGSM) ensures it doesn't make wild, unstable guesses.
It works: In tests, it solved math puzzles and reconstructed 3D human bodies from 2D photos better than both traditional methods and other "learned" methods, all while using less computing power.

In short, L-SR1 is a lightweight, self-taught navigator that learns to feel the shape of the terrain, ensuring you always take the fastest, safest path to the bottom of the hill.

Technical Summary: L-SR1: Learned Symmetric-Rank-One Preconditioning

1. Problem Statement

End-to-end deep learning has achieved significant success but faces three primary limitations: a heavy reliance on large labeled datasets, limited generalization to unseen scenarios, and substantial computational costs. Conversely, classical optimization methods are data-efficient and lightweight but often suffer from slow convergence due to their iterative nature. While "learned optimizers" have emerged to bridge this gap by using deep neural networks to accelerate convergence, existing approaches have predominantly focused on first-order methods (e.g., Adam, NAG). Learned second-order optimization, which leverages curvature information for faster convergence, remains underexplored.

Furthermore, specific applications like Monocular Human Mesh Recovery (HMR)—estimating 3D human meshes from single 2D images—present high-dimensional, non-convex optimization challenges. Traditional optimization-based HMR methods are slow and initialization-sensitive, while modern deep learning approaches require massive annotated datasets and large model sizes. There is a need for a lightweight, generalizable optimizer that combines the efficiency of second-order methods with the adaptability of learned optimization, without requiring task-specific fine-tuning or extensive annotated data.

2. Methodology: L-SR1

The authors propose Learned-Symmetric-Rank-One (L-SR1), a learned second-order optimizer that extends the classical Symmetric Rank-One (SR1) Quasi-Newton method. The core innovation is the integration of a Projection-Guided Secant Mechanism (PGSM) into a lightweight, self-supervised framework.

Core Components

Learned Preconditioning: Unlike classical SR1 which uses fixed update rules, L-SR1 employs a learnable module to generate data-driven rank-one updates to approximate the inverse Hessian.
Architecture: The method utilizes three lightweight Multi-Layer Perceptron (MLP) modules operating element-wise to ensure invariance to problem dimension:
1. Input Encoder ( $E$ ): Constructs a latent representation from the optimization state (current point, step, descent direction, gradient, and gradient step).
2. Vector Generator ( $P$ ): Produces a direction vector $v_k$ at each iteration.
3. Learning Rate Generator ( $G$ ): Outputs element-wise learning rates $\alpha_k$ .
Limited-Memory Buffer: To ensure scalability, L-SR1 maintains a fixed-size buffer $B_L$ of the most recent $L$ vectors. The preconditioner is reconstructed implicitly as a sum of rank-one outer products, avoiding the storage of full matrices.

The Projection-Guided Secant Mechanism (PGSM)

A known limitation of classical SR1 is that updates do not guarantee the preconditioner remains Positive Semi-Definite (PSD), potentially leading to non-descent directions. L-SR1 addresses this through PGSM:

PSD-by-Construction: The preconditioner is parameterized as $\tilde{B}_k = B_0 + \sum v_i v_i^\top$ , where $B_0$ is the identity matrix. This structure ensures the matrix is PSD by construction.
Secant Penalty: Instead of solving an explicit projection subproblem at inference (which is computationally expensive), PGSM incorporates a secant violation penalty into the meta-training loss. The loss function minimizes the mismatch $\|p_k - \tilde{B}_k q_k\|_2^2$ (where $p_k$ and $q_k$ are the step and gradient difference vectors) alongside the primary objective loss. This biases the meta-training toward satisfying the quasi-Newton secant relation while maintaining the PSD structure.

3. Key Contributions

L-SR1 Framework: The proposal of a lightweight, self-supervised learned optimizer that integrates a trainable preconditioning unit into the SR1 framework. It enables data-driven curvature estimation without requiring annotated data or supervised meta-training.
PGSM Mechanism: The introduction of a projection-guided secant mechanism that enforces PSD properties through rank-one parameterization and a meta-training secant penalty. This preserves the core quasi-Newton structure without incurring extra inference costs.
Empirical Validation: Demonstration that L-SR1 can be integrated into optimization-based HMR pipelines, consistently outperforming both classical solvers (like LBFGS) and existing learned optimization baselines (like LGD) in terms of convergence speed and generalization.

4. Experimental Results

The authors evaluated L-SR1 on analytic benchmarks and a real-world HMR task.

Analytic Benchmarks:
- Quadratic Functions: L-SR1 with PGSM demonstrated superior convergence and alignment with Newton directions compared to variants without PGSM. It showed robust generalization across problem dimensions (e.g., training on $N=2$ and testing on $N=10$ ) without re-training.
- Standard Benchmarks: On a suite of challenging functions (Rosenbrock, Rastrigin, Quadratic), L-SR1 with PGSM achieved the highest performance profiles, outperforming LBFGS, Adam, and AdaHessian across dimensions 50–1000.
- Buffer Sensitivity: Performance improved monotonically with larger evaluation buffers, and the model remained robust to variations in training buffer sizes.
Monocular Human Mesh Recovery (HMR):
- Evaluated on the 3DPW dataset, L-SR1 was integrated into the HMR pipeline of Song et al. (2020).
- Performance: L-SR1 achieved a PA-MPJPE of 51.74 (at 4 steps) and 51.58 (at optimal horizon), outperforming the learned baseline LGD (55.90) and classical SMPLify (106.80).
- Efficiency: L-SR1 achieved these results with a smaller model size and lower wall-clock inference time (364ms for 4 steps vs. 664ms for LGD) compared to the learned baseline.
- Generalization: The method required no task-specific fine-tuning and was trained exclusively on the AMASS dataset, yet generalized effectively to the in-the-wild 3DPW test set.

5. Significance and Claims

The paper claims that L-SR1 offers a compelling path to bridging classical optimization techniques with modern deep learning. Its significance lies in:

Efficiency and Generalization: It provides a compact, general-purpose optimization framework that accelerates convergence and reduces the required number of iterations without relying on task-specific fine-tuning or large annotated datasets.
Second-Order Learning: It advances the field of learned optimization by successfully integrating learnable components with second-order methods, addressing the stability and PSD constraints that have historically limited such approaches.
Practical Applicability: By demonstrating success on the high-dimensional, non-convex problem of HMR, the authors argue that L-SR1 is suitable for a broad range of iterative optimization problems where computational efficiency and data scarcity are critical constraints.

The authors maintain a modest tone, noting that while L-SR1 shows strong generalization and performance, its behavior is established empirically through meta-training and experiments, particularly regarding the descent guarantees of the composite linear operator in the learned setting.