🤖 machine learning

Geometric--Nongeometric Optimizer Calculus: A Modular Language for Reachable Gradient Methods

This paper introduces a modular "geometric–nongeometric optimizer calculus" that decomposes adaptive optimizers into distinct components to formally analyze their reachable gradient directions, establish expressivity theorems for various metric families, and frame optimizer design as a Pareto optimization problem rather than a search for a single universal solution.

Original authors: Zavier Li

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zavier Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guide a hiker down a mountain to find the lowest valley (the best solution). In the world of computer science, this "hiker" is an algorithm trying to solve a problem, and the "mountain" is a complex mathematical landscape.

For years, researchers have built different sets of rules (optimizers) to tell the hiker which way to step. Some say, "Run fast!" (Momentum). Others say, "Check the ground carefully before stepping!" (Adam). But these rules are often mixed together in a confusing soup, making it hard to know why a specific step was taken or if it was actually a good idea.

This paper, "Geometric–Nongeometric Optimizer Calculus," proposes a new way to organize and audit these rules. Think of it as a modular language or a recipe book that separates the "cooking" from the "ingredients."

Here is the breakdown using simple analogies:

1. The Two Main Ingredients: The Map and the Extras

The authors split every optimization method into two distinct parts:

The Geometric Module (The Map): This is the core "compass." It looks at the current slope (the gradient) and draws a straight line pointing downhill. If the map is perfect (a full, detailed map), it can point in any downhill direction.
- Analogy: Imagine a GPS that gives you the perfect, direct route down the hill.
The Nongeometric Modules (The Backpack & The Crew): These are the seven other things that influence the hiker but aren't part of the map itself:
1. Information: What data do we have? (Are we looking at the whole mountain or just a blurry photo?)
2. Memory: Did we remember where we stepped last time? (Momentum).
3. Control: How big of a step should we take? (Learning rate).
4. Operator: Are we hitting a wall and need to bounce off? (Constraints/Projections).
5. Noise: Did a gust of wind push us sideways? (Randomness).
6. Target: Are we actually trying to reach a slightly different valley than the one we started at? (Changing the goal).
7. Discretization: Are we taking giant, clumsy steps because we can't walk smoothly? (Computer limitations).

2. The "Audit" Question

The paper asks a very specific question: "Can we explain this specific step the hiker took just by looking at the Map?"

If Yes: The step was a pure "geometric" move. The map told the hiker exactly where to go.
If No: There is a "Residual." This means the step was influenced by the backpack (memory), the wind (noise), or a change in the goal (target).

The authors prove a cool mathematical fact: If you have a perfect, full map, you can explain any step that goes downhill. But if you have a restricted map (like a map that only shows North-South and East-West lines, but not diagonals), you can only explain steps that fit those lines. If the hiker tries to go diagonally, the map fails, and you have to blame the "backpack" or "wind" for that extra movement.

3. The "Budget" Concept

The paper argues that you can't just say "The best optimizer is the one with the perfect map." Why? Because a perfect map is expensive to draw and carry.

The Trade-off: You have a limited budget for memory, computing power, and time.
The Goal: You don't want the "best" optimizer in a vacuum; you want the best Pareto optimizer. This means finding the sweet spot where you get the best results without breaking your budget.
- Analogy: A Ferrari is faster than a bicycle, but if you only have $5 and need to go 1 mile, the bicycle is the better "optimizer" for your budget.

4. What They Actually Tested (The "Prototypes")

The authors didn't claim to have built the world's fastest AI. Instead, they built diagnostic tools (like a mechanic's scanner for cars) to see how these rules work in small, controlled settings:

The "Perfect Map" Test: On simple, smooth hills (mathematical quadratics), they showed that if you give the algorithm enough data to draw a perfect map, it solves the problem instantly.
The "Real World" Test: They looked at a small, standard AI task (recognizing handwritten numbers). They used their new language to "audit" popular methods like Adam and a new experimental method called "Muon."
- They found that some steps taken by these methods couldn't be explained by the map alone. This proved that those methods were relying heavily on "memory" or "noise" to move forward.
- They showed that their new language can tell you exactly how much of an algorithm's behavior is "smart geometry" and how much is just "memory tricks."

Summary

This paper is a theoretical framework, not a new super-optimizer. It provides a new vocabulary and a set of math tools to:

Separate the "geometry" (the map) from the "tricks" (memory, noise, etc.).
Measure how much of an algorithm's success is due to a good map versus other factors.
Design better algorithms by balancing the cost of the map against the budget of memory and time.

The authors are very clear: They are not claiming their method is the new state-of-the-art for training giant AI models. Instead, they are offering a "ruler" to measure and understand the tools we already use, helping researchers design better tools in the future by knowing exactly what each part of the tool is doing.

Technical Summary: Geometric–Nongeometric Optimizer Calculus

Problem Statement

Current descriptions of gradient-based optimizers (e.g., Adam, Shampoo, K-FAC) often conflate distinct algorithmic roles. They mix the definition of a descent geometry (mapping gradients to directions) with mechanisms for information gathering, memory, step-size control, noise injection, target modification, and discretization. This conflation makes it difficult to audit whether a specific update direction is genuinely explained by a geometric preconditioner or if it relies on residual mechanisms like momentum, projection, or stochasticity. Furthermore, there is a lack of a formal language to decompose "reachable" gradient methods under explicit constraints regarding oracle access, memory budgets, and state classes.

The paper addresses the question: Given an oracle class, a budget, a state class, and admissible rules, can reachable gradient methods be decomposed into a positive geometric module and structured nongeometric modules in a way that exposes expression gaps, mechanism budgets, and optimizer-design tradeoffs?

Methodology: The Geometric–Nongeometric Optimizer Calculus

The authors propose a modular language, the Geometric–Nongeometric Optimizer Calculus, which separates optimizer components into two distinct categories:

1. The Geometric Module

This module consists of a positive cometric family $\mathcal{P}_F(\theta)$ . It maps covectors (gradients) to parameter-space descent directions.

Formalism: For a covector $\alpha_\theta = df_\theta$ , the geometric descent direction is $u_{geo} = -P_\theta \alpha_\theta$ , where $P_\theta \succ 0$ .
Families: This includes full SPD matrices, diagonal matrices, block-diagonal, Kronecker-factored, low-rank, or proxy curvature families.

2. The Nongeometric Modules

The remaining mechanisms are split into seven explicit modules:

Information: Gradient estimates, mini-batches, curvature proxies, and Hessian-vector probes.
Memory: Momentum, Nesterov lookahead, secant pairs, and historical subspaces.
Control: Learning rates, damping, trust regions, line search, and schedules.
Operator: Projections, proximal maps, clipping, normalization, and decoupled weight decay.
Noise: Explicit sampling noise, Langevin perturbations, and dropout-induced noise.
Target: Regularized, smoothed, or surrogate objectives differing from the original $f$ .
Discretization: Euler/implicit steps, finite precision, and approximate matrix roots.

3. Audit and Decomposition

The calculus employs a canonical audit order to decompose a real optimizer:

Fix the target objective $f_k$ and the visible covector $\tilde{g}_k$ .
Within the declared family $\mathcal{F}$ , choose $P_k$ to best explain the observed direction via positive geometry.
Attribute scale and damping to Control.
Attribute the remaining residual to Memory, Operator, Noise, Target, or Discretization.

Key Contributions

1. Direction-Expressivity Theorems

The paper proves that full positive-definite geometry expresses exactly the set of strict descent directions away from critical points.

Theorem: For a nonzero gradient $g$ and direction $u$ , there exists a positive definite $P$ such that $u = -Pg$ if and only if $g^\top u < 0$ .
Implication: Any update that is not a strict descent direction (e.g., neutral or ascending steps caused by momentum) or any motion at a critical point ( $g=0$ ) must originate from a nongeometric module.

2. Restricted Direction Residuals

For implementable optimizers using restricted families (e.g., diagonal or block), the authors define the restricted direction residual:
$\rho_{\mathcal{F},h}(u; \alpha) = \inf_{P \in \mathcal{F}(\theta)} \|u + P\alpha\|_h$

This measures how much of an update is unexplained by the chosen geometry.
The paper derives exact expressivity conditions for diagonal and block geometries, showing that diagonal geometry is coordinate-dependent and fails to explain directions where signs of $g_i$ and $u_i$ do not align.

3. Diagnostic Framework (The Four Quadrants)

The authors separate two diagnostics:

Curvature Reachability: Measures how well a family can handle the condition number of the Hessian.
Direction Residual: Measures how well the family explains the actual update direction.
These form a four-quadrant diagnostic tool to distinguish between geometric preconditioning, memory effects, noise, and target modifications.

4. Trajectory-Level Residual Complexity

Moving beyond pointwise analysis, the paper defines trajectory-level residual complexity. This asks whether an entire training trace can be explained by a coherent path of preconditioners with bounded geometric variation.

It introduces a coherence gap to measure the cost of maintaining temporal consistency in the geometric explanation.
This prevents rich families from explaining every step via unrelated or singular matrices.

5. Pareto Optimization Formulation

Optimizer design is reframed not as finding a single "best" ordering, but as a constrained Pareto optimization over module budgets.

Designers must balance performance against budgets for geometry (e.g., affine-invariant metric length), information (gradient calls), memory, and communication.
The paper proves a budgeted class monotonicity: while full-SPD geometry is an expression superset of restricted geometry, the budget-feasible optimum may favor a cheaper, restricted family.

Results and Prototypes

The paper includes diagnostic prototypes to demonstrate the language, explicitly stating these are not claims of state-of-the-art performance on large-scale benchmarks.

Full-Metric Probe (Deterministic Quadratics): A prototype using full SPD inverse metrics (reconstructed via coordinate probes) solves deterministic quadratic benchmarks to numerical precision, validating the "expression envelope" of full geometry.
Krylov and Hybrid Variants: A Krylov-subspace variant and a residual-selected BFGS hybrid were tested. The results showed that limited information budgets and residual selection rules significantly impact performance, often preventing restricted geometries from matching full-geometry capabilities.
Muon-Style Candidate: A prototype combining matrix-orthogonalized momentum (for tensors) and AdamW (for vectors) was audited.
- On a small MNIST subset, it achieved ~77.0% accuracy vs. ~72.6% for AdamW.
- The audit revealed that the Newton–Schulz orthogonalization acts as a matrix-operator/memory mechanism rather than a pure positive cometric applied to the current gradient.
Trace Audits: The calculus was applied to short training traces of AdamW, SGD/Nesterov, and the Muon-style candidate.
- Finding: SGD/Nesterov showed strong pointwise diagonal geometric explanations but a large "coherence gap" (requiring rapidly changing metrics). AdamW and the Muon candidate showed lower raw residuals and smaller coherence gaps, suggesting more stable geometric shadows.

Significance and Claims

The paper positions itself as a theory and benchmark-language manuscript. Its primary significance lies in:

Providing a Formal Audit Language: It offers a rigorous framework to separate "geometry" from "engineering" in optimizers, allowing researchers to quantify how much of an update is geometrically explainable versus how much relies on memory, noise, or target transformations.
Redefining Optimizer Design: It shifts the focus from searching for a universal optimizer to solving a Pareto optimization problem over explicit resource budgets (compute, memory, oracle calls).
Clarifying Limits: It establishes that full positive-definite geometry is an "expression envelope" that strictly captures descent directions, while all other behaviors (momentum, projection, etc.) are structurally distinct nongeometric mechanisms.

The authors explicitly state that the paper does not claim large-scale state-of-the-art performance. The experimental results are small-scale and diagnostic, intended to illustrate the utility of the calculus in attributing optimizer behavior rather than to compete with tuned baselines like AdamW, Lion, or Shampoo on modern vision or language benchmarks.