🤖 AI

Quotient Dynamics, Effective Curvature, and Implicit Bias in Positive Quadratic Networks

This paper analyzes the training dynamics, curvature, and implicit bias of positive quadratic networks by leveraging their quotient structure on the rank-r PSD manifold to demonstrate how factor gradient flow and descent converge to specific interpolants, such as minimum-trace solutions, through exact projections to Riemannian flows and entropy-based mirror dynamics.

Original authors: Pengcheng Cheng

Published 2026-07-29

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Pengcheng Cheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a giant puzzle, but you have a secret shortcut: instead of moving the final picture, you are only allowed to move the pieces that make the picture. In the world of machine learning, this is exactly what happens when we train "overparameterized" models. These are smart algorithms that have way more moving parts (parameters) than they actually need to describe the final answer. It's like trying to describe a perfect circle by juggling a thousand invisible strings; many different ways of holding the strings can result in the exact same circle. The big question scientists have been asking is: when the computer learns by adjusting these strings, which specific circle does it actually choose? Does it pick the simplest one? The most balanced one? Or does it just stumble into a random shape?

This paper dives deep into a specific type of puzzle called "positive quadratic networks." Think of these as a special kind of math machine that takes an input (like a number or a list of numbers) and squares it in a fancy way to make a prediction. The researchers realized that the "strings" holding this machine together have a hidden geometric structure, like a spinning top that looks the same no matter how you rotate it. They wanted to understand how the machine's learning process (gradient descent) behaves when it's forced to navigate this spinning, redundant landscape. By treating the problem as a journey on a curved surface where redundant moves are ignored, they discovered that the machine doesn't just wander aimlessly. Instead, it follows a very specific, predictable path that reveals a hidden bias: a tendency to choose solutions that are "small" in a very specific mathematical sense, often picking the solution with the smallest total size (trace) or the one that balances entropy in a unique way.

The Secret Dance of Redundant Strings

Let's start with the core mystery. Imagine you have a machine that predicts the weather based on temperature and humidity. To build this machine, you use a factor $U$ , which is like a set of dials. The machine's actual prediction, $Q$ , is made by squaring these dials together ( $Q = UU^\top$ ). Here's the catch: there are infinite ways to set the dials to get the exact same prediction. If you spin the dials in a specific way (multiplying by an orthogonal matrix), the prediction $Q$ doesn't change at all. It's like having a Rubik's cube where you can twist a whole face without changing the color of the center piece.

The paper proves that this isn't just a coincidence; it's a fundamental geometric rule. The space of all possible dials is huge, but the space of actual predictions is a smaller, smoother surface called a "quotient manifold." The researchers showed that when you train the machine using standard methods (Euclidean gradient flow), the dials move in a way that perfectly aligns with the geometry of this prediction surface. The "redundant" spinning motion is naturally filtered out. It's as if the learning algorithm has an internal compass that only cares about moving the prediction forward, ignoring the useless spinning of the dials.

The Invisible Map and the Speed of Learning

One of the coolest findings is about how fast the machine learns. Usually, when we look at how fast an algorithm converges, we look at the "curvature" of the landscape—how steep the hills are. But because of the redundant dials, the landscape looks weirdly flat in some directions. The authors invented a new kind of map called the "effective curvature." This map ignores the flat, useless directions and only measures the steepness of the directions that actually change the prediction.

They found that this effective curvature perfectly predicts how fast the machine learns. In their experiments, they changed the "steepness" of the problem and watched the learning speed. The results were spot-on: the machine slowed down exactly as much as the new map predicted. It's like driving a car on a road with invisible potholes; the paper figured out that the car's speed is determined not by the road's surface, but by a hidden map of the potholes that only affects the steering wheel, not the engine.

The Magic of "Small" Starts and the Entropy Tie-Breaker

Now, let's talk about what happens when the puzzle isn't fully solved. Imagine you have a few clues about the weather, but not enough to know the exact temperature. There are infinite possible answers that fit the clues. Which one does the machine pick?

The paper reveals a fascinating rule: how you start matters. If you start the machine with the dials set to a tiny, uniform value (a "small initialization"), the machine has a strong bias toward picking the solution with the minimum trace. In plain English, "trace" is a way of measuring the total "size" or "energy" of the prediction. The machine naturally gravitates toward the smallest, most compact solution that fits the data.

But what if there are multiple solutions that are all equally small? The machine doesn't just pick one at random. It uses a tie-breaker based on entropy, which is a measure of disorder or randomness. The paper shows that the machine picks the solution that is the most "balanced" or "spread out" among the smallest options. It's like having a pile of sand that you want to make as small as possible; if you can't make it smaller, you spread it out as evenly as possible so no single grain is too heavy.

The researchers proved this mathematically for a specific type of problem where the clues (measurements) all "commute," meaning they can be solved simultaneously without fighting each other. In this scenario, the learning process is exactly equivalent to a "mirror flow," a fancy mathematical dance that minimizes a specific type of distance (Bregman divergence) from the starting point.

The Gap Between Theory and Reality

While the math is beautiful, the paper is also very honest about its limitations. The authors derived a formula for how many data points are needed to guarantee the machine finds the right answer. However, they admit this formula is extremely conservative. It's like a safety manual that says, "To cross this bridge, you need a million people holding hands," when in reality, the bridge holds up with just ten.

In their experiments, the machine successfully learned and found the correct solution with far fewer data points than the theory required. The theory is a "sufficient" guarantee (it works if you have this much), but it's not "necessary" (you might get away with less). The paper explicitly states that their sample size requirement is not the best possible one and relies on a "worst-case" scenario analysis. They also note that their neat entropy tie-breaking rule only works when the clues commute; for more chaotic, non-commuting problems, the rule might not hold.

The Finite Step: When the Dance Gets Stuttery

Finally, the paper looked at what happens when the machine doesn't learn in a smooth, continuous flow but takes tiny, discrete steps (like a video game character moving frame by frame). They found that the final answer the machine picks is very close to the smooth, continuous answer, but with a small error. This error is proportional to the step size ( $\eta$ ). If you take smaller steps, the answer gets closer to the "perfect" continuous solution. It's like walking toward a target; if you take giant strides, you might overshoot or land slightly off, but if you take tiny steps, you land almost exactly where the smooth path would have taken you.

The Takeaway

This paper doesn't just say "machine learning works"; it explains why it works in a very specific, geometric way. It shows that the way we represent a problem (the dials) and the way we train it (the gradient flow) are deeply connected. The machine isn't just minimizing error; it's navigating a curved, redundant landscape that naturally guides it toward simple, balanced solutions. While the math provides a rigorous map for this journey, the real-world experiments show that the machine is even more capable than the strictest theories predict, finding the right answers with less data and fewer steps than the "safety manual" suggests. It's a story of hidden geometry, natural biases, and the surprising elegance of how machines learn.

Technical Summary: Quotient Dynamics, Effective Curvature, and Implicit Bias in Positive Quadratic Networks

1. Problem Statement

The paper investigates the training dynamics and implicit bias of positive quadratic networks, a specific class of overparameterized models defined by the factorization:
$f_U(x) = \sum_{j=1}^r (u_j^\top x)^2 = x^\top UU^\top x$
where $U \in \mathbb{R}^{d \times r}$ is the parameter matrix and the realized predictor is the rank- $r$ positive semidefinite (PSD) matrix $Q = UU^\top$ .

The core challenge addressed is the non-identifiability of the factor $U$ . The mapping $U \mapsto UU^\top$ is invariant under right multiplication by any orthogonal matrix $R \in O(r)$ (i.e., $U$ and $UR$ yield the same predictor). This creates a "gauge" redundancy where the ambient parameter space $\mathbb{R}^{d \times r}$ is significantly larger than the intrinsic predictor space. The paper seeks to answer:

How does this quotient structure govern the dynamics of standard Euclidean gradient methods?
What is the effective curvature governing local convergence once gauge directions are removed?
In underdetermined settings, which specific interpolant does the factor dynamics select (implicit bias)?

2. Methodology

The authors employ a rigorous geometric framework based on quotient manifolds and Riemannian optimization, applied directly to the ordinary Euclidean training of the factor $U$ .

A. Exact Quotient Geometry

The study restricts attention to the full-column-rank stratum $\mathcal{F}_{d,r} = \{U \in \mathbb{R}^{d \times r} : \text{rank}(U)=r\}$ .

Fiber Identification: It is proven that $UU^\top = VV^\top$ if and only if $V = UR$ for a unique $R \in O(r)$ . Thus, the predictor space $\mathcal{M}_r^+$ (rank- $r$ PSD matrices) is diffeomorphic to the quotient manifold $\mathcal{F}_{d,r} / O(r)$ .
Horizontal/Vertical Decomposition: The tangent space of the factor space is decomposed into vertical directions (tangent to the orthogonal orbit, representing gauge changes) and horizontal directions (orthogonal to the orbit under the Frobenius metric).
Key Insight: The Euclidean gradient of any objective $L(U) = \ell(UU^\top)$ is horizontal. Consequently, Euclidean gradient flow in the factor space projects exactly to the Riemannian gradient flow on the quotient manifold $\mathcal{M}_r^+$ .

B. Effective Curvature and Recovery

For quadratic regression with Gaussian rank-one measurements, the authors derive:

Effective Hessian: The Hessian of the loss restricted to the quotient manifold is the measurement Gram form restricted to the tangent space of the predictor, evaluated relative to the quotient metric. This removes orthogonal gauge redundancy while preserving genuine sample-level non-identifiability.
Population Operator: The population normal operator is computed explicitly as $T(H) = 2H + \text{tr}(H)I$ .
Concentration: A uniform deviation bound for the empirical operator is established over the entire ambient space $S_d$ (not just the tangent space), using a second-moment argument.
Initialization: A moment-based spectral initializer is constructed by truncating a moment matrix to its $r$ largest positive eigenvalues.

C. Implicit Bias in Commuting Regimes

In an underdetermined regime where measurement matrices commute (sharing a joint spectral decomposition), the dynamics are reduced to the joint spectral algebra.

Mirror Flow: The factor gradient flow is shown to be an exact entropy mirror flow in the spectral coordinates.
Selection Laws: The limit of the flow is the Bregman projection of the initialization onto the feasible set.
- Small Initialization: As the initialization scale $\epsilon \to 0$ , the selected predictor approaches the minimum-trace solution.
- Entropy Tie-Breaking: If the minimum-trace solution is non-unique, the limit selects the unique minimizer of a weighted entropy functional within the invariant algebra.
Discrete vs. Continuous: For finite-step gradient descent, the selected interpolant differs from the continuous Bregman projection by $O(\eta)$ .

3. Key Contributions

Exact Projection of Euclidean Dynamics: The paper establishes that ordinary Euclidean gradient flow on the factor $U$ is exactly horizontal and projects to the Riemannian gradient flow on the manifold of rank- $r$ PSD matrices. It also derives an exact congruence recursion for the predictor $Q_k$ under finite-step gradient descent, identifying the precise discretization error term.
Effective Curvature and Local Rates: It defines an effective Hessian relative to the quotient metric. The generalized eigenvalues of this Hessian describe the intrinsic linearized dynamics. The smallest effective eigenvalue is shown to govern the slowest local convergence rate of the factor trajectory.
Explicit (Conservative) Recovery Guarantee: Under Gaussian rank-one measurements, the paper provides an end-to-end sufficient condition for local recovery. This includes a spectral initializer, a full-space concentration bound, and local exponential convergence guarantees for gradient flow and linear convergence for gradient descent (with an oracle step size).
Explicit Interpolation Principles: In the commuting underdetermined regime, the paper derives explicit selection laws:
- Finite positive initialization $\implies$ Bregman projection.
- Isotropic small initialization $\implies$ Minimum-trace solution.
- Non-unique minimum-trace face $\implies$ Weighted entropy tie-breaking.
- Finite step size $\implies$ $O(\eta)$ deviation from the continuous limit.

4. Results and Numerical Verification

The paper validates its theoretical mechanisms through numerical experiments:

Quotient Identities: Verified that orthogonally equivalent initializations yield identical predictor trajectories (up to floating-point precision) and that the exact predictor recursion holds.
Curvature Prediction: The measured local decay rate of gradient flow matches the smallest effective eigenvalue with relative errors $< 2.4 \times 10^{-4}$ .
Recovery Conservatism: While the theoretical sample complexity bound is extremely conservative (scaling as $O(r^2 d^8)$ due to full-space concentration), empirical recovery succeeds at sample sizes orders of magnitude smaller. This confirms the qualitative mechanism while highlighting the looseness of the quantitative constants.
Selection Laws: Experiments confirm that gradient flow converges to the Bregman projection, small initialization leads to minimum-trace solutions, and finite-step descent introduces an $O(\eta)$ error consistent with theory.

5. Significance and Claims

The paper claims to provide a closed chain of reasoning from the exact quotient geometry to concrete training dynamics and implicit bias, without relying on heuristic assumptions.

Modesty in Claims: The authors explicitly state that their recovery bounds are deliberately conservative. They do not claim statistical optimality or that the sample complexity scales with the intrinsic dimension of the manifold. The bounds rely on full-space concentration rather than restricted isometry properties.
Scope Limitations:
- The smooth quotient analysis is restricted to the full-column-rank stratum; singular boundaries (rank-deficient factors) are not included.
- The explicit Bregman and entropy selection theorems require a commuting measurement algebra. No claim is made for arbitrary non-commuting matrix sensing.
- The equivalence between small-initialization limits of gradient flow and gradient descent is only established sequentially ( $\eta \to 0$ then $\epsilon \to 0$ ), not simultaneously.
Core Contribution: The primary significance lies in demonstrating that the quotient geometry is not just a theoretical abstraction but directly dictates the trajectory of standard Euclidean optimizers. It provides a precise language to describe how parameter redundancy is removed by the dynamics and how specific structural biases (trace, entropy) emerge from the interplay of initialization, step size, and the geometry of the predictor space.