🤖 machine learning

No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training

This paper demonstrates that the low-rank gradient subspace assumed to be trackable by memory-efficient optimizers like GaLore is fundamentally non-identifiable due to high estimator noise, revealing that performance gains instead stem from correctly transporting optimizer state across subspace refreshes rather than from the subspace's stability.

Original authors: Noel Thomas

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Noel Thomas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Chasing a Ghost

Imagine you are trying to teach a giant robot (a Large Language Model) to speak. To do this efficiently, the robot uses a special trick called GaLore.

The trick works like this: Every few minutes, the robot looks at the "mistakes" it just made (the gradients) and tries to find the top 128 directions where those mistakes are happening. It then ignores everything else and only learns in those 128 directions. The assumption is that these 128 directions are like a slow-moving river; they drift a little bit, but they stay mostly the same, so the robot can "track" them.

The paper's main discovery is shocking: The river isn't flowing slowly. It's not even a river. It's a waterfall that completely changes its shape every single time you look at it.

The authors prove that beyond a tiny core of about 39 directions, the "top 128 directions" the robot picks are essentially random noise. If the robot picks a set of directions now, and picks another set 10 seconds later, the two sets will be almost completely different (like two people picking 128 random numbers from a hat and getting almost no matches).

Why Does GaLore Work Then?

You might ask: "If the directions keep changing randomly, why does the robot still learn?"

The paper explains that GaLore works not because it is tracking a specific path, but because it is catching energy.

The Analogy: Imagine you are trying to catch rain in a bucket. You don't need to know exactly where every single drop will fall. You just need to hold the bucket in the general area where the rain is falling.
Even though the "top 128 directions" change completely every time, they still capture about 67–73% of the total "energy" (the useful information) of the mistakes. So, the robot keeps learning, even though it is constantly switching its map.

Why Averaging Doesn't Help

A natural idea to fix the "changing map" problem is: "Let's just look at the last 10,000 maps and average them to get a stable one."

The authors tested this and found it doesn't work.

The Analogy: Imagine the rain isn't just random noise; it's a specific pattern of wind and clouds (a signal). If you try to average the rain over time, you don't get a clear picture of where the rain is; you just get a blurry mess.
The "noise" in the robot's mistakes isn't just random static; it's a complex signal that shrinks very slowly when you try to average it out. No matter how much you average, you can never create a stable "top 128" list because the list doesn't truly exist in a stable form.

The Real Problem: The Robot's Memory

The real issue isn't the map; it's the robot's memory.
The robot uses an optimizer called Adam, which has a long-term memory (it remembers mistakes from the last 1,000 steps).

The Problem: Every time the robot refreshes its map (every 160 steps), the map rotates 90 degrees. But the robot's memory is still pointing at the old map. It's like driving a car where the steering wheel suddenly turns 90 degrees, but your hands are still holding the wheel in the old position. The robot is trying to steer based on a map that no longer exists.

The Solutions Found in the Paper

The authors tested two ways to fix this "steering wheel" problem:

Rotate the Memory (Transport): Instead of leaving the memory stuck in the old position, the robot should physically rotate its memory to match the new map.
- Result: This works very well. It's like realizing the steering wheel turned and immediately turning your hands to match it.
Shorten the Memory: The robot's memory is too long (remembering 1,000 steps back). Since the map changes every 160 steps, remembering 1,000 steps is useless because those old steps are from a completely different world.
- Result: By telling the robot to only remember the last 100 steps (instead of 1,000), it stops trying to steer based on outdated maps. This also works well.

The Takeaway for Builders

If you are building a system that uses these low-rank tricks, here is the advice from the paper:

Don't try to track the subspace. It's a ghost. It changes too fast to be tracked.
Check your "Reproducible Rank" ( $k^*$ ). Before you trust a rank of 128, check how many directions are actually real. The paper found that only about 39 are real; the rest are noise.
If you use a rank higher than the real number:
- Rotate your memory: Make sure your optimizer's memory moves with the new map.
- Shorten your memory: Don't let the robot remember too far back.
- Stop averaging: Don't waste time trying to average the maps to make them stable; it won't work.

In short: The "low-rank subspace" isn't a stable object you can follow. It's a rapidly changing cloud. The trick to making these systems work isn't to follow the cloud better; it's to stop trying to follow it and instead adjust your memory to handle the constant changes.

Technical Summary: "No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training"

Problem Statement

Memory-efficient optimizers like GaLore (Zhao et al., 2024) reduce the memory footprint of training large language models by projecting gradients onto a periodically recomputed low-rank subspace (rank $r$ ) and running Adam within that subspace. The underlying geometric premise of this approach is that the top- $r$ gradient subspace is a meaningful, slowly drifting object that can be tracked across refresh intervals ( $T$ steps). If this premise held, consecutive estimates of the subspace would be similar, and carrying optimizer state (moments) across a refresh would require only minor corrections.

This paper challenges that premise, arguing that beyond a small reproducible core, the top- $r$ subspace is not a stable object but is statistically non-identifiable. Consequently, the apparent "rotation" of the subspace at each refresh is dominated by estimator noise rather than the motion of a stable geometric structure.

Methodology

The authors employ a rigorous measurement framework to quantify subspace stability and the impact of non-identifiability on optimizer performance:

Subspace Rotation Measurement: The paper introduces a sign-corrected chordal distance metric ( $\|\Delta U\|_F$ ) to measure the rotation between the old basis ( $U_{old}$ ) and the new basis ( $U_{new}$ ) at each refresh. This metric ranges from 0 (identical) to $\sqrt{2r}$ (orthogonal).
Same-Step Control (The "No Time" Test): To distinguish between genuine temporal drift and estimator noise, the authors split a single minibatch into two disjoint halves at the same training step. They compute the top- $r$ subspace for each half and measure the disagreement. If the subspace were a stable property of the gradient, same-step estimates should agree. If they disagree as much as estimates taken $T$ steps apart, the rotation is noise.
Spectral Analysis: The authors analyze the singular value spectrum of gradients to determine if a spectral gap exists at rank $r$ that would justify a stable subspace cut.
Averaging Experiments: The paper tests whether averaging gradients over $N$ steps (or averaging the basis over time) can "rescue" identifiability by opening a spectral gap.
Remedy Evaluation: The authors test three strategies to mitigate the costs of subspace redraws:
- Basis Averaging: Using an exponential moving average (EMA) of the covariance matrix to stabilize the basis.
- State Transport: Explicitly transporting optimizer moments (first and second) through the rotation matrix $R = U_{new}^\top U_{old}$ (as done in LDAdam).
- Memory Shortening: Reducing the Adam decay parameter $\beta_2$ to shorten the memory of the second moment, preventing it from integrating stale statistics from previous subspaces.

Experiments were conducted across four model families (Pythia, GPT-2, Qwen2.5, Llama-3) ranging from 70M to 6.9B parameters, and a Vision Transformer (ViT-B).

Key Findings

1. The Subspace is Non-Identifiable

Maximal Rotation: At every refresh, the measured rotation $\|\Delta U\|_F$ saturates at 96.7–99.6% of the geometric maximum $\sqrt{2r}$ . The new basis is nearly orthogonal to the old one.
Noise Dominance: The same-step control reveals that two estimates from disjoint halves of the same batch disagree by $0.725\sqrt{2r}$ , which is nearly identical to the across-time rotation of $0.742\sqrt{2r}$ . The "rotation" is not motion; it is estimator noise.
Reproducible Core ( $k^\star$ ): Only a small fraction of directions are reproducible. For $r=128$ , only $k^\star \approx 39$ directions are consistent across minibatches. The remaining $r - k^\star$ directions are effectively random draws from a gapless spectral tail.

2. Spectral Causes and the Failure of Averaging

Gapless Spectrum: The gradient spectrum consists of a small dominant spike (a few to ~20 directions) followed by a smooth, near-isotropic tail with no gap at rank $r$ . The adjacent singular-value ratio at $r=128$ is $\approx 1.005$ .
Signal, Not Noise: The deep spectral tail is not sampling noise; it is signal with a power-law spectrum. Under $N$ -fold averaging, this tail shrinks only as $N^{-1/4}$ rather than the $N^{-1/2}$ expected for pure noise. Therefore, no finite averaging budget can open a gap or stabilize the subspace.
Averaging Fails: Averaging the basis over 10,000 steps reduces the per-refresh rotation only slightly (from 97% to 84% of the maximum) and has no measurable effect on final perplexity (a span of only 0.36).

3. Why GaLore Still Works

Despite the lack of a stable subspace, GaLore works because the projection's primary role is energy capture, not tracking. Since the gradient tail is near-isotropic, any random $r$ -dimensional frame that includes the dominant spike captures a roughly constant fraction of the gradient energy ( $\gamma_g \approx 0.67–0.73$ ). The specific choice of directions matters little for the descent direction, provided the energy fraction is preserved.

4. Effective Remedies

The cost of the redraw is the staleness of the carried optimizer state, particularly the second moment (variance).

State Transport: Transporting the first moment via $R$ and the second moment via the squared entries of $R$ is optimal. Blindly carrying the second moment is provably suboptimal by a factor of $\approx (r - k^\star)/2$ . In controlled tests, transporting state at the default $\beta_2=0.999$ achieved 18.73 perplexity, beating untransported GaLore (19.28).
Shortening Memory: Reducing $\beta_2$ from 0.999 to 0.99 helps refreshing optimizers by preventing the integration of stale statistics from previous, now-invalid subspaces. This improved perplexity by 1.8 to 26 points depending on the optimizer and recipe.
Ineffectiveness of Basis Averaging: As noted, stabilizing the basis via averaging does not work because the subspace is fundamentally non-identifiable.

Significance and Claims

The paper claims to provide the first direct measurement of the geometric premise underlying low-rank gradient optimizers, finding it false at the ranks typically used.

Clarification of Mechanism: The paper clarifies that GaLore works not because it tracks a stable structure, but because it captures a constant fraction of gradient energy in a frame that is effectively re-drawn.
Diagnostic Tool: The authors propose a practical probe: measure the reproducible rank $k^\star$ by splitting a batch. If the chosen rank $r > k^\star$ , the excess directions are noise.
Engineering Prescription: The paper prescribes two actionable fixes for low-rank training:
1. Transport state through the rotation (as in LDAdam) rather than carrying it blindly.
2. Lower $\beta_2$ to 0.99 to reduce the memory of the second moment, mitigating the staleness caused by periodic redraws.
Scope: The findings apply specifically to optimizers that explicitly compute and refresh a gradient subspace (GaLore family). They do not apply to parameterizations like LoRA that do not compute a gradient subspace.

The authors conclude that the "object" (the stable low-rank subspace) does not exist beyond a small core, and attempts to track it are chasing a phantom. The path forward lies in acknowledging the redraw and adapting the optimizer state accordingly.