🤖 AI

LoRDO: Distributed Low-Rank Optimization with Infrequent Communication

The paper proposes LoRDO, a distributed training framework that unifies low-rank optimization with infrequent communication by introducing a full-rank quasi-hyperbolic update to restore subspace exploration, thereby achieving near-parity with standard DDP performance while reducing communication overhead by approximately 10 times.

Original authors: Andrej Jovanović, Alex Iacob, Mher Safaryan, Ionut-Vlad Modoranu, Lorenzo Sani, William F. Shen, Xinchi Qiu, Dan Alistarh, Nicholas D. Lane

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Andrej Jovanović, Alex Iacob, Mher Safaryan, Ionut-Vlad Modoranu, Lorenzo Sani, William F. Shen, Xinchi Qiu, Dan Alistarh, Nicholas D. Lane

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a massive team of students (a computer model) to write a story. To do this, you have a giant library of books (data) and a classroom full of teachers (computers/workers).

In the standard way of doing this (DDP), every teacher reads a few pages, writes down their notes, and then immediately runs to the front of the room to share them with everyone else. They combine all the notes, update the lesson plan, and start again.

The Problem:
As the story gets more complex (the model gets bigger), the "notes" the teachers need to share become huge. The hallway connecting the classrooms (the network bandwidth) gets clogged. The teachers spend more time running back and forth sharing notes than actually learning.

To fix this, some researchers tried a new method: Low-Rank Optimization. Instead of writing down every single detail of their notes, teachers summarize them into a tiny, low-resolution sketch. This makes the notes much smaller to carry. However, there's a catch:

Local Sketches: If every teacher makes their own sketch based only on their own small pile of books, the sketches are noisy and inconsistent.
Global Sketches: If they wait until the end to make one "perfect" sketch based on everyone's books, the sketch becomes too rigid. The teachers get stuck looking at the same few ideas over and over, unable to explore new creative directions. The learning process "stagnates."

The Solution: LoRDO
The authors of this paper propose LoRDO (Distributed Low-Rank Optimization). Think of it as a clever new rule for the classroom that combines the best of both worlds.

Here is how LoRDO works, using a creative analogy:

1. The "Group Sketch" (Global Projection)

Instead of letting every teacher make their own messy sketch, the teachers wait until they have finished a block of reading. They then pool their notes together to create one single, high-quality "Group Sketch."

Why this helps: Because this sketch is based on the combined knowledge of the whole class, it is much clearer and less noisy than any individual teacher's sketch. It gives everyone a solid foundation to stand on.

2. The "Creative Spark" (Full-Rank Quasi-Hyperbolic Momentum)

Here is the paper's big innovation. If the teachers only used the "Group Sketch," they would all be forced to think in the same narrow lane. They would stop exploring new ideas, and the story would get boring (stagnate).

To fix this, LoRDO adds a "Creative Spark" to the update process.

Imagine that while the teachers are following the "Group Sketch," they are also allowed to add a tiny bit of their own full-resolution, wild imagination back into the mix.
This "spark" is a mathematical trick (called a full-rank quasi-hyperbolic momentum term) that injects a little bit of full-detail information into the low-detail sketch.
The Result: The teachers stay on the same page (using the efficient low-rank sketch) but are free to explore the entire library of ideas, not just the narrow path the sketch suggests.

3. The "Infrequent Check-ins"

LoRDO also lets the teachers work for a long time (many steps) before they have to run to the front of the room to sync up.

Because they are using the efficient "Group Sketch" and the "Creative Spark," they can work independently for much longer without getting lost.
This reduces the traffic in the hallway by about 10 times compared to the old methods.

What Did They Find?

The researchers tested this on language models ranging from small (125 million parameters) to medium-large (720 million parameters).

Performance: LoRDO performed almost exactly as well as the standard, slow, high-bandwidth method. The quality of the story (measured by "perplexity") was nearly identical.
Efficiency: It used 8 to 12 times less memory and reduced communication traffic by 10 times.
Low Memory Settings: When the computers had very little memory (forcing them to use very small "sketches"), LoRDO actually performed better than the standard methods because it handled the noise much better.

In Summary

LoRDO is like a smart classroom management system. It lets teachers work independently for longer periods to save time (communication). It uses a shared, high-quality summary to keep everyone aligned (Global Projection). But crucially, it adds a special "creative spark" that prevents the group from getting stuck in a rut, ensuring they can still explore all the best ideas in the library (Full-Rank Exploration).

The paper claims this allows us to train large AI models on hardware that is cheaper and more limited, without sacrificing the quality of the final result.

Technical Summary: LoRDO – Distributed Low-Rank Optimization with Infrequent Communication

1. Problem Statement

The distributed training of foundation models via Distributed Data Parallel (DDP) is increasingly constrained by interconnect bandwidth and the memory overhead required to store optimizer states (e.g., Adam momenta). While strategies employing infrequent communication (local updates) reduce synchronization frequency, they remain bottlenecked by the memory and communication costs of full-rank optimizer states.

Low-rank adaptive optimizers (e.g., GaLore, LDAdam) offer a path to alleviate these constraints by projecting gradients into a low-rank subspace. However, generalizing these methods to the infrequent-synchronization regime presents a critical optimization challenge:

Local Projections: Computing low-rank projections locally on individual workers using stochastic gradients introduces significant noise due to small effective batch sizes per worker. This leads to suboptimal projection subspaces and performance degradation.
Global Projections: Using a global projection matrix derived from aggregated pseudo-gradients (as in DDP) provides a more stable basis. However, in the local-update regime, this approach permanently restricts the optimization trajectory to a fixed low-rank subspace. Without full-rank signals, the optimization stagnates because the projection basis fails to "refresh" or explore new directions outside the initial subspace.

The core question addressed is: How can we design high-performance low-rank optimizers for communication-efficient training that avoid subspace stagnation while maintaining low memory and communication overhead?

2. Methodology: LoRDO

The authors propose LoRDO (Low-Rank Distributed Optimization), a framework that unifies low-rank optimization with infrequent synchronization. The methodology addresses the tension between projection stability and subspace exploration through three key mechanisms:

A. Global Projection Matrices

Instead of each worker determining its own projection matrix $Q_m$ based on local stochastic gradients (which suffers from high variance), LoRDO employs a global projection matrix $Q_t$ . This matrix is computed at the synchronization boundary using the aggregated pseudo-gradient ( $\Delta_t$ ), which represents the total change in model parameters over $K$ local steps.

Benefit: The pseudo-gradient has an effective batch size of $M \times B$ (where $M$ is the number of workers), yielding a more stable signal with reduced variance compared to local gradients. It also incorporates curvature information baked into the pseudo-gradient, leading to higher-quality basis estimation.

B. Full-Rank Quasi-Hyperbolic Momentum (QHM)

To resolve the "stagnant subspace" problem inherent in global projections, LoRDO injects a full-rank quasi-hyperbolic momentum term into the local update.

Mechanism: The update rule combines the low-rank momentum with a full-rank gradient signal scaled by the second momentum.
$\text{Update} \propto (1-\omega)\frac{\hat{G}_t}{\mu(\sqrt{\hat{v}_t} + \epsilon)} + \omega Q_t \left[ \frac{u_t}{\sqrt{v_t} + \epsilon} \right]$
Effect: This injection prevents the aggregated pseudo-gradient from remaining trapped within a fixed rank- $r$ subspace. It allows the optimization procedure to temporally aggregate rank- $r$ subspaces from different workers, enabling eventual complete subspace exploration. This restores the ability to explore the full solution space without increasing communication overhead.

C. Error Feedback and Momentum Rotation

LoRDO incorporates error feedback locally to account for information lost during low-rank compression. Additionally, following the computation of a new projection matrix, the optimizer states (momenta) are rotated to ensure they remain accumulated on the same subspace, a technique adapted from LDAdam.

3. Key Contributions

Analysis of Projection Failure Modes: The authors demonstrate that local projections harm performance due to high variance from small worker batch sizes, while global projections without full-rank signals induce subspace stagnation.
Restoring Full Subspace Exploration: LoRDO introduces a low-rank optimizer that injects a full-rank gradient signal (via QHM) into local updates while maintaining a global projection derived from aggregated pseudo-gradients. This prevents stagnation without increasing communication or memory overheads.
DDP Parity and Efficiency: The framework achieves near-parity with synchronous low-rank DDP (perplexity gap < 1%) while reducing optimizer memory and communication by up to $8\times$ – $12\times$ . Under heavy memory constraints necessitating low ranks, LoRDO even surpasses DDP in perplexity.

4. Experimental Results

Experiments were conducted on decoder-only transformers scaled to 16M, 125M, and 720M parameters using the SmolLM2 data mixture.

Stagnation vs. Exploration: In 16M models, global projections without QHM failed to learn (perplexity ~~46.8) compared to DDP (~~36.6), confirming subspace stagnation. Adding full-rank QHM restored performance, allowing LoRDO-Global to outperform local variants.
Projection Quality: LoRDO-Global demonstrated superior stability and performance, particularly at lower ranks ( $r=8, 16$ ), due to the higher-quality projection basis derived from the aggregated pseudo-gradient. It showed less sensitivity to changes in local batch size compared to local projection methods.
Scaling to 720M: At the 720M scale, LoRDO-Global ( $r=256$ ) achieved near-parity with low-rank DDP ( $r=256$ ) with a perplexity gap of less than 1% (10.41 vs. 10.34). It also matched downstream task accuracy (ARC, HellaSwag, MMLU, PIQA) with DDP.
Efficiency Gains:
- Communication: LoRDO reduces communication overhead by approximately 10 $\times$ compared to low-rank DDP and 25 $\times$ compared to full-rank DDP.
- Memory: It reduces optimizer state memory overhead by 8 $\times$ compared to full-rank counterparts.
- Wall-Clock Time: In low-bandwidth environments (modeled 10 Gbps), LoRDO significantly outperforms DDP in wall-clock time due to reduced synchronization frequency.

5. Significance and Claims

The paper claims that LoRDO resolves the fundamental tension between projection stability and subspace exploration in low-rank communication-efficient optimization. By utilizing aggregated pseudo-gradients for projection stability and full-rank quasi-hyperbolic momentum for exploration, LoRDO enables:

Scalable Training: Competitive performance at scale (up to 720M parameters) with significantly reduced communication and memory costs.
Resource Efficiency: Feasibility of pre-training on hardware with limited resources (low memory/bandwidth) where full-rank optimizers are intractable.
Decentralized Learning: Paving the way for efficient learning in decentralized environments by extending model training beyond single data centers.

The authors note limitations, including the lack of a formal convergence proof and the current focus on adaptive optimizers (Adam) rather than matrix-based optimizers (e.g., Muon), leaving these for future work. However, they posit that the observed trends in smaller models hold for larger scales.