🤖 machine learning

Unifying Local Communications and Local Updates for LLM Pretraining

The paper introduces GASLoC, a novel decentralized pre-training algorithm that unifies local updates with sparse, gossip-based peer communication to overcome the bandwidth and heterogeneity bottlenecks of synchronous All-Reduce methods, achieving performance competitive with or superior to state-of-the-art approaches like DiLoCo in both homogeneous and heterogeneous network settings.

Original authors: Pietro Cagnasso, Eugene Belilovsky, Edouard Oyallon

Published 2026-06-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pietro Cagnasso, Eugene Belilovsky, Edouard Oyallon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Slowest Worker" Bottleneck

Imagine you are trying to paint a massive mural with a team of 32 artists (these are the computer workers). To make sure the mural looks consistent, every artist has to stop every few minutes to compare their section with everyone else's. They do this by gathering in a circle, holding hands, and shouting out their colors so everyone averages them out. This is called All-Reduce.

In a perfect world, everyone paints at the same speed. But in the real world, some artists have slow hands, or their paint buckets are heavy, or their internet connection is spotty. In the current "shout-out" method, everyone has to wait for the slowest artist before they can take another step. If one artist is slow, the whole team stands around idle, wasting time.

The Old Solution: The "Gossip" Method

Researchers have tried a different approach called Decentralized Learning (or Gossip). Instead of gathering in a big circle, artists only talk to their immediate neighbors. Artist A talks to B, B talks to C, and so on. Information trickles through the group like a rumor.

The problem with this old gossip method is that it's often too slow to learn effectively. The "rumors" get distorted, and the team ends up painting a messy, inconsistent mural. Also, most existing gossip methods don't work well with the modern, complex tools (optimizers) we use to train giant AI brains today.

The New Solution: GASLoC

The authors of this paper introduce a new method called GASLoC. Think of it as a smart, flexible way to organize the painting team that combines the best of both worlds.

Here is how GASLoC works, broken down into three simple ideas:

1. The "Local Break" (Local Steps)

Instead of stopping to talk after every single brushstroke, the artists are allowed to take a "local break." They paint a whole section of the wall on their own (doing many local updates) before they are required to check in with anyone. This saves a lot of time because they aren't constantly stopping to chat.

2. The "Random Handshake" (Sparse Peer Communication)

When it is time to check in, they don't gather in a big circle. Instead, they use a randomized handshake system.

In one round, Artist A might shake hands with Artist B.
In the next round, Artist A might shake hands with Artist C.
They only talk to one or two people at a time, not the whole group.

This is much faster than the big circle. Even if Artist A is slow, they only have to wait for Artist B or C, not the whole team. The "rumor" still spreads across the whole group eventually, but it happens much more efficiently.

3. The "Momentum Coach" (Outer Optimizer)

This is the secret sauce. In the old gossip methods, the team just averaged their colors. GASLoC adds a "Coach" (an outer optimizer with momentum).

Imagine the Coach remembers where the team was heading yesterday.
When the artists share their local updates, the Coach uses that memory to correct their course.
This helps the team stay on track even though they are only talking to a few people at a time. It prevents the "rumors" from getting too distorted.

Why This Matters (The Results)

The paper tested this method on training large AI models (LLMs) and found two major wins:

Speed in a Perfect World: Even when everyone has fast internet, GASLoC is just as good at learning as the best existing methods, but it doesn't need the heavy, slow "big circle" meetings.
Superpower in a Messy World: This is the big one. Imagine one artist has a very slow internet connection (a "straggler").
- Old Method: The whole team stops and waits for the slow artist. The fast artists just stand around doing nothing.
- GASLoC: The fast artists keep painting their local sections. The slow artist is allowed to do fewer local steps before they have to catch up. Because the fast artists don't have to wait for the slow one, the whole team finishes the mural much faster.

The Bottom Line

GASLoC is like a team of painters who stop waiting for the slowest person to catch up. Instead, they let everyone work at their own pace, only talking to a couple of neighbors at a time, and using a smart coach to keep everyone aligned. This makes training giant AI models faster, cheaper, and much more resilient when some computers are slower than others.

Technical Summary: Unifying Local Communications and Local Updates for LLM Pretraining (GASLoC)

Problem Statement

Large Language Model (LLM) pretraining increasingly relies on distributed compute across clusters and data centers. A primary bottleneck in this setting is communication, particularly when utilizing standard synchronous All-Reduce operations. While All-Reduce is efficient on tightly coupled clusters, its cost scales linearly with the number of nodes, and its synchronous nature makes it highly vulnerable to stragglers (workers with slower compute or lower bandwidth). Existing communication-efficient methods, such as DiLoCo (Distributed Low-Communication), reduce communication frequency by performing multiple local steps before a global synchronization. However, DiLoCo still relies on global All-Reduce collectives, which remain a bottleneck in heterogeneous environments where worker speeds or network bandwidths vary significantly.

Decentralized (gossip-based) learning offers an alternative by replacing global synchronization with local peer-to-peer exchanges. However, applying gossip methods to modern LLM training with adaptive optimizers (like AdamW) and multiple local steps presents challenges:

Consensus Error: Without global synchronization, local model replicas can drift apart, degrading optimization performance.
Communication Acceleration: Classical gossip methods often lack the "outer momentum" mechanisms found in DiLoCo that accelerate consensus.
Heterogeneity: Existing decentralized methods often assume homogeneous data or fixed communication patterns that do not adapt to varying worker speeds.

Methodology: GASLoC

The authors propose GASLoC (Generalized Accelerated Sparse Communication Local Computation), a decentralized pre-training algorithm that unifies local gradient steps, sparse randomized peer communication, and an outer optimizer with momentum.

Core Algorithm

GASLoC operates in two loops:

Inner Loop (Local Steps): Each worker $i$ performs $H_i$ local gradient updates using an inner optimizer (e.g., AdamW). The number of local steps $H_i$ can be adapted per worker to compensate for stragglers.
Outer Loop (Communication & Aggregation): Instead of a global All-Reduce, workers engage in sparse, randomized peer-to-peer communication.
- Randomized Topology: At each round, workers sample a random permutation to define a sparse communication graph. The paper proposes 1-Peer (random disjoint pairs) and 2-Peer (random ring) schemes.
- Outer Optimizer: The algorithm uses an outer optimizer (e.g., Nesterov momentum) to update parameters based on the difference between the current local state and the received peer states. This outer momentum acts as a communication acceleration mechanism, reducing the spectral gap dependence from $\chi$ to $\sqrt{\chi}$ .

Key Theoretical Properties

Generalization of DiLoCo: When the communication graph is complete (all-to-all), GASLoC recovers the DiLoCo update rule exactly. Thus, GASLoC is a strict generalization of DiLoCo to gossip-based algorithms.
Convergence: The paper proves standard convergence rates for homogeneous data settings. The analysis utilizes a Lyapunov function combining optimization progress and consensus error.
Robustness to Heterogeneity: The algorithm allows each worker to perform a flexible number of local steps ( $H_i$ ). In a bandwidth-straggler scenario, the straggler can reduce $H_i$ to match the communication time of faster workers, eliminating the idle time inherent in All-Reduce barriers.

Communication Schemes

1-Peer: Workers are randomly paired. The expected spectral gap scales as $O(1)$ , independent of $n$ .
2-Peer: Workers form a random ring. This scheme has lower variance in mixing than 1-Peer and also achieves an $O(1)$ spectral gap in expectation.
Randomization vs. Determinism: Unlike deterministic exponential graphs or cycles (which have poor mixing for large $n$ ), GASLoC's randomized permutation ensures efficient information propagation without requiring complex, pre-defined topologies.

Key Contributions

Novel Algorithm: Introduction of GASLoC, which integrates communication acceleration (outer momentum) with sparse, randomized peer communication and local steps.
Theoretical Unification: Proof that GASLoC is a strict generalization of DiLoCo to gossip settings and that the outer momentum mechanism reduces communication complexity from $\chi$ to $\sqrt{\chi}$ in gossip settings.
Efficiency of Sparse Communication: Demonstration that using a subset of $k \le 2$ neighbors is more communication- and algorithmically-efficient than standard All-Reduce, even in homogeneous settings.
Heterogeneity Handling: A mechanism to adapt local step counts ( $H_i$ ) per worker, significantly improving wall-clock time in the presence of bandwidth stragglers.
Empirical Validation: Comprehensive evaluation on LLM pretraining tasks (134M and 551M parameter models) showing competitive performance against state-of-the-art baselines.

Experimental Results

The authors evaluated GASLoC on Llama-3-style decoder-only Transformers trained on the FineWeb dataset.

Homogeneous Setting (Standard Bandwidth)

Single-Step Regime ( $H=1$ ): GASLoC variants (particularly GASLoC-2-Peer) outperformed decentralized baselines like DAdam and matched or exceeded performance on ring and complete topologies, despite using significantly sparser communication.
Multi-Step Regime ( $H=30$ ): GASLoC achieved validation loss competitive with DiLoCo (the strong baseline using All-Reduce) across various worker counts (8, 16, 32).
- Notably, a modified baseline Local-DAdam (delaying DAdam mixing) failed to scale well with multiple local steps, whereas GASLoC maintained stability.
- GASLoC-2-Peer matched DiLoCo's performance while avoiding global synchronization.

Heterogeneous Setting (Bandwidth Stragglers)

Scenario: One worker was simulated with 10% or 20% of the standard bandwidth.
Result: DiLoCo suffered significant delays because the global All-Reduce barrier forced all workers to wait for the straggler.
GASLoC Advantage: By reducing the local steps ( $H_i$ ) for the straggler and using sparse peer exchanges, GASLoC allowed the straggler to synchronize without stalling the entire cluster. This resulted in substantial wall-clock time gains (faster time-to-loss) compared to DiLoCo, demonstrating robustness to heterogeneous resources.

Sensitivity Analysis

GASLoC showed similar scaling properties with respect to the number of local steps ( $H$ ) as DiLoCo.
The method remained stable across a broad range of outer optimizer hyperparameters (learning rate and momentum), indicating that the introduction of sparse communication does not drastically increase tuning complexity.

Significance and Claims

The paper claims to provide the first evidence that Local SGD-style LLM pretraining can be effectively combined with decentralized peer-to-peer communication when global synchronization is costly.

Unification: GASLoC bridges the gap between the practical success of DiLoCo (local steps + outer momentum) and the theoretical benefits of gossip algorithms (decentralization, fault tolerance).
Practical Impact: The method offers a path to more robust and efficient distributed training in real-world scenarios where network heterogeneity (e.g., internet-based clusters or mixed hardware) is common.
Scalability: By removing the dependence on global collectives, GASLoC mitigates the bandwidth bottlenecks that limit the scalability of current All-Reduce-based methods.

The authors emphasize that their claims are grounded in the specific experimental regimes tested (homogeneous data, specific model sizes, and simulated heterogeneity) and that the method serves as a principled generalization of existing frameworks rather than a radical departure from established optimization theory.