🤖 machine learning

FlashOverlap: Minimizing Tail Latency in Communication Overlap for Distributed LLM Training

FlashOverlap is a novel communication-computation overlap technique for distributed LLM training that eliminates tail latency by replacing conventional collective operations with decomposed peer-to-peer communication and fine-grained partitioned computation scheduling.

Original authors: Rezaul Karim, Austin Wen, Wang Zongzuo, Weiwei Zhang, Yang Liu, Walid Ahmed

Published 2026-04-28

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Rezaul Karim, Austin Wen, Wang Zongzuo, Weiwei Zhang, Yang Liu, Walid Ahmed

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Kitchen Bottleneck"

Imagine you are running a massive, high-speed pizza restaurant. To keep up with the orders, you have a team of chefs (these are the GPUs/Accelerators) working together.

In a large-scale operation, one chef might chop the vegetables, another rolls the dough, and another prepares the sauce. This is Distributed Training.

However, there is a problem: the chefs need to share ingredients constantly. The dough chef needs the sauce from the sauce chef. In the current "state-of-the-art" way of doing things, the chefs often have to stop working to wait for a delivery of ingredients to arrive.

Even if they try to be clever by sending small batches of ingredients (this is what the paper calls "Data Slicing"), they always hit a "tail latency" problem. It’s like the chef finishing the last pizza but having to stand there idle for 30 seconds waiting for the very last pepperoni slice to arrive. That tiny delay at the end of every cycle adds up, slowing down the whole restaurant.

The Solution: "Flash-Overlap"

The researchers at Huawei have invented a new way to run the kitchen called Flash-Overlap.

Instead of the chefs stopping to wait for deliveries, or sending tiny, inefficient little boxes of ingredients, Flash-Overlap changes the entire workflow.

1. The "Juggling" Act (Decomposed P2P Communication)

Instead of waiting for a massive truck to deliver all the cheese at once (which is a "Collective Operation"), Flash-Overlap breaks the delivery down into a continuous stream of small, hand-to-hand passes between chefs.

Think of it like a bucket brigade at a fire. Instead of one person running a heavy bucket back and forth (which is slow and exhausting), everyone stands in a line and passes smaller buckets quickly. Because the buckets are moving constantly, the "delivery" never actually stops.

2. The "Smart Schedule" (Eliminating Tail Latency)

This is the "secret sauce" of the paper. In the old way, the chef would finish their work and then wait for the last delivery.

With Flash-Overlap, the chefs use a "Rank-Adaptive" schedule. They look at what they have and what they are about to receive. They prioritize the tasks that depend on incoming ingredients first, and they save the tasks that don't need new ingredients for the very end.

The Analogy: Imagine you are a chef. You have a pile of dough that is already ready to be baked, and a delivery of cheese that is 10 seconds away.

Old Way: You finish the dough, put it down, and stare at the door for 10 seconds waiting for the cheese.
Flash-Overlap Way: You realize the cheese is coming, so you immediately start prepping the next batch of dough. By the time you are done with that, the cheese has arrived. You never stood still.

Why Does This Matter?

By making sure the "chefs" (the computers) are almost never standing idle, the researchers achieved three major things:

Speed (Lower Latency): The "pizzas" (the AI's answers) come out much faster.
Efficiency (Higher MFU): The computers are working at their maximum potential, rather than wasting electricity while "waiting for the delivery."
Scale: This method works even when the "restaurant" gets massive—spanning across multiple buildings (nodes) rather than just one kitchen.

In short: Flash-Overlap turns a stop-and-go kitchen into a seamless, high-speed assembly line, ensuring that the "ingredients" of AI intelligence are always moving and the "chefs" are always cooking.

Technical Summary: FlashOverlap

FlashOverlap is a research paper presented by a team from Huawei's Toronto Ascend Team that introduces a novel communication-computation overlap technique designed to minimize tail latency in distributed Large Language Model (LLM) training and inference.

1. The Problem: The "Tail Latency" Bottleneck

As LLMs grow in size, they must be partitioned across multiple accelerators (GPUs, TPUs, NPUs) using various parallelism strategies:

Data Parallelism (DP): Requires reduce-scatter (RS) operations.
Tensor Parallelism (TP/TPSP): Requires all-reduce (AR), all-gather (AG), and reduce-scatter (RS) operations.
Ulysses Parallelism (UP): Requires all-to-all communication.

Existing Solutions & Their Flaws:
Current state-of-the-art methods (like those in Megatron or MindSpeed) primarily use data slicing. They break data into smaller chunks and interleave the computation of one chunk with the communication of another. However, these methods suffer from tail latency: even if you slice data into many pieces, the final chunk's communication cannot be hidden by computation, leaving a residual overhead. Furthermore, making slices too small can reduce arithmetic intensity, making the system memory-bound rather than compute-bound.

2. Methodology: Algorithm-Level Decomposition

Instead of just slicing data, FlashOverlap decomposes the collective communication algorithms themselves into asynchronous Peer-to-Peer (P2P) operations and re-schedules computations to eliminate the tail.

The core of the methodology lies in two new primitives:

Flash-All-Gather (Flash-AG): Decomposes the AG operation into a ring-based P2P sequence. It schedules the computation of the existing data slice first, then computes the received slices, ensuring the communication is effectively hidden under the compute.
Flash-Reduce-Scatter (Flash-RS): Decomposes the RS operation into P2P exchanges. Crucially, it uses a rank-adaptive partial output scheduling strategy. It schedules the computation of the data slice that will remain on the current node as the very last step. This specific ordering ensures that the "tail" communication is fully absorbed by the final computation step.

Architectural Versatility:
The authors demonstrate that these primitives can be instantiated for various layer types:

Row/Column Parallel Projections: For MLP and QKV layers.
Query-Split Attention: A specialized version that decomposes attention computation (scores, softmax, and weighted sum) to provide even more computation to overlap with communication, making it highly effective for high-FLOPS/low-bandwidth hardware.
Flash All-to-All: An extension for Ulysses Parallelism (UP).

3. Key Contributions

Elimination of Tail Overhead: Provides an exact algorithm that removes the residual communication latency found in data-slicing methods.
Unified Framework: A single approach that works across DP, TP, TPSP, and UP, and is compatible with both training and inference (prefill and decode).
Batch-Size Agnostic: Unlike slicing methods that degrade with small batches or long contexts, FlashOverlap maintains efficiency across varying regimes.
Orthogonality: It is compatible with existing optimized kernels like Flash Attention.

4. Experimental Results

The researchers evaluated the method using a model with $d_{model}=4096$ and $TP=4$.

Communication Reduction: While traditional data slicing reduces communication overhead by 84%, FlashOverlap achieves a **99.8% reduction**.
Latency Improvement: In MLP layer tests, FlashOverlap achieved a 36.9% reduction in end-to-end latency compared to the vanilla baseline, significantly outperforming the 31.4% reduction provided by data slicing.
Scalability: The method showed consistent performance gains (approx. 35–40% latency improvement) as sequence lengths increased from 1K to 16K, proving its robustness for long-context LLMs.
Attention Performance: The "Query-Split Attention" implementation showed superior latency reduction compared to standard row-parallel or slicing approaches.

5. Significance

FlashOverlap represents a shift from "data-centric" overlap to "algorithm-centric" overlap. By mathematically restructuring how collective communications are executed, it allows distributed training to scale more efficiently across multiple nodes. This is critical for training the next generation of extremely large models, as it maximizes Model FLOPS Utilization (MFU) and reduces the operational costs and energy consumption associated with communication bottlenecks.