🤖 AI

PipeFusion: Patch-level Pipeline Parallelism for Diffusion Transformers Inference

PipeFusion is an innovative parallel inference methodology for Diffusion Transformers that partitions images into patches and model layers across multiple GPUs, leveraging patch-level pipeline parallelism and stale feature map reuse to significantly reduce communication costs and memory usage while achieving state-of-the-art performance on high-resolution image generation.

Original authors: Jiarui Fang, Jinzhe Pan, Aoyu Li, Xibo Sun, Jiannan Wang

Published 2026-05-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiarui Fang, Jinzhe Pan, Aoyu Li, Xibo Sun, Jiannan Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to paint a massive, incredibly detailed mural on a wall. To do this, you have a team of 8 artists (GPUs) working together. The painting process is unique: you don't just paint the whole thing once. You start with a blank, noisy canvas and slowly refine it, step-by-step, removing the noise and adding details until the image is clear. This is how modern AI image generators (called Diffusion Transformers) work.

The problem is that for high-resolution images, this process is slow. If you try to make the artists work together using traditional methods, they spend more time arguing about who has which part of the painting and passing buckets of paint back and forth than they do actually painting.

PipeFusion is a new, clever way to organize these artists to finish the job much faster. Here is how it works, using simple analogies:

1. The Old Way: Passing the Whole Bucket

In previous methods (like "Tensor Parallelism" or "Sequence Parallelism"), every time an artist finishes a small step, they have to stop and share everything they just did with everyone else before moving to the next step.

The Analogy: Imagine 8 chefs making a soup. Every time one chef adds a pinch of salt, they have to stop, call out the recipe to the other 7 chefs, wait for them to confirm, and then everyone adds their own salt. It's safe, but it's incredibly slow because of all the talking and waiting.

2. The "PipeFusion" Way: The Assembly Line with a Twist

PipeFusion changes the game by splitting the work in two ways:

Splitting the Wall: Instead of one chef doing the whole wall, the wall is cut into 8 vertical strips. Each chef is responsible for their own strip.
Splitting the Layers: The recipe (the AI model) is also cut into 8 parts. Chef 1 does the first part of the recipe for their strip, then passes the result to Chef 2, who does the second part, and so on.

This creates an assembly line. While Chef 2 is working on the second step of the recipe, Chef 1 is already starting the first step of the next batch. They are working in a pipeline, overlapping their tasks so no one sits idle.

3. The Secret Sauce: "Stale" Ingredients

Here is the real magic trick. In this painting process, the image changes very slowly from one step to the next. The "noise" at step 10 looks almost exactly like the noise at step 11.

The Analogy: Imagine you are baking a cake. Usually, you need fresh eggs for every batch. But PipeFusion realizes that the eggs you used 10 minutes ago are still good enough to use for the current batch.
How it helps: Instead of waiting for the "fresh" data from the current step (which would require waiting for the whole team to finish), PipeFusion lets the artists use the "stale" (slightly old) data from the previous step to keep working.
The Result: The artists don't have to stop and wait for the fresh data to arrive. They keep painting using the slightly older data, which is almost identical anyway. This hides the time it takes to pass information between artists.

4. Why It's Better

Less Talking: Because they are using the "stale" data they already have, they don't need to shout across the kitchen as often. This saves a huge amount of time.
Less Memory: Traditional methods require every chef to keep a copy of the entire wall's current state in their head. PipeFusion only requires them to remember their own small strip. This means you can run these massive models on standard computers without running out of memory (RAM).
Better Quality: Because PipeFusion uses "fresh" data for a longer portion of the process compared to other similar tricks, the final painting looks sharper and more accurate.

The Bottom Line

The paper tested this on 8 powerful graphics cards (GPUs) using famous AI models like Flux.1, Stable Diffusion 3, and Pixart.

Speed: PipeFusion was up to 1.5 times faster than the best existing methods.
Memory: It used significantly less memory, allowing it to run on very large models that other methods couldn't handle at high resolutions.
Quality: The images generated were virtually indistinguishable from the original, high-quality versions.

In short, PipeFusion is like turning a chaotic group of artists into a highly efficient, overlapping assembly line that uses "yesterday's news" to keep working today, resulting in faster, cheaper, and high-quality AI art generation.

Technical Summary: PipeFusion

Problem Statement

Diffusion Transformers (DiTs) have emerged as the leading architecture for high-resolution image and video synthesis, surpassing traditional U-Net models. However, DiTs face significant latency challenges during inference, particularly when generating high-resolution images or long sequences. This latency stems from the quadratic growth of attention computation with respect to sequence length.

Existing parallel inference strategies struggle to address these challenges efficiently on multi-GPU setups:

Tensor Parallelism (TP): While effective for Large Language Models (LLMs), TP is inefficient for DiTs due to large activation volumes. The communication overhead (specifically All-Reduce operations) often outweighs the benefits of parallel computation, especially for models with many layers.
Sequence Parallelism (SP): Methods like DeepSpeed-Ulysses and Ring-Attention partition the input sequence (image patches) across devices. While they reduce memory pressure, they require extensive data exchange (transmitting Query, Key, and Value activations) for every layer, leading to high communication costs.
DistriFusion: An asynchronous sequence parallelism method that exploits input temporal redundancy (the similarity of inputs and activations across consecutive diffusion timesteps). While it overlaps communication with computation, it suffers from memory inefficiency. DistriFusion requires maintaining full spatial dimensions of Key (K) and Value (V) activations for all layers on every device, causing memory costs to scale linearly with sequence length and preventing effective scaling for large models like Flux.1.

Methodology: PipeFusion

The paper proposes PipeFusion, a novel patch-level pipeline parallelism strategy designed specifically for DiT inference. PipeFusion addresses the limitations of existing methods by simultaneously partitioning the model layers and the input image patches across multiple GPUs.

Core Mechanisms

Dual Partitioning:
- Layer Partitioning: The DiT backbone is split along the data flow direction. Each GPU is assigned a contiguous set of transformer layers.
- Patch Partitioning: The input image is divided into $M$ non-overlapping patches. Each GPU processes one patch through its assigned layers in a pipelined fashion.
Exploiting Temporal Redundancy:
- Diffusion processes exhibit high similarity between inputs and activations of successive timesteps. PipeFusion leverages this by reusing "stale" feature maps from the previous diffusion timestep ( $T+1$ ) to provide context for the current timestep ( $T$ ).
- Instead of waiting for fresh activations from all devices for the current timestep, a device can begin computation using stale activations from the previous step. This allows the pipeline to run continuously with minimal idle time (bubbles), except during the initial startup.
Communication Optimization:
- Unlike SP and TP, which transmit activations for every layer, PipeFusion only transmits the input of the initial layer and the output of the final layer for a specific stage.
- It utilizes asynchronous Point-to-Point (P2P) communication to send micro-step patch activations to the subsequent device, effectively overlapping communication with computation.
Memory Efficiency:
- PipeFusion distributes model parameters across devices, reducing parameter memory usage by a factor of $N$ (the number of devices).
- Crucially, regarding the KV buffers required for attention: PipeFusion only needs to store $1/N$ of the full spatial K and V tensors for the layers assigned to that device. In contrast, DistriFusion requires full spatial K and V tensors for all layers on every device. This makes PipeFusion significantly more memory-efficient for large models.

Key Contributions

Novel Parallelism Strategy: Introduction of PipeFusion, which combines layer-wise pipeline parallelism with patch-level sequence partitioning, tailored for the full-attention mechanisms of DiTs.
Reduced Communication Costs: PipeFusion reduces inter-device communication by transmitting only stage inputs/outputs rather than layer-by-layer activations. Theoretical analysis shows communication costs are independent of the number of layers ( $L$ ) and scale only with sequence length ( $p$ ) and hidden size ( $h_s$ ).
Improved Memory Scalability: By distributing parameters and reducing the required KV buffer size to $1/N$ , PipeFusion enables the inference of massive models (e.g., 12B Flux.1) on commodity hardware where other methods fail due to Out-of-Memory (OOM) errors.
Quality Preservation: Theoretical analysis and empirical results indicate that PipeFusion maintains higher generation quality than DistriFusion because it utilizes a larger area of "fresh" activations during the diffusion process, whereas DistriFusion relies more heavily on stale data.

Experimental Results

The authors evaluated PipeFusion on an 8×L40 PCIe GPU cluster using three state-of-the-art DiT models: PixArt (0.6B), Stable Diffusion 3 (2B), and Flux.1 (12B).

Latency Performance:
- PipeFusion achieved the lowest inference latency across all tested resolutions (1024px to 4096px) and models.
- On 8×L40 GPUs, PipeFusion was 1.48× to 1.55× faster than the best sequence parallelism baselines (USP) for PixArt and SD3.
- For the 12B Flux.1 model, PipeFusion achieved a 1.23× to 1.25× speedup over the best sequence parallelism.
- In high-resolution tasks (4096px), DistriFusion encountered OOM issues, while PipeFusion maintained strong performance with a communication share of only 4.6% of total latency (compared to 17.9% for SP).
Memory Efficiency:
- PipeFusion demonstrated superior memory scalability. For Flux.1 at 2048px, PipeFusion's total memory usage was only 32% to 36% of that required by Sequence Parallelism.
- It successfully handled 8192px resolution tasks for PixArt, where DistriFusion failed.
Image Quality:
- Quantitative evaluation using Fréchet Inception Distance (FID) showed PipeFusion consistently outperformed DistriFusion, achieving scores closer to the original single-device implementation.
- Visual comparisons confirmed that images generated by PipeFusion were nearly indistinguishable from the original outputs.

Significance and Claims

The paper claims that PipeFusion represents a fundamental shift in parallelizing DiT inference. By systematically exploiting input temporal redundancy, it achieves a balance between communication efficiency, memory footprint, and generation quality that existing methods cannot match.

The authors position PipeFusion not as a universal replacement for all parallel strategies but as a highly effective component for hybrid parallelization schemes. They suggest that PipeFusion is particularly suited for inter-node, low-bandwidth environments, while Sequence Parallelism remains effective for intra-node, high-bandwidth (NVLink) settings.

The work highlights that temporal redundancy in diffusion processes is a viable and underutilized resource for designing new forms of parallelism, offering a path toward scalable, high-resolution image generation on commodity multi-GPU clusters. The authors also note that the approach is naturally compatible with video generation models, which share similar MM-DiT architectures and longer sequence lengths.