Training-free Mixed-Resolution Latent Upsampling for Spatially Accelerated Diffusion Transformers

This paper proposes RALU, a training-free spatial acceleration framework for Diffusion Transformers that employs mixed-resolution latent upsampling with region-adaptive edge processing and noise-timestep matching to achieve significant inference speedups (up to 15.9$\times$) while maintaining high generation quality.

The Gist

Imagine you are an artist trying to paint a massive, hyper-detailed mural on a giant wall. You have a magical robot assistant (the Diffusion Transformer, or DiT) that can create these paintings from scratch, but it's incredibly slow. Every time the robot adds a new brushstroke, it has to look at the entire wall to decide where to put the next one. As the wall gets bigger (higher resolution), the robot gets overwhelmed, and the painting takes hours to finish.

The paper you shared introduces a new trick called RALU (Region-Adaptive Latent Upsampling) to make this robot work much faster without ruining the painting.

Here is the story of how they solved the problem, using simple analogies:

The Problem: The "Blurry Sketch" Trap

To speed things up, some people tried telling the robot: "Hey, let's just paint the whole thing small first, then blow it up to the big size at the end."

This is like drawing a tiny, quick sketch on a post-it note and then trying to blow it up to the size of a billboard.

• The Result: The smooth parts of the picture look okay, but the edges (like the outline of a dog's ear or a tree branch) turn into jagged, pixelated messes. This is called aliasing.
• The Other Problem: Because the robot was trained to paint at a specific pace, suddenly jumping from a tiny sketch to a big wall confuses it. It starts adding weird noise or "mismatching" the colors, making the picture look glitchy.

The Solution: The "Smart Zoom" (RALU)

The authors realized you don't need to paint the whole wall at high detail from the start, but you also can't wait until the very end to fix the edges. They came up with a three-step "Smart Zoom" strategy:

1. The Rough Draft (Low Resolution)

First, the robot paints the entire image at a low resolution (small size). This is fast because there are fewer pixels to calculate. It gets the general shapes and colors right, but the edges are still fuzzy.

2. The "Edge-Only" Fix (Region-Adaptive)

Instead of blowing up the entire fuzzy image immediately (which causes the jagged mess), the robot uses a special scanner to find the edges (the important lines).

• The Analogy: Imagine you are fixing a blurry photo. You don't need to sharpen the blue sky or the green grass; you only need to sharpen the outline of the dog.
• What RALU does: It takes the fuzzy low-res image, finds the "edge" patches, and only those specific patches get zoomed in and sharpened early. The rest of the image stays small and fuzzy for now. This saves a ton of time while fixing the jagged lines before they become permanent.

3. The "Time-Travel" Sync (NT-Matching)

Here is the tricky part. When you zoom in on just the edges, the robot gets confused about "time." It thinks it's at a different stage of the painting process than it actually is, leading to those glitchy artifacts.

• The Analogy: Imagine you are running a race. If you suddenly teleport from the starting line to the finish line, you might trip because your body isn't ready for the speed.
• What RALU does: It adds a tiny bit of "mathematical noise" and adjusts the robot's internal clock (timestep) so that the zoomed-in edges and the fuzzy background feel like they belong to the same moment in time. This prevents the glitchy artifacts.

The Final Result

By using this method, the robot can paint the mural 7 times faster (on FLUX models) or 3 times faster (on Stable Diffusion 3) without losing quality.

• No Training Needed: The best part is that you don't have to retrain the robot. You just give it these new instructions on how to paint.
• Double Speed: If you combine this "Smart Zoom" with other speed tricks that skip steps in time, you can get the robot to paint 16 times faster than normal!

Summary

Think of RALU as a smart editor for a video game. Instead of rendering the whole world in high-definition (which makes the game lag), it renders the background in low-def and only switches the camera to high-def when you look at something important (like an enemy or a tree). This way, the game runs super smooth, but it still looks amazing.

This paper gives us a way to make AI image generators fast enough to use in real-time apps (like on your phone) without sacrificing the beautiful details we love.

Technical Summary

1. Problem Statement

Diffusion Transformers (DiTs) have achieved state-of-the-art results in high-fidelity image generation but suffer from high computational overhead and inference latency, primarily due to the quadratic scaling of self-attention with respect to the number of input tokens. While existing acceleration methods focus on the temporal dimension (e.g., feature caching, distillation, skipping steps), spatial acceleration (reducing computation by processing at lower resolutions) remains underexplored for DiTs.

Previous attempts at spatial acceleration, such as naive latent upsampling or the "Bottleneck Sampling" method, introduce significant visual artifacts:

1. Aliasing Artifacts: Occur when upsampling late in the diffusion process, particularly in high-frequency regions (edges), because low-resolution latents lack the representation power for sharp boundaries.
2. Distribution Mismatching Artifacts: Occur because upsampling alters the noise distribution and timestep trajectory. Naive upsampling breaks the flow-matching trajectory, and adding uncorrelated noise fails to restore the original distribution, leading to global inconsistencies.

2. Methodology: Region-Adaptive Latent Upsampling (RALU)

The authors propose RALU, a training-free framework that achieves spatial acceleration by dynamically transitioning between latent resolutions while mitigating the artifacts described above. The method operates in three stages:

A. Mixed-Resolution Latent Upsampling (Addressing Aliasing)

Instead of upsampling the entire latent map at a single timestep, RALU employs a region-adaptive strategy:

1. Low-Resolution Denoising: The process begins with the latent at a reduced resolution (e.g., 1/4 of the original tokens) to maximize speed.
2. Edge-Selective Early Upsampling:
   • The model approximates the clean latent ($\hat{x}_0$) using Tweedie's formula.
   • It decodes this latent via the VAE and applies Canny edge detection in image space to identify artifact-prone regions.
   • Only the top-$r$ fraction of patches (typically 20–30%) corresponding to strong edge signals are early upsampled to full resolution.
   • The remaining non-edge regions continue denoising at low resolution.
   • Rationale: Early upsampling prevents aliasing in edges, while keeping the rest at low resolution preserves computational efficiency.
3. Full-Resolution Refinement: In the final stage, all remaining low-resolution latents are upsampled to full resolution to ensure global consistency.

B. Noise and Timestep Matching (NT-Matching) (Addressing Distribution Mismatch)

To correct the trajectory deviation caused by upsampling and noise injection, RALU introduces a mathematical framework:

1. Correlated Noise Injection: Standard upsampling creates a non-isotropic covariance matrix ($\Sigma$). RALU injects correlated noise ($z \sim \mathcal{N}(0, \Sigma')$) where $\Sigma' = I - c\Sigma$. This forces the upsampled latent back onto the original flow-matching trajectory.
2. Timestep Rescheduling: The injection of noise shifts the effective timestep. RALU analytically derives new start/end timesteps ($s_{k+1}, e_k$) and scaling factors ($a, b$) to ensure the conditional distribution matches the target.
3. Distribution Alignment: The method minimizes the Jensen-Shannon Divergence (JSD) between the actual sampling distribution (after rescheduling) and the original model's timestep distribution. This ensures the model samples the correct intervals without oversampling or frequency imbalance.

3. Key Contributions

• Analysis of Spatial Acceleration: The paper identifies and analyzes the two primary failure modes of naive latent upsampling in DiTs: aliasing in high-frequency regions and distribution mismatching due to trajectory skew.
• RALU Framework: Proposes a training-free, mixed-resolution approach that selectively upsamples only edge regions early (to prevent aliasing) and applies NT-Matching (to prevent distribution errors).
• Synergy with Other Methods: Demonstrates that RALU is complementary to existing temporal acceleration methods (e.g., caching, forecasting) and timestep-distilled models, allowing for compounded speedups.
• State-of-the-Art Performance: Achieves significant speedups with negligible quality degradation on leading DiT models.

4. Experimental Results

The method was evaluated on FLUX.1.dev and Stable Diffusion 3 (SD3) using metrics like ImageReward, CLIP-IQA, and GenEval.

• Speedup vs. Quality:
  • FLUX.1.dev: Achieved up to 7.0× speedup (7 steps vs. 50 steps) with negligible quality loss.
  • Stable Diffusion 3: Achieved up to 3.0× speedup with minimal artifact introduction.
  • Artifact Reduction: RALU significantly outperforms "Bottleneck Sampling" and naive upsampling, reducing the artifact ratio by a large margin at similar computational costs (TFLOPs).
• Combined Acceleration:
  • When combined with temporal acceleration (e.g., TaylorSeer, ToCa), RALU achieved up to 9.0× speedup.
  • When combined with timestep-distilled models (e.g., FLUX.1-schnell), it achieved a total 15.9× speedup over the base model with minimal quality degradation.
• Efficiency: The method adds negligible overhead (e.g., ~2.48 TFLOPs for edge detection) compared to the total computation, making it highly efficient.

5. Significance

• Practical Deployment: RALU provides a viable path for deploying large DiT models in real-time or on-device scenarios without requiring expensive retraining or fine-tuning.
• Training-Free Advantage: Unlike previous spatial acceleration methods that required unified training or learned modules, RALU works out-of-the-box with pre-trained models.
• New Paradigm: It shifts the focus from purely temporal acceleration to spatial acceleration, demonstrating that mixed-resolution processing can be highly effective if the specific artifacts (aliasing and distribution mismatch) are mathematically addressed.
• Scalability: The ability to combine spatial and temporal acceleration suggests a future where DiTs can generate high-fidelity images at speeds previously thought impossible without distillation.