💻 computer science

Multi-Scale Local Speculative Decoding for Image Generation

This paper introduces Multi-Scale Local Speculative Decoding (MuLo-SD), a novel framework that accelerates autoregressive image generation by combining low-resolution drafting with spatially informed local rejection and resampling, achieving up to 5× speedup while maintaining high semantic alignment and perceptual quality.

Original authors: Elia Peruzzo, Guillaume Sautière, Amirhossein Habibian

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Elia Peruzzo, Guillaume Sautière, Amirhossein Habibian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to paint a massive, incredibly detailed mural on a wall. You are an artist who works one tiny square inch at a time, following a strict rule: you must finish one square before you can even think about the next one. This is how current AI image generators (called Autoregressive models) work. They build an image token-by-token, like a painter filling in a grid one dot at a time. While this produces beautiful results, it is painfully slow because the artist can't speed up; they have to wait for every single dot to dry before moving on.

The paper introduces a new technique called MULO-SD (Multi-Scale Local Speculative Decoding) to help this artist paint much faster without ruining the masterpiece. Here is how it works, broken down into simple analogies:

1. The Problem: The "Slow and Steady" Artist

Current AI models are like a perfectionist who refuses to guess. They calculate the exact color for the first pixel, then the second, then the third, all the way to the end. For a high-resolution image, this means thousands of steps. It's like reading a book one letter at a time, waiting for the printer to finish each letter before printing the next.

2. The Solution: The "Sketch and Verify" Team

The authors propose a two-person team to speed things up:

The Sketch Artist (The Drafter): A smaller, faster artist who works on a tiny, low-resolution version of the mural (like a rough sketch).
The Master Painter (The Target): The original, slow, high-resolution artist.

How they work together:
Instead of the Master Painter doing every single step, the Sketch Artist quickly draws a whole row of the low-res sketch. Then, a "magnifying glass" (an up-sampler) blows this sketch up to the size of the real mural. The Master Painter then looks at this blown-up sketch and says, "Yes, that looks right!" or "No, that's wrong."

If the Master Painter says "Yes," they accept the whole row instantly. If they say "No," they only fix that specific spot. This allows the team to skip thousands of slow, individual calculations.

3. The Secret Sauce: "Local Neighborhoods"

In older methods, if the Master Painter rejected even one tiny dot in a row, they would have to throw away the entire row and start over from the beginning. That's like a writer erasing their whole paragraph because of one typo.

MULO-SD changes the rules. It uses Local Verification.

The Analogy: Imagine you are proofreading a book. If you find a typo in the middle of a sentence, you don't throw away the whole page. You just fix that word and the few words immediately around it.
The Tech: If the AI rejects a token (a pixel block), it only re-draws that specific spot and its immediate "neighbors" (the surrounding pixels). It leaves the rest of the row alone. This saves a massive amount of time because most of the sketch was actually correct.

4. The Result: Painting 5x Faster

By combining these tricks—using a low-res sketch to guess the future, and only fixing small "islands" of errors rather than the whole page—the AI can generate images up to 5 times faster than before.

The paper tested this on a dataset of 5,000 images (MS-COCO). They found that:

Speed: It was significantly faster than previous "guessing" methods (like LANTERN or EAGLE-2) and even faster than other "parallel" methods (like ZipAR).
Quality: The images didn't look "rushed." They still matched the text prompts perfectly (semantic alignment) and looked just as good to human eyes (perceptual quality) as the slow, original method.

Summary

Think of MULO-SD as hiring a fast intern to sketch out the whole picture first, then having the boss quickly check the sketch. If the boss spots a mistake, they only tell the intern to fix that small corner, not the whole drawing. This way, the final masterpiece gets done in a fraction of the time, with the same high quality.

Technical Summary: Multi-Scale Local Speculative Decoding for Image Generation

Problem Statement

Autoregressive (AR) models have achieved significant success in image synthesis, particularly within unified multimodal large language models (MLLMs) that excel at text-to-image alignment and semantic understanding. However, the sequential nature of AR decoding imposes severe latency constraints, especially for high-resolution outputs where the token sequence grows quadratically with resolution. While Speculative Decoding (SD) offers a promising avenue for acceleration by employing a lightweight "drafter" model to propose tokens for parallel verification by a "target" model, existing approaches face critical limitations in the visual domain:

Token Ambiguity: Visual token distributions are often flatter and more ambiguous than text, leading to low acceptance rates for standard SD.
Lack of Spatial Awareness: Existing methods operate at the token level, ignoring the inherent spatial structure and multi-scale nature of images.
Inefficiency in Resampling: Standard SD rejects the entire sequence following the first rejection (raster-scan order), which is inefficient for images where local dependencies are strong but distant dependencies are weak.

Methodology: MULO-SD

The authors propose Multi-Scale Local Speculative Decoding (MULO-SD), a framework designed to accelerate AR image generation by integrating multi-resolution drafting with spatially informed verification. The method operates through three core mechanisms:

1. Multi-Scale Drafting

Instead of drafting tokens at the target resolution, MULO-SD employs a low-resolution drafter ( $M_q$ ) paired with an up-sampler ( $U_r$ ).

Process: The drafter generates a sequence of tokens at a lower resolution ( $s_q$ ). These tokens are up-sampled to expand the sequence length by a factor of $r^2$ (where $r$ is the resolution ratio) to match the target resolution ( $s_p$ ).
Rationale: This leverages the natural hierarchy of image resolutions. By generating coarse structures first, the method reduces the number of function evaluations (NFE) required from the high-resolution target model ( $M_p$ ).
Drafting Strategy: Unlike standard SD which drafts $n$ tokens, the drafter here generates full rows to ensure the up-sampler produces coherent high-resolution patches.

2. Local Verification and Expansion

To address the low acceptance rates caused by token ambiguity and the inefficiency of raster-scan rejection, MULO-SD introduces a local verification strategy.

Relaxed Acceptance: Inspired by LANTERN, the method pools probability mass over the $k$ nearest neighbors in the codebook space. A draft token is accepted if the pooled probability exceeds a threshold $\tau$ .
Local Rejection & Resampling: Crucially, when a token is rejected, the method does not discard the entire subsequent sequence. Instead, it defines a local neighborhood around the rejected token. Only tokens within this neighborhood (and the rejected token itself) are resampled by the target model.
Mechanism: This exploits the spatial locality of AR visual models, where token predictions are heavily influenced by immediate neighbors but weakly by distant regions. This "rejection island" approach preserves accepted distant tokens, significantly improving efficiency.

3. Integration with Parallel Decoding

MULO-SD is designed to be orthogonal to parallel decoding strategies.

Parallel Drafting: The low-resolution drafting stage utilizes parallel decoding (e.g., ZipAR) to generate draft rows simultaneously rather than sequentially.
Parallel Resampling: The set of rejected tokens (expanded to their local neighborhoods) is grouped into "rejection islands" (8-connected components). These islands are resampled in parallel by the target model, further reducing sequential dependencies.

Key Contributions

Multi-Scale Drafting Framework: The first speculative decoding method to leverage a low-resolution drafter and up-sampling to propose candidate tokens for high-resolution image generation, injecting a coarse-to-fine bias into the drafting strategy.
Local Verification Mechanism: A novel rejection and resampling strategy that operates over local spatial neighborhoods rather than full raster-scan sequences, addressing the inefficiency of standard SD in visual domains.
Orthogonal Integration: Demonstrates that MULO-SD can be combined with parallel decoding techniques to achieve substantial speedups without compromising the quality of generated tokens.

Experimental Results

The method was evaluated on the MS-COCO 2017 5k validation split using Tar-1.5B/7B (multimodal LLMs) and LlamaGen (XL). Metrics included decoding efficiency (speedup), semantic alignment (GenEval, DPG-Bench), and perceptual quality (FID, HPSv2).

Speedup: MULO-SD achieves speedups of up to 5× (specifically 5.33× for Tar-7B at 1024p) compared to standard sequential decoding.
Comparison: It outperforms strong baselines including EAGLE-2 (which often results in negative speedups on images due to low acceptance), LANTERN, and ZipAR.
- At 1024p, MULO-SD (4× upscaling) achieved a 5.33× speedup with minimal degradation in GenEval (+0.2) and HPSv2 (-0.8) compared to the baseline.
- It consistently delivers higher speedups than LANTERN while maintaining comparable or better semantic alignment.
Ablation Studies:
- Upsampling: A pixel-space upsampler (training-free) was found to be competitive and preferred for its model-agnostic nature.
- Probability Pooling: Pooling probabilities over codebook neighbors stabilizes performance and improves acceptance rates.
- Local Expansion: The local rejection and expansion mechanism is critical; omitting it degrades performance significantly.
- Parallel Decoding: Enabling parallel decoding for drafting and resampling yields consistent end-to-end latency gains.

Significance and Claims

The paper claims that MULO-SD sets a new state-of-the-art in speculative decoding for image synthesis, successfully bridging the gap between inference efficiency and generation fidelity.

Efficiency vs. Fidelity: The framework demonstrates that substantial acceleration (up to 5×) can be achieved without sacrificing semantic alignment or perceptual quality, a trade-off that often plagues existing acceleration methods.
Compatibility: Unlike bespoke multi-scale models (e.g., VAR) that require specific sampling schedules, MULO-SD integrates seamlessly with next-token prediction objectives and unified MLLMs, making it a practical solution for high-resolution image synthesis.
Scalability: The method is particularly effective at higher resolutions (1024p), where the quadratic growth of token sequences makes the reduction in function evaluations most impactful.

The authors note limitations, primarily the requirement for two distinct checkpoints (drafter and verifier) which increases memory usage, and the dependency on distributional alignment between the two models. They suggest self-speculative decoding as a potential future remedy.