💻 computer science

SSD: Spatially Speculative Decoding Accelerates Autoregressive Image Generation

The paper introduces Spatially Speculative Decoding (SSD), a framework that leverages the inherent 2D spatial correlations of images to predict multiple tokens simultaneously, thereby accelerating autoregressive image generation by up to 13.3x while maintaining high fidelity.

Original authors: Shilong Xiang, Zirui Zhang, Lijun Yu, Chengzhi Mao

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shilong Xiang, Zirui Zhang, Lijun Yu, Chengzhi Mao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to paint a massive, high-resolution mural, but you are forced to follow a very strict, old-fashioned rule: you can only paint one single dot at a time, and you must move in a perfect "snake" pattern. You finish the first row from left to right, then jump to the very end of the second row, go left, jump to the end of the third, and so on.

This is how current AI image generators (called "autoregressive models") work. They treat an image like a long, flat list of words. Even though a picture is naturally a 2D grid (up/down, left/right), the AI flattens it into a 1D line. This makes the process incredibly slow because the AI has to load its entire "brain" (billions of parameters) just to decide the color of the next single dot, over and over again. It's like walking to the grocery store to buy a single apple, then walking back home, then walking to the store again for the next apple.

The Problem: The "Memory Wall"

The paper calls this the "Memory Wall." The AI spends most of its time just loading its brain into memory to make a tiny decision, rather than actually thinking. Because it has to do this thousands of times for one image, generating a picture takes a long time.

The Solution: SSD (Spatially Speculative Decoding)

The authors introduce a new method called SSD. They realized that images aren't actually 1D lists; they are 2D grids. If you know what a dot looks like, you can often guess what the dot directly below it will look like, just as easily as you can guess the dot to its right.

Here is how SSD changes the game, using a few analogies:

1. The "Guessing Game" Analogy

Old Way (1D): The AI guesses the next dot, checks if it's right, then guesses the next one. It's a slow, step-by-step relay race.
SSD Way (2D): The AI acts like a team of guessers. While one person guesses the next dot to the right, another person simultaneously guesses the dot directly below. They don't wait for the first guess to finish before starting the second. They guess a whole block of dots at once.

2. The "Drafting" Analogy
Think of the AI as a writer.

Standard AI: Writes one word, stops, checks a dictionary, writes the next word.
SSD: Writes a whole sentence (or even a paragraph) in one go as a "draft." Then, it quickly reads through that draft to see if it makes sense. If a word is slightly off, it fixes just that word without rewriting the whole paragraph.

3. The "Auto-Correct" Twist
The paper mentions a clever trick. Usually, if an AI guesses a block of tokens (dots) and one is wrong, it throws the whole block away and starts over. SSD is smarter. It treats the "wrong" guesses as rough drafts. It runs a quick check (verification) and fixes the specific errors in place without throwing away the whole block. It's like a spellchecker that fixes typos instantly rather than making you rewrite the whole page.

The Results: Speeding Up the Mural

The paper tested this on three powerful AI models. The results were dramatic:

Speed: They made the image generation up to 13 times faster.
- Example: A model that took 339 seconds (almost 6 minutes) to make an image now takes only 25 seconds.
Quality: Despite being so much faster, the images look just as good as the slow versions. The "guesses" were accurate enough that the final picture didn't lose any detail.
Plug-and-Play: This method doesn't require rebuilding the AI's brain. It's like adding a turbocharger to an existing car engine. You can turn it on when you want speed, or turn it off, and the car runs exactly as it did before.

Summary

The paper argues that by respecting the natural 2D shape of images (up/down and left/right) instead of forcing them into a 1D line, we can break the "Memory Wall." By guessing multiple dots at once and fixing small mistakes on the fly, SSD turns a slow, step-by-step process into a fast, parallel one, making high-quality AI art generation nearly instant.

Technical Summary: Spatially Speculative Decoding (SSD)

1. Problem Statement

Autoregressive models have achieved significant success in visual generation by treating images as 1D sequences of discrete tokens, mirroring language modeling. However, this approach imposes a geometric compromise: it flattens the intrinsic 2D spatial locality of visual signals into a linear raster-scan sequence.

This flattening creates a severe computational bottleneck during inference, often referred to as the "memory wall." Generating an $n \times n$ image requires $n^2$ sequential forward passes. Since each pass must reload the entire set of transformer parameters to predict a single token, the process is heavily constrained by memory bandwidth rather than compute.

Existing acceleration techniques, such as standard Speculative Decoding and Jacobi-based iteration, attempt to import Natural Language Processing (NLP) methods into vision. However, these methods remain bound to the 1D sequential assumption. Because visual patches exhibit high localized entropy without rigid grammar, anticipating multiple tokens along a flattened 1D horizon yields low draft acceptance rates. Consequently, these methods offer only modest speedups (1.8× to 3.7×) or degrade image quality by enforcing unnatural independence assumptions.

2. Methodology: Spatially Speculative Decoding (SSD)

The authors propose Spatially Speculative Decoding (SSD), a framework that aligns the predictive objective with the natural 2D geometry of images. Instead of predicting only the immediate next token in a 1D sequence, SSD simultaneously predicts the adjacent horizontal token and the token directly below it.

Core Mechanisms

2D Spatial Anticipation:
- SSD factorizes 2D anticipation into two orthogonal 1D prediction streams: horizontal (along the raster scan) and vertical (down the columns).
- The model uses lightweight heads to draft an entire row horizontally, and then drafts subsequent rows vertically in parallel.
- This approach leverages the observation that a token's dependency on the token directly above it is as strong as its dependency on the immediate left token, despite the large offset in the flattened sequence.
- Complexity Reduction: By drafting entire spatial blocks in parallel, SSD reduces the theoretical inference complexity from $O(n^2)$ to $O(n)$ for an $n \times n$ image.
Continuous Latent Space Prediction:
- Predicting discrete token IDs directly is difficult due to flat probability distributions over large codebooks, leading to low acceptance rates (<5%).
- SSD instead predicts the continuous latent features of the last transformer layer (specifically, the hidden state before the final RMSNorm).
- A lightweight predictor $f_\phi$ takes the hidden state and token embedding as input to predict the hidden state at a specific spatial offset.
- The predicted hidden states are then decoded through the base model's existing output layers to obtain token candidates.
Verification as Auto-Correction:
- Unlike standard speculative decoding, which rejects an entire block upon the first mismatch, SSD treats verification as a self-correcting mechanism.
- Because drafts form cohesive spatial blocks, minor representational mismatches can be resolved locally.
- Rejected tokens are not discarded; instead, the system samples from a residual distribution to repair them within the same forward pass.
- This process is repeated for $r$ rounds, where rejected positions are updated and the KV cache is rolled back for re-evaluation, requiring only $r+1$ forward passes per block rather than sequential regeneration.
Modularity:
- SSD is a plug-and-play module. It requires no modifications to the pretrained backbone.
- Only lightweight drafting heads are trained via self-distillation, keeping the FLOPs overhead minimal.

3. Key Contributions

Geometric Realignment: The paper demonstrates that respecting the underlying 2D geometry of vision unlocks massive computational efficiencies, moving beyond the "computationally expensive artifact" of 1D sequential decoding.
Latent Space Drafting: The introduction of continuous latent feature prediction significantly improves draft accuracy compared to discrete token prediction in visual domains.
Auto-Correcting Verification: A novel verification strategy that repairs rejected tokens in parallel rather than discarding them, preserving the benefits of spatial drafting.
Plug-and-Play Acceleration: A framework that accelerates generation without retraining the massive backbone models, applicable to any unified autoregressive model producing discrete visual tokens.

4. Experimental Results

The authors evaluated SSD on three state-of-the-art autoregressive models: Janus-Pro-7B, Lumina-mGPT-7B, and Emu3-8B, using benchmarks DPG-Bench and GenEval.

Speedup: SSD achieves up to 13.3× wall-clock speedup compared to standard autoregressive inference.
- Emu3-8B: 339s $\to$ 25.5s (13.27× speedup).
- Lumina-mGPT-7B: 91.6s $\to$ 7.5s (12.19× speedup).
- Janus-Pro-7B: 7.8s $\to$ 1.4s (5.74× speedup).
Quality: The method maintains high fidelity, with generation quality metrics on DPG-Bench and GenEval remaining comparable to the baseline autoregressive models and significantly outperforming 1D-MTP baselines (which suffered severe quality degradation).
Comparison:
- 1D-MTP: Achieved ~2.0–2.4× speedup but caused significant quality drops (e.g., Emu3 overall score dropped from 78.69 to 53.11).
- SJD (Jacobi-based): Preserved quality but was limited to 1.5–2.9× acceleration.
- SSD: Achieved the highest speedup while preserving quality.

5. Significance and Claims

The paper claims that the inefficiency of current autoregressive visual generation is not an inherent limitation of the models but a result of the 1D sequential assumption inherited from language modeling. By abandoning this assumption and aligning the decoding process with the intrinsic 2D geometry of images, SSD overcomes the memory wall.

The authors assert that this geometric shift paves the way for real-time, high-resolution autoregressive generative models. The results suggest that respecting visual locality allows for massive computational savings without compromising the unification of vision and language in a single backbone. The method is presented as a general solution that can be applied to existing unified models to make them competitive in terms of inference speed.