💻 computer science

SJD-VP: Speculative Jacobi Decoding with Verification Prediction for Autoregressive Image Generation

This paper proposes SJD-VP, a plug-and-play method that accelerates autoregressive image generation by leveraging the observation that tokens with increasing probabilities across iterations are more likely to be accepted, thereby improving speculative decoding efficiency and image quality.

Original authors: Bingqi Shan, Baoquan Zhang, Xiaochen Qi, Xutao Li, Yunming Ye, Liqiang Nie

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bingqi Shan, Baoquan Zhang, Xiaochen Qi, Xutao Li, Yunming Ye, Liqiang Nie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to write a long story, one word at a time. In the world of AI image generation, the "story" is an image, and the "words" are tiny pieces of pixels (called tokens).

Normally, AI models are like very careful, slow writers. They write one word, check if it makes sense, then write the next. This is accurate but very slow.

To speed this up, researchers invented a technique called Speculative Jacobi Decoding (SJD). Think of this as a "Drafting Team." Instead of writing one word at a time, the AI guesses a whole paragraph of words at once, then checks them all at the end. If the guesses are good, it saves a ton of time. If they are bad, it has to throw them away and start over.

The Problem:
The current "Drafting Team" is a bit clumsy. They guess words based on what seems likely right now, but they don't pay attention to what the "Editor" (the verification step) will actually accept.

The Analogy: Imagine a student guessing answers for a test. They pick answers that seem okay, but they don't realize the teacher (the verifier) has a very specific, strict rule: "I only accept answers that get stronger and more confident as you think about them." Because the student ignores this rule, the teacher rejects most of their answers, and the student has to keep re-doing the work. The speed-up fails.

The Solution: SJD-VP (The "Smart Predictor")
The authors of this paper noticed a secret pattern. They looked at the answers the teacher did accept and realized: "The answers that get accepted almost always show a trend of getting more confident (higher probability) step-by-step before they are finally checked."

They built a new system, SJD-VP, which acts like a Crystal Ball for the drafting team.

Here is how it works, using a simple metaphor:

1. The "Confidence Tracker" (Verification Prior)

Instead of just guessing the next word, the AI now watches the "confidence meter" of every possible word it could pick.

Old Way: "I think 'cat' is a good word."
New Way (SJD-VP): "I think 'cat' is a good word, AND look! Every time we thought about 'cat' in the last few seconds, our confidence in it went up. It's climbing the ladder! That means the Editor is going to love it."

2. The "Bayesian Fusion" (The Smart Mix)

The AI combines two pieces of information:

What the model thinks is likely right now.
The "Confidence Tracker" (the pattern of growing confidence).

It mixes them together like a chef combining a fresh ingredient with a secret spice. The result is a "Super-Guess" that is much more likely to be accepted by the Editor.

3. The Result

Because the drafting team is now guessing words that fit the Editor's secret rule (growing confidence), the Editor accepts way more of them.

Before: The team guesses 10 words, the Editor accepts 3. (Lots of wasted time).
With SJD-VP: The team guesses 10 words, the Editor accepts 7 or 8. (Much faster!).

Why is this a big deal?

It's Plug-and-Play: You don't need to rebuild the whole AI engine. You just swap out the "Drafting Team" with this new "Smart Predictor." It works with existing AI models like LlamaGen or Emu3 without needing extra training.
Better Quality: Because the AI is guided toward words that are "growing in confidence," it accidentally ends up picking the correct words more often, not just the fast ones. The images look sharper and more accurate.
Speed: The paper shows this method makes generating images 3x faster than previous methods, while making the pictures look even better.

In a nutshell:
The old method was like a sprinter running blindly, hoping to hit the finish line. The new method (SJD-VP) is like a sprinter with a GPS that knows exactly where the finish line is and guides them there without stumbling. It makes the AI faster, smarter, and more accurate.

1. Problem Statement

Autoregressive image generation models (e.g., Lumina-mGPT, LlamaGen) produce high-quality images but suffer from slow inference due to their inherently sequential token-by-token sampling process. Speculative Jacobi Decoding (SJD) has emerged as a promising acceleration technique that generates multiple tokens in parallel and validates them through iterative refinement.

However, existing SJD approaches face a critical bottleneck: low speculative acceptance rates.

Token Selection Ambiguity: Image tokens often exhibit semantic redundancy, meaning multiple tokens can convey similar meanings.
Structural Decoupling: Current methods treat the drafting step (sampling candidate tokens) and the verification step (validating tokens against the base model) as independent processes. The drafting step ignores the verification history, often generating tokens that are unlikely to pass the deterministic verification criteria.
Limitations of Current Fixes: Recent works (e.g., LANTERN, GSD) attempt to improve acceptance by relaxing the verification criteria to accept semantically similar tokens. While effective, this structural decoupling remains, limiting the potential for further acceleration.

The core question addressed by the paper is: Can the deterministic verification criterion be leveraged to predict which tokens will be accepted, thereby guiding the drafting process to generate higher-quality candidates?

2. Methodology: SJD-VP

The authors propose Speculative Jacobi Decoding with Verification Prediction (SJD-VP), a plug-and-play framework that aligns the drafting and verification steps using a Bayesian fusion approach.

Key Insight: Consecutive Probability Growth

Through in-depth analysis of verification-accepted tokens, the authors observed a distinct pattern:

Observation: Approximately 91.4% of tokens that eventually pass verification exhibit consecutive probability growth across Jacobi iterations before the verification step.
Correlation: Tokens showing this growth pattern are highly likely to be the correct tokens (prediction accuracy >96% for $n=3$ steps), whereas incorrect tokens typically show fluctuating or decreasing probabilities.
Conclusion: This growth pattern serves as a reliable Verification Prior that can predict acceptance and correctness.

The SJD-VP Framework

The method modifies only the Token Drafting Step of existing SJD variants. It operates in two phases:

Verification Prior Estimation:
- The system maintains a history buffer of token probabilities from previous Jacobi iterations.
- It calculates a historical reference probability ( $\bar{p}_t$ ) using Exponentially Weighted Aggregation (EWA) to smooth fluctuations while remaining sensitive to recent trends.
- It computes a prediction score ( $S_t$ ) based on the log-gain between the current probability and the historical reference.
- A Consecutive Growth Mask ( $M_t$ ) is applied to filter tokens that do not show a strict consecutive increase in probability over $N$ steps.
- The result is a Verification Prior that highlights tokens likely to be accepted.
Bayesian Verification Fusion:
- The method treats the Verification Prior as a Bayesian prior and the current drafting probability ( $p_t$ ) as the conditional likelihood.
- It fuses these to derive a posterior probability ( $p'_t$ ) for the final sampling distribution.
- Formula: For Top- $k$ candidates, the log-probability is updated as:
  $\log p'_t(x) = \log p_t(x) + M_t(x) \cdot S_t(x)$
- Tokens are then sampled from this posterior distribution, ensuring the draft tokens are biased toward those with a high probability of passing verification.

Theoretical Guarantee

The paper provides a theoretical proof showing that SJD-VP reduces the Total Variation Distance (DTV) between the draft distribution and the verification distribution. By correctly predicting the adjustment direction (increasing probability for correct tokens), the method mathematically guarantees a reduction in DTV, leading to higher acceptance rates.

3. Key Contributions

Identification of a Key Limitation: The authors pinpoint that the structural decoupling between drafting and verification is the primary cause of low acceptance rates in existing SJD methods.
Novel Observation: They reveal that "consecutive probability growth" is a strong, model-agnostic indicator of both verification success and token correctness.
SJD-VP Framework: They propose a simple, plug-and-play sampling mechanism that integrates verification prediction into the drafting step via Bayesian fusion. It complements existing methods that relax verification criteria.
Comprehensive Validation: Extensive experiments demonstrate that SJD-VP consistently accelerates decoding while improving or maintaining image generation quality.

4. Experimental Results

The method was evaluated on Parti-prompts and MS-COCO 2017 benchmarks using models like Lumina-mGPT, LlamaGen, and Emu3.

Acceleration:
- On Parti-prompts, SJD-VP achieved a 2.94x acceleration over the base model (vs. 2.20x for standard SJD).
- When integrated with LANTERN (a state-of-the-art variant), LANTERN-VP achieved 3.06x acceleration.
- On MS-COCO, SJD-VP achieved 3.21x acceleration.
Efficiency (NFE):
- SJD-VP significantly reduced the Number of Function Evaluations (NFE). For example, on Parti-prompts, it reduced NFE from 1035 (SJD) to 720.65.
Quality:
- FID (Fréchet Inception Distance): SJD-VP maintained or slightly improved FID scores compared to baselines.
- CLIP-Score: Semantic alignment scores improved slightly (e.g., from 32.09 to 32.15 on Parti-prompts), indicating better text-image alignment.
Ablation Studies:
- Bayesian Fusion: Proved essential; using only the prior ( $S_t$ ) was better than the baseline but worse than the fused posterior ( $p'_t$ ).
- Growth Steps ( $N$ ): Optimal performance was found at $N=3$ ; stricter constraints ( $N=4$ ) reduced efficiency by being too restrictive.
- Window Size: Larger Jacobi windows ( $W=64$ ) paired exceptionally well with SJD-VP, yielding the best latency reductions.

5. Significance

Efficiency-Quality Trade-off: SJD-VP breaks the traditional trade-off where acceleration often comes at the cost of image quality. It accelerates inference while improving generation fidelity.
Generalizability: The method is model-agnostic and plug-and-play, requiring only modifications to the drafting step. It can be seamlessly integrated into any existing SJD variant (e.g., LANTERN, GSD) without retraining the base model.
Paradigm Shift: It shifts the decoding paradigm from "draft then verify" (independent) to "draft with verification prediction" (aligned), offering a new direction for accelerating autoregressive generation beyond simple parallelization or criterion relaxation.

In summary, SJD-VP leverages the dynamic history of token probabilities to predict verification outcomes, effectively guiding the generation process toward correct tokens and significantly boosting the speed of autoregressive image generation.