💻 computer science

SJD-PAC: Accelerating Speculative Jacobi Decoding via Proactive Drafting and Adaptive Continuation

The paper introduces SJD-PAC, an enhanced Speculative Jacobi Decoding framework that utilizes proactive drafting and adaptive continuation to overcome high-entropy bottlenecks in text-to-image synthesis, achieving a 3.8× speedup with lossless image quality.

Original authors: Jialiang Kang, Han Shu, Wenshuo Li, Yingjie Zhai, Xinghao Chen

Published 2026-03-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jialiang Kang, Han Shu, Wenshuo Li, Yingjie Zhai, Xinghao Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to paint a massive, incredibly detailed mural, but you have to do it one tiny brushstroke at a time. You can't see the whole picture until you finish. This is how current AI image generators work: they build an image pixel by pixel (or "token by token") in a strict, slow line.

The paper introduces a new method called SJD-PAC to make this painting process much faster without ruining the quality of the art. Here is how it works, explained through simple analogies.

The Problem: The "Guess and Check" Bottleneck

Imagine you are trying to guess a long, complex sentence word by word.

The Old Way (Autoregressive): You guess the first word, wait for the computer to check it, then guess the second, wait, and so on. This is very slow.
The Previous "Fast" Way (Speculative Jacobi Decoding): The computer tries to guess a whole chunk of words at once based on what it just wrote. Then, it checks them all at the end.
- The Catch: If the computer guesses the first word wrong, it has to throw away the entire chunk of words it just guessed and start over.
- The Reality: In complex images (like a face or a landscape), the computer is often unsure about the first few "brushstrokes." So, it keeps guessing the first word, getting rejected, and restarting. This is like a runner who trips on the starting line, gets up, trips again, and never actually runs the race.

The authors found that in nearly 50% of the attempts, the computer only successfully accepted one word before failing. This wasted a huge amount of time.

The Solution: SJD-PAC

The authors created a new system with two "superpowers" to fix this: Proactive Drafting (PD) and Adaptive Continuation (AC).

1. Proactive Drafting (PD): The "Plan B" Tree

The Analogy: Imagine you are navigating a maze. In the old method, if you hit a wall at the entrance, you have to go back to the start and try a completely new path.
How PD works: Instead of just guessing one path, the computer now builds a small tree of options right at the moment it hits a wall.

If the first guess is wrong, it doesn't just give up. It immediately sprouts 4 or 5 different "Plan B" paths branching out from that spot.
When it checks these paths next time, it's much more likely that at least one of them will be correct.
Result: It stops wasting time restarting from zero. It keeps moving forward by having multiple options ready to go.

2. Adaptive Continuation (AC): The "Keep What Works" Rule

The Analogy: Imagine you are assembling a 1,000-piece puzzle. You place 500 pieces perfectly. Then, you realize the 501st piece is the wrong shape.

The Old Way: You throw away the whole 500-piece section you just built and start the puzzle over.
How AC works: You only throw away that one wrong piece (the 501st one). You keep the 500 perfect pieces you already placed. You then try to fit a new piece into the 501st spot and continue checking the rest of the puzzle.
Why it works: In image generation, changing one small part of the image usually doesn't mess up the entire rest of the picture. The "tail" of the image is still valid. AC realizes this and saves the good work, only fixing the bad part.

The Result: A Speed Boost Without the Blur

By combining these two tricks:

PD ensures the computer doesn't get stuck restarting constantly.
AC ensures the computer doesn't throw away good work just because of one mistake.

The paper shows that this new method makes image generation 3.8 times faster (almost 4x speedup) compared to the standard method.

The Best Part?
Usually, when you speed up a process, you lose quality (like a blurry photo). But because SJD-PAC is "lossless," the final images look exactly the same as the slow, high-quality versions. It's like finding a shortcut that gets you to the destination in half the time, but you arrive with the exact same luggage you started with.

Summary

Old Problem: The AI kept tripping over the first step, forcing it to restart the whole race.
New Fix:
- Proactive Drafting: "If I trip, I have 4 backup shoes ready to put on immediately."
- Adaptive Continuation: "If I trip, I only fix my shoe; I don't throw away the whole race I've already run."
Outcome: The AI runs the race 4x faster, and the finish line photo is just as perfect as before.

1. Problem Statement

Autoregressive (AR) models for Text-to-Image (T2I) synthesis have achieved state-of-the-art quality but suffer from severe inference latency due to the sequential generation of thousands of tokens. While Speculative Decoding (SD) is a standard acceleration technique for Large Language Models (LLMs), it is largely ineffective for T2I generation because:

High Entropy: Visual generation inherently exhibits high entropy, making it difficult for a draft model to accurately predict the target model's output.
Draft Model Requirement: Standard SD requires training and maintaining a separate, smaller draft model, which is resource-intensive.
Limitations of Existing Lossless Methods: Speculative Jacobi Decoding (SJD) offers a training-free, lossless alternative by using the target model's own previous iteration outputs as drafts. However, SJD suffers from a skewed acceptance-length distribution:
- In nearly 50% of forward passes, SJD accepts only a single token (or zero acceleration) because the first draft token is rejected.
- This rejection triggers a "context mismatch," causing subsequent tokens in the sequence to be invalid, leading to cascading rejections and forcing a full resampling of the remaining sequence.
- Consequently, the average speedup is limited to ~2×, which is insufficient for practical deployment.

2. Methodology: SJD-PAC

The authors propose SJD-PAC, a training-free and strictly lossless framework that enhances SJD through two synergistic mechanisms: Proactive Drafting (PD) and Adaptive Continuation (AC).

A. Proactive Drafting (PD)

Goal: Mitigate the frequency of single-token acceptances caused by context mismatches after a rejection.

Mechanism: In standard SJD, if a token at position $i$ is rejected, the subsequent draft tokens (based on the old context) become invalid. PD addresses this by immediately constructing a diverse set of proposal paths rooted at the newly resampled token $x^t_i$ .
Implementation:
- Upon rejection at position $i$ , PD builds a shallow K-ary tree (e.g., depth $D=3$ , branching factor $K=4$ ) for the next few positions ( $i+1$ to $i+D$ ).
- It samples $K$ candidate tokens from the target distribution for each node in the tree.
- One path is extended into a long chain to fill the rest of the sequence.
Effect: By providing multiple diverse alternatives at the critical boundary, PD increases the probability that at least one path remains valid during the next verification step, suppressing the spike of single-token acceptances.

B. Adaptive Continuation (AC)

Goal: Maximize the number of accepted tokens per forward pass by avoiding premature termination.

Mechanism: Standard SJD terminates the verification loop immediately upon the first rejection, discarding all subsequent tokens. AC eliminates this "hard break."
Implementation:
- When a token at position $i$ is rejected, AC resamples it but continues verifying the remaining tokens ( $j > i$ ) using the "stale" (cached) distributions from the previous iteration.
- Theoretical Justification: The authors demonstrate that image tokens exhibit strong locality; the Total Variation (TV) distance between distributions at position $i$ and $i-j$ drops rapidly as $j$ increases. Therefore, using stale distributions for subsequent tokens is highly effective, preserving valid tokens that would otherwise be discarded.
Effect: This allows the system to retain valid tokens beyond the point of failure, creating a more stable initial state for the next iteration and significantly increasing the average acceptance length.

C. Algorithm Flow

Perform a parallel forward pass to get target probabilities.
Iterate through the sequence (Adaptive Continuation Loop):
- Verify each token using rejection sampling.
- If rejected, record the index but do not break; continue verifying the rest of the sequence using stale distributions.
If any rejection occurred, trigger Proactive Drafting to generate a diverse tree of candidates starting from the first rejection point for the next iteration.

3. Key Contributions

Bottleneck Analysis: Identified that the skewed distribution of acceptance lengths (frequent single-token rejections) is the primary performance limiter in existing SJD methods for T2I.
Novel Framework (SJD-PAC): Introduced a training-free, lossless framework combining:
- Proactive Drafting (PD): Uses K-ary trees to recover from context mismatches.
- Adaptive Continuation (AC): Preserves valid tokens after a rejection by leveraging the locality of image token distributions.
State-of-the-Art Performance: Demonstrated that SJD-PAC achieves acceleration competitive with lossy methods while strictly preserving the target model's output distribution.

4. Experimental Results

The method was evaluated on Lumina-mGPT (7B) and Emu3 (8B) using MS-COCO and PartiPrompts benchmarks.

Speedup:
- Lumina-mGPT: Achieved a 3.80× wall-clock speedup and 4.51× step compression on MS-COCO.
- Emu3: Achieved a 3.25× wall-clock speedup.
- This significantly outperforms standard SJD (~~2×) and EAGLE (~~2.1×).
Quality (Lossless Guarantee):
- FID and CLIP-Score: SJD-PAC results were virtually identical to the baseline (e.g., FID 30.69 vs. 30.79 for Lumina-mGPT), confirming the lossless nature.
- Comparison to Lossy Methods: SJD-PAC outperformed lossy methods like GSD and SJD2 in speed while maintaining higher image fidelity (lossy methods often showed FID degradation).
Ablation Studies:
- Both PD and AC contributed significantly to the speedup.
- Optimal hyperparameters were found to be a tree depth $D=3$ and branching factor $K=4$ with a window size $L=64$ .

5. Significance

Practical Viability: SJD-PAC bridges the gap between theoretical acceleration and practical deployment for AR T2I models, making high-resolution image generation significantly faster without requiring additional training or auxiliary models.
Quality Preservation: It proves that substantial speedups in T2I synthesis do not require compromising image fidelity, addressing a critical limitation of previous "lossy" speculative decoding approaches.
Generalizability: The framework is model-agnostic and relies on the inherent properties of autoregressive generation (locality and high entropy), making it applicable to a wide range of AR vision models.

In conclusion, SJD-PAC reshapes the acceptance distribution of speculative decoding, shifting from inefficient short runs to highly efficient long runs, establishing a new state-of-the-art for training-free, lossless T2I acceleration.