Technical Summary: Continuous Speculative Decoding for Autoregressive Image Generation

Problem Statement

Continuous visual autoregressive (AR) models have emerged as a promising alternative to discrete token-based approaches for image generation, offering better reconstruction quality and avoiding the instability associated with vector quantization. However, like their discrete counterparts and Large Language Models (LLMs), these continuous AR models suffer from inherently slow inference speeds due to their sequential decoding nature.

While speculative decoding has successfully accelerated discrete visual AR models and LLMs by employing a "draft-and-verify" mechanism, its direct application to continuous distributions faces two fundamental theoretical and practical barriers:

1. Low Acceptance Rate: In continuous domains, the draft and target models often learn divergent data distributions. Consequently, samples generated by the draft model frequently fall outside the high-probability regions of the target model. This leads to extremely low probability ratios and, subsequently, near-zero acceptance rates, rendering the acceleration ineffective.
2. Non-Analytic Modified Distribution: In speculative decoding, if a draft token is rejected, a new token must be resampled from a "modified distribution" defined as the normalized difference between the target and draft distributions ($p(x) - q(x)$). In discrete spaces, this normalization constant is a simple sum. In continuous spaces, it requires a complex integral over the entire space, which lacks an analytic expression, making direct sampling intractable.

Methodology

The paper proposes Continuous Speculative Decoding, a framework designed to overcome these specific challenges in continuous visual AR models. The approach integrates three synergistic strategies:

1. Approximated Acceptance Criterion and Trajectory Alignment

To address the low acceptance rate caused by distributional inconsistency, the authors introduce a practical approximation and a novel alignment strategy:

• Approximated Criterion: Instead of computing the ratio of joint probabilities over the entire denoising trajectory (which is computationally expensive and prone to divergence), the method computes the ratio of joint probabilities $p(Y_p)/q(Y_q)$ where the draft and target trajectories share the same initial latent variable $x_0$.
• Denoising Trajectory Alignment: To minimize the divergence between the draft and target outputs, the authors propose aligning their denoising trajectories. Based on the Reparameterization Proximity theorem, setting the same noise vector $\epsilon_t$ for both the draft and target models during the reparameterization step ($x_{t-1} = \mu_\theta(x_t, t) + \Sigma_\theta(x_t, t)\epsilon_t$) significantly reduces the expected distance between their outputs. This alignment ensures that the draft model's trajectory remains closer to the target model's high-probability regions, thereby increasing the acceptance ratio.

2. Token Pre-filling

The authors observe that acceptance rates are particularly low during the initial autoregressive steps due to inconsistencies in prefix embeddings between the draft and target models. To mitigate this, they propose token pre-filling, where a small portion (e.g., 5%) of the initial tokens are generated by the target model rather than the draft model. This ensures a consistent prefix, stabilizing the acceptance rate in the early stages without adding significant inference latency.

3. Acceptance-Rejection Sampling for Modified Distribution

To handle the non-analytic modified distribution ($p'(x)$), the paper employs an Acceptance-Rejection Sampling scheme.

• Upper Bound Derivation: The authors derive an appropriate upper bound $M$ such that $M \cdot p(Y) \geq p'(Y)$. By setting $M = 1/Z$ (where $Z$ is the intractable normalization constant), they eliminate the need to compute the integral explicitly.
• Efficient Threshold Calculation: Crucially, the rejection threshold $\alpha_s$ is derived using the same denoising trajectory alignment property. This allows the threshold to be computed using only the Gaussian distributions defined by the models' parameters, avoiding the need for repetitive, expensive diffusion model inferences that would otherwise be required to sample from the modified distribution.

Key Contributions

The paper claims the following contributions:

• First Continuous Speculative Decoding: It is the first work to bridge the gap of speculative decoding to continuous distributions, enabling substantial acceleration for continuous visual AR models.
• Solutions for Low Acceptance Rates: It addresses the low acceptance rate problem through a novel approximated criterion, denoising trajectory alignment (based on reparameterization proximity), and token pre-filling.
• Tractable Resampling: It enables sampling from the modified distribution via a tailored acceptance-rejection sampling scheme with a proper upper bound, avoiding complex integrals.
• Plug-and-Play Integration: The method is validated on three existing continuous visual AR models (MAR, xAR, and Harmon) without requiring extra training or architectural changes.

Experimental Results

Extensive experiments were conducted on ImageNet (256×256 and 512×512 resolutions) and MSCOCO/MJHQ datasets using MAR, xAR, and Harmon models.

• Speedup: The approach achieves a wall-time speedup of over 2× across various models and resolutions. Specific results include:
  • MAR: Up to 2.33× speedup.
  • xAR: Up to 2.72× speedup.
  • Harmon: Up to 2.54× speedup.
• Quality Preservation: Quantitative metrics (FID, Inception Score, CLIPScore, Geneval) demonstrate that the accelerated generation maintains image quality comparable to the original target models. The standard deviation in metrics remains low, indicating stability.
• Ablation Studies:
  • Trajectory Alignment: Reduces the average distance between draft and target tokens from >2.0 to ~1.0, increasing acceptance rates from ~7% to >30%.
  • Pre-filling: A 5% pre-filling ratio was found to be optimal, significantly improving early-stage acceptance rates and reducing visual artifacts compared to 0% pre-filling.

Significance and Claims

The paper positions this work as a necessary evolution in accelerating autoregressive image generation. While speculative decoding has been successful in discrete domains, the absence of an analogous theory for continuous distributions has hindered the efficiency of continuous AR models. By solving the theoretical challenges of distributional inconsistency and non-analytic integrals, this work enables the practical deployment of faster continuous AR models. The authors claim that their method offers a robust, lossless (in terms of distribution) acceleration strategy that can be seamlessly integrated into existing architectures, providing a pathway for more efficient inference in vision and potentially other domains utilizing continuous autoregressive processes.