Technical Summary: Autoregressive Visual Generation Needs a Prologue

1. Problem Statement

Autoregressive (AR) visual generation typically follows a two-stage pipeline: first, a tokenizer (encoder-decoder) learns to reconstruct images into discrete tokens; second, an AR model learns to predict these tokens sequentially. This approach suffers from a fundamental reconstruction–generation gap.

The core issue lies in the Evidence Lower Bound (ELBO) formulation. In standard two-stage training, the tokenizer is optimized solely for reconstruction (minimizing the negative log-likelihood of the data), while the AR model attempts to match the prior distribution of these tokens. Because the tokenizer's latent variables ($z_v$) are fixed by the reconstruction objective, the AR model must fit the often complex marginal distribution $q(z_v)$. Existing attempts to bridge this gap by jointly training the tokenizer and AR model (backpropagating AR loss to the tokenizer) face a Pareto front dilemma: increasing the AR loss weight ($\lambda$) improves generation but degrades reconstruction quality, while decreasing $\lambda$ weakens the generation signal. Furthermore, directly applying AR gradients to 2D visual tokens can disrupt their inherent spatial locality, which is crucial for high-quality reconstruction.

2. Methodology: Prologue

The authors propose Prologue, a framework that decouples the objectives of reconstruction and generation by introducing a separate set of latent variables.

Core Architecture

Instead of modifying the visual tokens ($z_v$) to satisfy both objectives, Prologue generates a small set of prologue tokens ($z_p$) prepended to the visual token sequence.

• Shared Encoder: A bidirectional Transformer encoder processes both learnable prologue queries ($q_1, \dots, q_K$) and image patch embeddings simultaneously.
• Dual Quantization:
  • Visual Tokens ($z_v$): Quantized via a standard visual quantizer. These tokens are dedicated exclusively to reconstruction (via a decoder) and do not receive direct AR gradients.
  • Prologue Tokens ($z_p$): Quantized via a separate, smaller codebook. These tokens are dedicated exclusively to generation (via the AR model) and do not participate in image reconstruction.
• Training Objective: The total loss is a weighted sum:
  $$\mathcal{L} = \mathcal{L}_{\text{recon}}(x, \hat{x}) + \lambda \cdot \mathcal{L}_{\text{AR}}(z_p, z_v)$$
  Crucially, the AR cross-entropy (CE) loss gradients are routed only to the prologue tokens (using a technique called Prob-STE for discrete differentiation), leaving the visual token representation intact.

Theoretical Foundation (ELBO Perspective)

From an ELBO perspective, Prologue shifts the AR prior matching task.

• Standard AR: The model must fit the marginal distribution $q(z_v)$.
• Prologue AR: The model fits the conditional distribution $q(z_v | z_p)$.
  Since $H(z_v | z_p) \leq H(z_v)$, modeling the conditional distribution is theoretically easier, provided $z_p$ carries non-trivial information about $z_v$. The authors prove that the optimization landscape naturally prevents $z_p$ from collapsing (becoming uninformative) because a collapsed $z_p$ offers no reduction in the AR loss compared to the baseline. Thus, $z_p$ is forced to encode a compact, AR-friendly summary of the image.

Implementation Details

• Prob-STE: A straight-through estimator variant that computes soft assignments for the discrete quantization of prologue tokens, allowing gradients to flow back to the encoder.
• Visual Token Dropout: During training, visual tokens in the AR input are randomly dropped (probability $p_{drop}=0.5$) to force the AR model to rely on prologue tokens for prediction, strengthening the gradient signal to $z_p$.
• Prologue-Post: A variant where the visual tokenizer is frozen (pre-trained), and only the prologue tokenizer and AR model are trained. This allows upgrading existing tokenizers without retraining the reconstruction pipeline.

3. Key Contributions

1. Decoupled Design: The proposal of a separate latent variable ($z_p$) for generation that is structurally isolated from the reconstruction variable ($z_v$). This resolves the reconstruction-generation trade-off, allowing AR optimization without degrading image fidelity.
2. Theoretical Justification: Formalization of the method via ELBO, demonstrating that conditioning on prologue tokens reduces the data-model gap by transforming a marginal matching problem into a conditional one.
3. Emergent Semantics: The discovery that prologue tokens, driven solely by AR gradients, spontaneously develop semantic structure. Linear probing on 16 prologue tokens achieves 35.88% Top-1 accuracy on ImageNet classification, significantly outperforming the first 16 tokens of a standard tokenizer (23.71%).
4. Plug-and-Play Capability: The introduction of Prologue-Post, which enables the improvement of generation quality for existing, frozen tokenizers by simply adding a lightweight prologue component.

4. Experimental Results

Experiments were conducted on ImageNet 256×256.

• Generation Quality:
  • Prologue-Base: Reduces the generation FID (gFID) without Classifier-Free Guidance (CFG) from 21.01 (2D baseline) to 10.75 (a 48.8% reduction). With CFG, gFID drops from 5.02 to 4.11.
  • Prologue-Large: Achieves a competitive gFID of 1.46 and rFID of 0.99, matching state-of-the-art diffusion and masked prediction models without using auxiliary semantic supervision (e.g., DINO/SigLIP alignment) or complex codebook clustering.
• Reconstruction Quality:
  • Reconstruction remains nearly unchanged. For Prologue-Base, rFID shifts only slightly from 2.15 to 2.24, and PSNR remains stable (~20.64).
  • Prologue-Post maintains the exact rFID of the frozen baseline (2.15) while improving gFID by 22.7%.
• Scalability: The method scales effectively with both tokenizer size (Base to Large) and AR model size (Base to XL), showing consistent gains in generation quality.
• Attention Analysis: Attention maps reveal that visual tokens preserve their natural 2D spatial locality (attending to neighbors), while the AR model learns to attend heavily to the prologue prefix for global context, effectively using $z_p$ to guide the prediction of $z_v$.

5. Significance and Claims

The paper claims that Prologue offers a new direction for autoregressive visual generation by addressing the reconstruction-generation gap through representation decoupling rather than modification.

• Optimization Efficiency: By separating the objectives, Prologue allows the AR model to optimize the true data distribution (via the conditional prior) without compromising the reconstruction capability of the visual tokenizer.
• Simplicity: The method achieves state-of-the-art results without relying on complex auxiliary losses (semantic alignment, flow matching) or heuristic training tricks often required by other AR tokenizers.
• Generalizability: The principle of introducing a separate learned generative representation while keeping the original representation intact is posited as a generalizable strategy for any system suffering from a reconstruction-generation gap.

The authors remain modest, noting limitations such as validation only on ImageNet 256×256 and the need for further exploration on higher resolutions and different datasets. They also acknowledge that the current work focuses on Cross-Entropy loss, leaving other prior-matching objectives as future work.