🤖 machine learning

End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer

This paper proposes an end-to-end training pipeline that jointly optimizes a 1D semantic tokenizer and an autoregressive generative model, achieving state-of-the-art image generation performance with an FID of 1.48 on ImageNet 256x256.

Original authors: Wenda Chu, Bingliang Zhang, Jiaqi Han, Yizhuo Li, Linjie Yang, Yisong Yue, Qiushan Guo

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenda Chu, Bingliang Zhang, Jiaqi Han, Yizhuo Li, Linjie Yang, Yisong Yue, Qiushan Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a computer to draw pictures by describing them word-by-word, like a storyteller. This is called autoregressive generation. The computer guesses the next "word" (or in this case, a visual piece) based on the ones before it.

However, there's a big problem with how computers usually do this.

The Problem: The "Grid" vs. The "Story"

Most image-making computers break a picture into a 2D grid (like a chessboard). They try to describe the picture by reading this grid row-by-row, like reading a book.

The Issue: In a grid, the piece at the top-right corner depends on the piece at the bottom-left. But in a story (a 1D list), you can't talk about the end of the story before you've told the beginning. This mismatch makes the computer confused and the pictures look messy.

Previous attempts tried to fix this by either:

Changing the story order: Forcing the computer to read the grid in weird, random orders (like reading a book backwards or jumping around).
Flattening the grid: Crushing the 2D picture into a single long line of data. But often, this crushed the picture so hard that the details were lost, or the computer couldn't learn to draw them well.

The Solution: EOSTok (The "End-to-End" Team)

The authors of this paper created a new system called EOSTok. Think of it as a team of two workers who learn together, rather than one training the other.

1. The Two Workers

Worker A (The Tokenizer): This is the "compressor." It takes a high-definition photo and squishes it down into a short, 1D list of "tokens" (like a secret code).
Worker B (The Generator): This is the "storyteller." It looks at the secret code and tries to predict the next token in the list to recreate the image.

2. The Old Way (The "Two-Stage" Mistake)

In the past, these workers trained separately:

Stage 1: Worker A was trained only to squish the picture perfectly. Once it was done, it was frozen (locked in place).
Stage 2: Worker B was trained to predict the tokens.
The Flaw: Worker A didn't know that Worker B was going to be the one using the tokens. So, Worker A might have created a code that was great for saving space but terrible for predicting the next step. It was like a translator writing a book in a language that the next translator couldn't understand.

3. The New Way (End-to-End Training)

EOSTok trains both workers at the same time.

The Feedback Loop: If Worker B (the storyteller) makes a mistake, the error signal travels all the way back to Worker A. Worker A learns: "Oh, I need to change my secret code slightly so Worker B can predict the next part easier."
The Result: They evolve together. Worker A learns to create a code that is not just a good compression, but a code that is easy to predict.

The Secret Sauce: Three Tricks

To make this work perfectly, the authors added three special ingredients:

1. The "Pixel-Check" (APR Loss)
Usually, the computer only checks if the "next token" prediction is right. But sometimes, the computer gets really good at guessing tokens but produces a blurry, ugly picture in the end.

The Fix: The system takes the computer's guess of the next token, turns it back into a picture, and checks if that picture looks like the real one. This forces the computer to care about the final image quality, not just the token guessing game.

2. The "Smart Teacher" (Vision Foundation Models)
The authors brought in a "Smart Teacher" (a pre-trained AI that already knows what objects look like).

The Trap: If you force the 1D list to look exactly like the Teacher's 2D map, you lose the benefit of the 1D list (it becomes a grid again).
The Fix: They used a method called "Implicit Alignment." Instead of forcing the 1D list to copy the Teacher's map, they just made sure the hidden understanding inside the system matched the Teacher's knowledge. It's like letting a student learn the concepts from a teacher without forcing them to copy the teacher's handwriting. This keeps the 1D flow smooth but makes the content much smarter.

3. The "Codebook" Balance
The system uses a dictionary (codebook) of visual tokens.

If the dictionary is too small, the pictures look blocky.
If the dictionary is too huge, the computer gets confused trying to pick the right word.
The authors found that by making the computer bigger (more powerful), it could handle a bigger dictionary without getting confused, solving the trade-off between detail and predictability.

The Results

The paper claims that this new method is a huge success.

On a standard test (ImageNet 256x256), their model achieved a score of 1.48 (lower is better).
This is a State-of-the-Art result, meaning it is currently the best in the world for this specific type of image generation without using extra guidance tricks.
The model works better as it gets bigger (scaling up), proving the system is robust.

In short: EOSTok is a new way to teach computers to draw by having the "compressor" and the "predictor" learn together, checking their work against real pixels, and using a smart teacher to guide them without forcing them into a rigid grid. The result is a computer that can generate incredibly realistic images by simply predicting the next piece of a puzzle.

1. Problem Statement

Autoregressive (AR) image generation has emerged as a strong alternative to diffusion models, relying on discrete visual tokens to compress images into latent representations. However, existing approaches face two primary challenges:

Spatial Misalignment: Most image tokenizers use 2D grid structures (preserving spatial layout). This induces bidirectional dependencies among tokens, which conflicts with the unidirectional factorization required by standard raster-order AR modeling.
Suboptimal Training Paradigms: Current methods typically employ a two-stage training strategy: first training a tokenizer for reconstruction (frozen), then training an AR model on the resulting tokens. This prevents the tokenizer from adapting to the specific needs of the generative task.
Loss Function Gap: The Next-Token Prediction (NTP) loss used in AR models operates in discrete token space and does not directly correlate with final pixel-space generation quality. Optimizing solely for NTP can lead to "latent space collapse," where the tokenizer uses only a few tokens to minimize prediction loss, degrading image fidelity.

2. Methodology: EOSTok

The authors propose EOSTok (End-to-end One-dimensional Semantic Tokenizer), a framework that jointly optimizes a 1D vision tokenizer and an AR generative model in a single-stage pipeline.

A. 1D Vision Transformer Tokenizer

Architecture: Based on a ViT autoencoder structure (similar to TiTok).
Process: 2D image patches are flattened and concatenated with learnable query tokens. A causal ViT encoder processes this sequence, outputting only the 1D latent tokens (discarding spatial patch embeddings).
Quantization: Uses IBQ (Improved Bottleneck Quantization) to map continuous latent vectors to a discrete codebook, enabling AR modeling.
Decoder: A 1D ViT decoder reconstructs images by concatenating the quantized tokens with learnable mask tokens.

B. End-to-End Joint Training

Instead of freezing the tokenizer, the encoder, decoder, and AR model are trained simultaneously. The total loss function combines:

Reconstruction Loss ( $L_{recon}$ ): Standard VQ-VAE losses (L1/L2, Perceptual, GAN) to ensure image fidelity.
Next-Token Prediction Loss ( $L_{NTP}$ ): Standard cross-entropy loss for the AR model.
Autoregressive Prediction Reconstruction (APR) Loss ( $L_{APR}$ ):
- The Innovation: To bridge the gap between token prediction and pixel quality, the AR model's predicted tokens (in teacher-forcing mode) are decoded back into pixel space by the tokenizer's decoder.
- Mechanism: The loss compares these decoded pixels against the ground-truth image.
- Effect: This provides end-to-end generative supervision to the tokenizer, preventing the tokenizer from "hacking" the NTP loss by collapsing the latent space (using too few tokens).

C. Semantic Representation Alignment (VFM)

To inject global semantic information without enforcing 2D spatial priors (which would hurt AR performance), the authors propose Implicit Alignment:

Direct Alignment (Rejected): Aligning 1D latent tokens directly to 2D VFM features forces 2D structure onto the 1D space, degrading performance.
Direct Substitution (Rejected): Replacing image patches with VFM features as input.
Implicit Alignment (Proposed): Aligns the hidden patch embeddings of the 1D encoder (before the query tokens are extracted) with the VFM features (e.g., from DINOv2). This distills semantic knowledge into the sequential latent space without imposing explicit 2D spatial constraints.
Decoder Alignment: The decoder is also aligned with VFM features to treat reconstruction as a conditional generation task, aiding convergence.

3. Key Contributions

End-to-End Training Framework: A novel single-stage pipeline that jointly optimizes reconstruction and generation, allowing the tokenizer to learn representations specifically tailored for AR modeling.
APR Loss: A critical loss function that decodes AR predictions to pixel space, ensuring the tokenizer learns a latent space that is both reconstructive and generatively predictable, preventing latent collapse.
Implicit Semantic Alignment: A strategy to leverage Vision Foundation Models (VFMs) for 1D tokenizers that improves generation quality without reintroducing the 2D spatial biases that hinder AR modeling.
State-of-the-Art Performance: Achieving SOTA results on ImageNet 256×256 generation without classifier guidance.

4. Experimental Results

The model was evaluated on the ImageNet-1K 256×256 benchmark.

Generation Quality:
- EOSTok-H (644M parameters): Achieves a gFID of 1.48 (without guidance) and an IS of 239.5. This is a new state-of-the-art for AR models without guidance, outperforming diffusion-based models like REPA-XL/2 (gFID 1.42 with guidance) and other AR baselines.
- EOSTok-L (312M parameters): Achieves a gFID of 1.74 without guidance.
Reconstruction Quality:
- EOSTok-L achieves an rFID of 0.73, demonstrating high-fidelity reconstruction capabilities.
Ablation Studies:
- APR Loss: Without APR, the codebook usage drops drastically (from ~99% to ~51%), and gFID worsens significantly (8.01 vs 3.32).
- Semantic Alignment: Implicit alignment improves gFID from 12.27 (baseline) to 3.32. Direct alignment worsens performance, confirming that 2D priors are detrimental to 1D AR modeling.
- Sequence Ordering: Models trained with end-to-end supervision show significantly better performance on original token sequences compared to reversed or random sequences, proving the latent space is optimized for causal prediction.
- Scaling: Performance improves consistently with model size (S to H) and codebook size, with larger models mitigating the reconstruction-generation trade-off.

5. Significance

This paper fundamentally shifts the paradigm of autoregressive image generation:

From Two-Stage to One-Stage: It demonstrates that jointly training the tokenizer and generator yields superior results compared to the traditional frozen-tokenizer approach.
1D Tokenization Viability: It proves that 1D tokenizers, when properly trained with semantic guidance and end-to-end feedback, can outperform 2D grid-based tokenizers for AR generation by eliminating spatial dependency conflicts.
Bridging the Loss Gap: The introduction of APR loss solves the critical issue where token-level optimization fails to translate to pixel-level quality, a common bottleneck in discrete generative modeling.
Scalability: The method scales effectively, achieving SOTA results with a 644M parameter model, suggesting a promising path for scaling AR vision models to larger sizes.