VibeToken: Scaling 1D Image Tokenizers and Autoregressive Models for Dynamic Resolution Generations

1. Problem Statement

Autoregressive (AR) image generation models (e.g., LlamaGen) have shown competitive performance but face significant barriers to production adoption compared to diffusion models. The primary bottleneck is scalability regarding resolution and aspect ratios:

• Token Explosion: Conventional 2D tokenizers (e.g., VQGAN) generate token sequences that grow linearly with image resolution. For a 1024×1024 image, this results in thousands of tokens (e.g., 4,096 tokens).
• Quadratic Compute Cost: AR models rely on self-attention, which has $O(T^2)$ complexity where $T$ is the sequence length. As resolution increases, inference FLOPs grow quadratically, making high-resolution generation computationally prohibitive.
• Rigidity: Most AR models are trained on fixed resolutions (e.g., 256×256) and struggle to generalize to arbitrary aspect ratios or higher resolutions without retraining or complex super-resolution pipelines.
• Diffusion Dominance: Diffusion models naturally handle arbitrary resolutions, leaving AR models behind in flexible, production-grade use cases.

2. Methodology

The authors propose a two-part solution: VibeToken (a novel tokenizer) and VibeToken-Gen (an AR generator).

A. VibeToken: Resolution-Agnostic 1D Tokenizer

VibeToken is a 1D Transformer-based image tokenizer designed to encode images of any resolution into a fixed, user-controllable number of tokens (e.g., 32–256), decoupling token count from spatial resolution.

Key architectural innovations include:

1. Dynamic Grid Positional Embeddings: Instead of fixed absolute positional embeddings, VibeToken uses a learned grid of embeddings that is differentially resized (via bilinear/bicubic interpolation) to match the input patch grid ($T_H \times T_W$). This allows the model to handle arbitrary aspect ratios without quality degradation.
2. Adaptive Patch Embeddings: Inspired by FlexiViT, the model supports variable patch sizes ($k$) during training and inference. A weight-resizing operator maps the base projection weights to the current patch size, allowing the model to adapt to different resolutions without reparameterization.
3. Adaptive Decoder Resolution: The decoder predicts pixels at a fixed high resolution (4× the patch size) and uses a learnable 2D convolutional downsampling layer to map to the target resolution. This enables native super-resolution without a separate upscaler module.
4. Dynamic-Length Tokenization: The model is trained with a uniform sampling of latent lengths $L \in [32, 256]$. The encoder produces exactly $L$ tokens, and the decoder consumes exactly $L$, eliminating the need for padding or fixed-length constraints.
5. Quantization: Uses Multi-Vector Quantization (MVQ) with 8 codebooks to achieve a vocabulary size of 32,768, balancing compression and reconstruction quality.

B. VibeToken-Gen: Efficient AR Generator

VibeToken-Gen is a class-conditioned AR model (based on the LlamaGen architecture) that operates on the short, resolution-agnostic token sequences produced by VibeToken.

• Constant Compute Budget: Since the token sequence length $L$ is fixed (e.g., 64 tokens) regardless of the output resolution, the inference FLOPs remain constant.
• Conditioning: The model conditions on the target resolution $(H, W)$ and aspect ratio alongside the class label to prevent geometric stretching artifacts.
• Training: Trained on ImageNet-1k with mixed resolutions (256×256 to 512×512) and aspect ratios, using a token budget comparable to fixed-resolution baselines.

3. Key Contributions

1. Resolution-Agnostic Tokenization: The first 1D Transformer tokenizer that natively supports arbitrary resolutions and aspect ratios while maintaining a short, dynamic token budget (32–256 tokens).
2. Compute Efficiency: Decouples AR inference cost from image resolution. VibeToken-Gen maintains a constant 179 GFLOPs for any resolution, whereas baselines like LlamaGen scale to ~11 TFLOPs for 1024×1024 images (a 63.4× efficiency gain).
3. Native Super-Resolution: The architecture supports generating high-resolution images from low-resolution inputs (or arbitrary targets) without separate upscaling modules.
4. State-of-the-Art Performance: Achieves competitive reconstruction (rFID) and generation (gFID) quality compared to both 2D tokenizers and diffusion models, despite using significantly fewer tokens.

4. Experimental Results

The paper evaluates VibeToken and VibeToken-Gen on ImageNet-1k and FFHQ datasets across various resolutions (256×256 to 1024×1024) and aspect ratios.

• Reconstruction Quality:
  • VibeToken-LL achieves 0.40 rFID at 256×256 and 2.40 rFID at 1024×1024.
  • It outperforms other 1D tokenizers (e.g., TiTok, One-D-Piece) in resolution generalization and matches or exceeds 2D tokenizers (e.g., IBQ, MAGViT) in compression efficiency.
• Generation Quality:
  • VibeToken-Gen (XXL) synthesizes 1024×1024 images using only 64 tokens with a gFID of 3.94.
  • Comparison:
    • Vs. Diffusion (NiT): VibeToken-Gen is 2.35× faster (0.46s vs 1.08s) and achieves better quality (3.94 gFID vs 5.87 gFID) at 1024×1024.
    • Vs. AR Baseline (LlamaGen): LlamaGen requires ~11 TFLOPs for 1024×1024, while VibeToken-Gen uses a constant 179 GFLOPs.
• Efficiency:
  • Token count is reduced from ~4,096 (in 2D tokenizers) to 64 tokens for 1024×1024 images.
  • Inference speed is sub-second (0.46s) for high-resolution generation, compared to 32+ seconds for baseline AR models at similar resolutions.

5. Significance

This work bridges the gap between Autoregressive models and Diffusion models in terms of flexibility and efficiency.

• Production Viability: By enabling AR models to handle arbitrary resolutions with constant compute, VibeToken makes AR generation viable for real-world applications where input/output dimensions vary (e.g., mobile apps, diverse display formats).
• Scalability: It demonstrates that AR models can scale to high resolutions without the quadratic compute explosion, challenging the notion that diffusion is the only path to high-resolution generative AI.
• Future Direction: The approach opens the door for unified multimodal models (text, image, video) that can operate under a fixed, user-controllable compute budget, regardless of the spatial complexity of the generated content.

In summary, VibeToken solves the "resolution scaling" problem in AR generation by shifting the burden of resolution adaptation from the generative model to a flexible, resolution-agnostic tokenizer, resulting in a system that is faster, cheaper, and more flexible than current state-of-the-art alternatives.