🤖 AI

Frequency Autoregressive Image Generation with Continuous Tokens

This paper introduces the Frequency Autoregressive (FAR) paradigm, which leverages continuous tokens and a spectral regression direction where higher-frequency components build upon lower ones to overcome the modality gap in image generation, demonstrating superior efficacy on ImageNet and text-to-image tasks.

Original authors: Hu Yu, Hao Luo, Hangjie Yuan, Yu Rong, Jie Huang, Feng Zhao

Published 2026-03-19

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Hu Yu, Hao Luo, Hangjie Yuan, Yu Rong, Jie Huang, Feng Zhao

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot to paint a masterpiece.

For a long time, the standard way to do this (using Autoregressive models) was like teaching the robot to write a story, one word at a time. The robot would look at the previous word and guess the next one. To make this work for pictures, researchers forced the robot to turn the image into a string of "discrete blocks" (like Lego bricks) and then predict the next block in a strict line, like reading a book from left to right, top to bottom.

The Problem:
Images aren't like books. They are continuous (smooth gradients of color) and 2-dimensional (everything relates to everything else nearby).

The Lego Problem: Turning a smooth photo into Lego bricks loses a lot of detail. It's like trying to draw a sunset using only 16 colors; it looks blocky and blurry.
The Reading Problem: Reading an image line-by-line is unnatural. The top of a face doesn't depend on the bottom of the previous line; it depends on the nose right next to it. Forcing a "left-to-right" order breaks the natural flow of the picture.

The Solution: FAR (Frequency Autoregressive)
The authors of this paper, Hu Yu and colleagues, came up with a new way to paint called FAR. Instead of painting pixel-by-pixel or block-by-block, they teach the robot to paint from the blurry sketch to the sharp details.

Here is how it works, using a simple analogy:

The Analogy: The "Blurry Sketch" Artist

Imagine an artist who paints in layers of frequency (sharpness):

Layer 1 (Low Frequency): The artist starts with a very blurry, low-resolution sketch. They just block out the big shapes: "There's a mountain here, a river there, and a sun in the corner." This captures the structure and colors but has no details.
Layer 2 (Medium Frequency): Now, the artist adds some texture. The mountain gets some ridges, the river gets some waves. The shapes are still a bit fuzzy, but you can tell what they are.
Layer 3 (High Frequency): Finally, the artist adds the tiny, sharp details: the individual leaves on a tree, the reflection in the water, the texture of the rock.

Why is this better?

Natural Order: You can't paint the tiny details of a leaf until you know where the tree branch is. By building from "blurry" to "sharp," the robot follows a logical, causal path. It satisfies the "cause and effect" rule that AI models love.
Continuous Paint: Instead of using Lego bricks (discrete tokens), FAR uses "liquid paint" (continuous tokens). This means the colors are smooth and perfect, with no pixelation.
Speed: Because the robot only has to make a few "big jumps" (from blurry to sharp) instead of predicting millions of individual pixels one by one, it is incredibly fast. The paper shows it can generate a high-quality image in just 10 steps, whereas other methods might need hundreds.

The Secret Sauce: "Spectral Dependency"

The paper uses a fancy term called "spectral dependency," but think of it as the rule of building blocks.

Low frequencies are the foundation (the floor plan of a house).
High frequencies are the decorations (the wallpaper, the furniture).
You can't hang wallpaper before you build the walls. FAR respects this rule. It builds the walls first, then adds the decorations.

The Results

The researchers tested this on the famous ImageNet dataset (millions of images).

Quality: The images look just as good as the best current AI painters (like Diffusion models).
Efficiency: It uses much less computing power and time.
Text-to-Image: They even taught it to listen to text prompts (e.g., "A cat wearing a hat") and paint the result, doing so with a much smaller brain (model size) than previous giants.

In a Nutshell

Previous AI painters tried to write a picture like a sentence, one letter at a time, using a limited alphabet. FAR teaches the AI to paint like a human artist: starting with a rough, blurry sketch of the whole scene, and progressively sharpening the details until the image is crystal clear. It's faster, smoother, and respects the natural way images are built.

1. Problem Statement

Current autoregressive (AR) image generation models largely mimic the success of Large Language Models (LLMs) by adopting a two-stage paradigm:

Discretization: Using Vector Quantization (VQ) to convert continuous image data into a finite, discrete vocabulary.
Sequence Prediction: Flattening the 2D image into a 1D sequence (typically raster-scan order) and predicting the "next token."

The authors identify two fundamental flaws in directly inheriting this paradigm for images due to the modality gap between text and images:

Tokenizer Format: Images are inherently continuous, while text is discrete. Discretization via VQ introduces significant information loss and limits the performance upper bound.
Regression Direction: Text is causal and sequential (1D). Images are non-causal, non-sequential, and possess strong spatial locality (2D). Raster-scan or random token ordering violates the causal requirements of AR models and disrupts the inherent spatial priors of image data.

Existing methods attempting to bridge this gap (e.g., VAR, MaskGIT) often rely on discrete tokens or complex multi-scale architectures that struggle with continuous data or require excessive inference steps.

2. Methodology: The FAR Paradigm

The authors propose Frequency Autoregressive (FAR), a paradigm that leverages spectral dependency as the regression direction and utilizes continuous tokens.

A. Core Concept: Spectral Dependency

Instead of predicting tokens in spatial order (left-to-right, top-to-bottom) or random order, FAR predicts images in increasing frequency levels.

Low-frequency components capture global structure, brightness, and shapes.
High-frequency components capture edges, textures, and fine details.
Causality: High-frequency details naturally depend on the establishment of low-frequency structures. This hierarchy satisfies the causal requirement of AR models while preserving the 2D spatial locality of the image.

B. Architecture & Process

Continuous Tokenizer: The model uses a continuous tokenizer (based on VAE principles) rather than VQ, minimizing information loss.
Next-Frequency Prediction:
- An image $x$ is decomposed into $F$ frequency levels using spectral filters (Fourier transform).
- The model performs autoregressive generation from level $i$ to $i+1$ .
- At each step, the model predicts the distribution of the next frequency level conditioned on the previous lower-frequency levels.
- Within a frequency level, the model uses bidirectional attention to predict all tokens simultaneously, preserving spatial locality.
Diffusion Loss for Continuous Tokens:
- To handle continuous tokens, the authors employ a diffusion model as the loss function and sampler.
- The AR transformer predicts a vector $z$ for each token, which conditions a small MLP (noise estimator) to denoise the ground truth token distribution.

C. Key Technical Innovations

To make FAR efficient and trainable, the authors introduce several specific techniques:

Distribution Modeling Simplification: Instead of modeling the complex conditional distribution $p(x_{i+1} | x_i)$ , the model simplifies the task by directly modeling the full image distribution $p(x | x_i)$ and then filtering the output to get the next frequency level. This reduces optimization difficulty for the small MLP.
Frequency-Aware Training Loss: Different frequency levels have varying optimization difficulties (high frequencies are easier to predict). The authors introduce a sine-curve-based loss weighting strategy to assign higher weights to higher-frequency levels, ensuring balanced learning.
Mask Mechanism: To reduce training costs and improve diversity, the model randomly masks a subset of input tokens at lower frequency levels (where information is sparse), forcing the model to learn robust representations.
Frequency-Aware Diffusion Sampling: During inference, fewer diffusion sampling steps are allocated to lower frequency levels (which are easier to generate), significantly accelerating the inference process.

3. Key Contributions

FAR Paradigm: Proposes a new AR framework based on spectral dependency (next-frequency prediction) rather than spatial or random token prediction, aligning better with image data priors.
Continuous Token Integration: Successfully integrates continuous tokenizers with AR generation, solving optimization challenges through distribution simplification and frequency-aware loss strategies.
Efficiency: Achieves high-quality generation with drastically reduced inference steps (linear complexity $O(n)$ relative to token count, rather than quadratic $O(n^2)$ ) compared to traditional raster-scan AR models.
Scalability: Demonstrates that the method scales effectively with model size and is applicable to both class-conditional and text-to-image generation.

4. Experimental Results

The authors evaluated FAR on ImageNet (class-conditional) and JourneyDB/MS-COCO (text-to-image).

Class-Conditional Generation (ImageNet 256x256):
- Performance: FAR-H (791M parameters) achieves an FID of 3.21 and IS of 300.6.
- Efficiency: It achieves these results in only 10 inference steps, whereas comparable high-performing AR models (like LlamaGen or VAR) often require hundreds of steps or significantly larger parameter counts.
- Comparison: Outperforms discrete token-based AR models (VQGAN, LlamaGen) in visual quality metrics (IS, Precision) and is competitive with state-of-the-art diffusion models, despite using a much smaller model size and fewer steps.
Text-to-Image Generation:
- FAR achieves competitive results on MS-COCO and GenEval benchmarks.
- It utilizes significantly less training data (7.8M pairs vs. billions for DALL-E 3/SD3), fewer GPU days, and only 10 inference steps.
- Visual results show coherent structures and fine details, even with complex prompts.
Ablation Studies:
- Removing the mask mechanism reduces generation diversity (higher FID).
- Removing frequency-aware loss weighting degrades performance on high-frequency details.
- Attempts to combine the VAR paradigm with continuous tokens resulted in poor performance due to error accumulation and distribution modeling difficulties, validating the necessity of the FAR approach.

5. Significance and Impact

Bridging the Modality Gap: FAR provides a more natural framework for image generation by respecting the continuous and spatial nature of images, moving away from the "textification" of images via VQ.
Efficiency: By reducing inference steps to the number of frequency levels (e.g., 10 steps) rather than the number of tokens, FAR offers a highly efficient alternative to both diffusion models (which require many sampling steps) and traditional AR models (which require many autoregressive steps).
AGI Pathway: The work contributes to the unification of vision and language under a single autoregressive framework, demonstrating that AR models can rival diffusion models in quality while offering the speed and controllability benefits of autoregression.

In summary, FAR rethinks autoregressive image generation by shifting the regression direction from spatial tokens to spectral frequencies and utilizing continuous representations, resulting in a highly efficient, scalable, and high-quality generation paradigm.