🤖 machine learning

Rethinking Generative Image Pretraining: How Far Are We From Scaling Up Next-Pixel Prediction?

This paper investigates the scaling laws of autoregressive next-pixel prediction for vision models, revealing that optimal strategies diverge between classification and generation tasks and forecasting that compute-driven scaling will enable pixel-by-pixel image modeling within five years.

Original authors: Xinchen Yan, Chen Liang, Lijun Yu, Adams Wei Yu, Yifeng Lu, Quoc V. Le

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Xinchen Yan, Chen Liang, Lijun Yu, Adams Wei Yu, Yifeng Lu, Quoc V. Le

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to "see" the world. There are two main ways to do this:

The "Smart Summary" Method: You show the robot a picture, but you first compress it into a few key words or symbols (like turning a photo of a cat into the word "cat"). This is how most modern AI works today.
The "Raw Pixel" Method: You show the robot the picture exactly as it is, dot by dot, color by color, asking it to guess the very next dot based on the ones before it. This is like trying to learn a language by memorizing every single letter in a book, one by one, without ever seeing a whole word.

This paper asks a simple but difficult question: How far away are we from making the "Raw Pixel" method work really well?

The authors decided to test this by training AI models to predict the next pixel in an image, starting with small, low-resolution images (32x32 pixels, which look like tiny, blocky thumbnails). They wanted to see if this simple, "dumb" method could eventually scale up to become as powerful as the "smart summary" methods.

Here are the four big discoveries they made, explained with everyday analogies:

1. The "Pixel vs. Word" Gap

The Finding: To learn effectively, raw pixels need 10 to 20 times more data than words do.
The Analogy: Imagine teaching a child to read.

Words (Language): If you show a child the word "Cat," they instantly understand a furry animal that meows. It's a dense package of information.
Pixels (Images): If you show a child a single red dot, they have no idea what it is. Is it a nose? A stop sign? A strawberry? It's just a dot.
Because a single dot carries so little meaning, the AI has to look at millions more dots to figure out the pattern. The paper found that to get the same level of intelligence, the AI needs to "read" a massive amount more pixel-data than it would need to read text.

2. Different Goals Need Different Recipes

The Finding: The "best" way to scale up the AI depends entirely on what you want it to do.
The Analogy: Think of training an athlete.

If you want them to be a Sprinter (Image Classification): You need a balance of heavy lifting (bigger muscles/model) and running practice (data).
If you want them to be a Marathon Runner (Image Generation): You need to focus almost entirely on running practice (more data).
The paper found that if you want the AI to recognize what's in a picture, you can just make the AI "brain" bigger. But if you want the AI to create or complete a picture (like filling in the bottom half of a photo), you need to feed it a much larger library of examples (data) rather than just making the brain bigger. The "best" recipe for one task is actually a bad recipe for the other.

3. The Resolution Trap: Bigger Pictures Need Bigger Brains

The Finding: As images get higher resolution (sharper and more detailed), the AI needs to get much bigger much faster than the amount of data it sees.
The Analogy: Imagine trying to learn a city map.

Low Resolution (32x32): It's like looking at a map of a small town. You can learn the whole thing by walking around a lot (more data).
High Resolution (64x64 and up): It's like looking at a map of a massive metropolis with skyscrapers and tiny alleyways. Just walking around more won't help if your brain is too small to hold all the complex details.
The paper discovered that as images get sharper, the "complexity" of the task explodes. To handle this, you can't just feed the AI more pictures; you have to build a much, much larger AI brain to handle the intricate details.

4. The Bottleneck is Brainspower, Not Books

The Finding: We aren't waiting for more pictures to make this work; we are waiting for faster computers.
The Analogy: Imagine you have a library with every book ever written (the internet's visual data). You have a student who can read them all. The problem isn't that the library is empty; the problem is that the student is reading one page a year.
The authors predict that because computer power is growing so fast (doubling every year or so), we will be able to train these "Raw Pixel" models effectively within the next five years. The data is already there; we just need the computing muscle to process it.

The Bottom Line

The paper concludes that training AI to learn directly from raw pixels is not a dead end. It works, it follows predictable rules, and it can eventually reach high levels of performance. However, it is currently very expensive and slow because pixels are "dumb" data points.

To make this work in the future, we don't need to invent new ways to compress images; we just need to keep building bigger computers and feeding them massive amounts of data, specifically tailored to whether we want the AI to recognize things or create them.

Technical Summary: Rethinking Generative Image Pretraining

Problem Statement

Despite the success of autoregressive next-token prediction in natural language processing, the paradigm of next-pixel prediction for vision has seen a decline in popularity. Modern approaches often favor multi-stage generation or patch-level learning with image tokenizers (e.g., VQ-VAE, VQGAN) to mitigate the computational burden of raw pixel sequences. However, the fundamental question remains unanswered: How far are we from scaling up next-pixel prediction?

This paper investigates the scaling properties of autoregressive next-pixel prediction as a unified framework for vision. The authors address three core challenges:

Semantic Density: Pixels carry significantly less semantic information than language tokens, potentially requiring different scaling laws.
Spatial Complexity: Pixels exhibit complex spatial dependencies that are difficult to model sequentially.
Resolution Scalability: The complexity of prediction grows quadratically with image resolution, making high-resolution modeling computationally prohibitive.

The study aims to determine if raw pixel modeling follows predictable scaling laws, how these laws differ across tasks (recognition vs. generation), and whether the approach is feasible given current and projected compute budgets.

Methodology

Experimental Setup

Model Architecture: The authors utilize a family of standard Transformer architectures equipped with RMSNorm, 1D RoPE, and Gated Linear Units (GEGLU). They test five model scales ranging from 10 million to 449 million parameters.
Dataset: Pre-training is conducted on JFT-300M, a large-scale dataset with a long-tailed distribution reflecting open-world visual frequencies, rather than the more uniform ImageNet ILSVRC 2012.
Training Objective: Standard negative log-likelihood for autoregressive next-pixel prediction ( $Pr(x_i | x_{<i})$ ).
Scaling Strategy: The study employs IsoFLOP profiles (following Hoffmann et al., 2022) to identify compute-optimal configurations. For fixed compute budgets, the authors vary model depth and width to find the optimal token-to-parameter ratio.
Resolutions: Experiments are conducted at 16×16, 32×32, and 64×64 resolutions. The 32×32 resolution serves as the primary baseline for extensive experimentation due to computational tractability.

Evaluation Metrics

To assess scaling trends across different objectives, the authors evaluate three distinct metrics:

Next-Pixel Prediction Loss: The primary training objective.
ImageNet Classification Accuracy: Measured via linear probing (freezing the backbone and training a linear head) to assess representation quality.
Generation-Based Completion Quality: Measured using Fréchet Distance (FD). The top half of an image serves as a spatial prompt, and the model autoregressively predicts the bottom half.

Key Findings and Results

1. Inefficiency of Raw Pixels Compared to Text

Learning on raw pixels is significantly less efficient than learning on language tokens.

Token-to-Parameter Ratio: To achieve compute-optimal performance, raw pixel models require 10–20× more tokens per parameter than language models. While language models often operate at a ratio of ~20 tokens/parameter, pixel models require ratios exceeding 400.
Implication: A single pixel contains minimal semantic information compared to a word (e.g., "cat"), necessitating a much larger volume of data relative to model size to learn effective representations.

2. Task-Dependent Scaling Strategies

The optimal scaling strategy is not universal; it diverges significantly based on the target metric, even at a fixed resolution (32×32).

Classification vs. Generation:
- Classification: Optimal scaling favors a balance where model size and data size grow at similar rates ( $N_{opt} \propto C^{0.56}$ , $D_{opt} \propto C^{0.44}$ ).
- Generation: Optimal scaling for image generation requires a data-heavy approach. The dataset size must grow 3–5 times faster than the model size ( $N_{opt} \propto C^{0.49}$ , $D_{opt} \propto C^{0.51}$ ).
Conclusion: A scaling strategy optimized for next-pixel loss or classification is sub-optimal for image generation. Generation tasks demand vast, diverse data to capture high-order dependencies and long-tail visual distributions.

3. Resolution-Dependent Scaling Shifts

As image resolution increases, the optimal scaling strategy shifts from balancing data and model size to heavily favoring larger models.

Model vs. Data: At higher resolutions (e.g., 64×64), the optimal number of parameters per pixel grows rapidly, while the optimal number of images (data) grows much slower.
Reasoning: Higher resolutions inherently contain information from lower-resolution sub-sequences. Once a sufficient amount of data is available, the marginal value of new data points diminishes, and the complexity of the visual structure requires larger model capacities to model fine-grained textures and global structures.
Performance: Increasing resolution from 32×32 to 64×64 yields significant improvements in generation quality (Fréchet Distance) but only marginal gains in ImageNet classification accuracy.

4. Feasibility Projections

The authors project the requirements for achieving competitive performance (80% ImageNet accuracy and Fréchet Distance < 100) using data-driven theoretical estimates.

Compute Bottleneck: The primary barrier to scaling next-pixel prediction is compute, not data availability. Given the sheer volume of visual data on the internet, the data requirement is not the limiting factor.
Timeline: Assuming training compute for frontier AI models grows 4–5 times annually, the authors forecast that pixel-by-pixel modeling will become feasible within the next five years.
Projections: At $10^{24}$ FLOPs, a model could theoretically reach ~85% ImageNet accuracy (at 64×64) and a Fréchet Distance of ~93, requiring a model size of ~80B parameters and ~0.5B images (at 64×64).

Significance and Claims

The paper claims to provide the first rigorous, large-scale analysis of the scaling laws for raw next-pixel prediction across both recognition and generation tasks.

Re-evaluating the Paradigm: The work challenges the notion that next-pixel prediction is inherently unscalable or inferior. Instead, it demonstrates that the paradigm follows predictable power-law scaling, provided the compute budget and data-to-parameter ratios are adjusted correctly.
Unified Vision Models: The findings suggest that a single end-to-end framework can unify vision recognition and generation, though the optimal compute allocation differs significantly between the two tasks.
Future Viability: The study concludes that while current compute requirements for high-resolution raw pixel modeling are impractical, the trajectory of compute growth suggests that this approach will become a viable and competitive path for unified vision models in the near future, potentially obviating the need for complex tokenization schemes.

The authors maintain a modest tone, acknowledging that their projections rely on the stability of power-law exponents and that empirical validation is currently limited to resolutions up to 64×64. They note that further research is needed to explore scaling beyond these resolutions and to unify scaling trends across more complex downstream tasks like object detection.