💻 computer science

Where Does Generative Difficulty Reside? An Empirical Study of Target Representations

This empirical study demonstrates that the choice of target representation in continuous masked generators fundamentally redistributes generative difficulty across contextual modeling, per-token denoising, and inference-time control, revealing that factors like compression or reconstruction fidelity alone cannot predict a model's generative performance.

Original authors: Marcel Plocher, Bernhard Schölkopf, Andreas Geiger, Gege Gao

Published 2026-08-04

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Marcel Plocher, Bernhard Schölkopf, Andreas Geiger, Gege Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Great Image Puzzle: Where Does the Hard Work Happen?

Imagine you are trying to teach a robot to paint a picture of a cat. You could show it the raw pixels—the millions of tiny red, green, and blue dots that make up the screen. Or, you could give it a simplified sketch, like a cartoon outline. Or, maybe you hand it a list of abstract ideas, like "fluffy," "whiskers," and "meow," and ask it to figure out the rest. This is the world of generative AI, where computers learn to create new images from scratch.

For a long time, scientists thought the "language" the robot used to learn didn't matter too much. They assumed that if you had a good translator (an encoder) to turn a photo into a list of numbers, and a good translator back (a decoder) to turn those numbers into a photo, the robot could learn the same way no matter which list of numbers you gave it. It was like thinking a student would learn math just as well whether you taught them in English, Spanish, or a secret code, as long as the teacher was good.

But this paper asks a tricky question: Where does the actual difficulty of learning to draw hide? Is the hard part figuring out the big picture (the context), or is it just filling in the tiny details (the local noise)? The authors wanted to see if changing the "language" the robot speaks changes where the brainpower needs to go. They didn't just guess; they built a single, super-flexible robot and made it try to learn four very different languages to see which one made the job easiest.

The Experiment: Four Languages, One Robot

The researchers set up a massive experiment using a unified robot model called a masked autoregressive rectified-flow generator. That's a mouthful, so let's break it down with a simple game.

Imagine a 16x16 grid of tiles, like a giant Sudoku board. The robot's job is to fill in the empty tiles to make a picture of a cat. But here's the catch: the robot can only see a few tiles at a time. It has to guess what the hidden tiles look like based on the ones it can see. To make this fair, the team gave this same robot four different "languages" to speak, all describing the same 256x256 pixel images:

Raw Pixels: The robot sees the actual, messy, colorful dots of the image.
SD-VAE Latents: The robot sees a compressed, "sketchy" version of the image, like a low-resolution drawing where the details are squished together.
DINOv2 Features: The robot sees a list of "ideas" or semantic concepts (like "fur texture" or "eye shape") extracted by a smart pre-trained brain.
MAE Features: Another list of ideas, but this one was trained specifically to be really good at reconstructing the original picture from a messy version.

The team trained all four versions of the robot on the same massive dataset (ImageNet) with the exact same amount of computer time and energy. They wanted to see: Which language made the robot learn fastest? Which one made the best pictures? And most importantly, where did the robot struggle the most?

The Big Surprise: It's Not About How "Clean" the Data Is

You might think the robot would do best with the "cleanest" or most "faithful" language. If you want to draw a cat, wouldn't you want the most accurate description of a cat? The paper tested this by looking at reconstruction fidelity—how perfectly the robot could turn its secret language back into a picture.

Here is the twist: The robot that was best at reconstructing the picture was NOT the one that generated the best new pictures.

The MAE language was the best at reconstruction. It could turn its notes back into a photo with incredible detail (an LPIPS score of 0.11). But when asked to create new images, it performed poorly.
The DINOv2 language was actually worse at reconstruction (LPIPS of 0.255). It lost some fine details when turning notes back into photos. However, when it came to generating new, beautiful images, DINOv2 crushed the competition. It reached a fantastic score (FID of 6.43) much faster than the others.

This proves that just because a language is good at describing an image perfectly doesn't mean it's good for creating one. The "best" language for learning isn't the one that keeps every single pixel perfect; it's the one that organizes the information in a way that helps the robot understand the story of the image.

Where Does the Difficulty Hide?

The paper's main discovery is that changing the language doesn't just change the data; it moves the difficulty to a different part of the robot's brain.

1. The "Context" vs. "Detail" Trade-off
Think of the robot's brain as having two parts: a Context Brain (which looks at the visible tiles and figures out the big picture) and a Detail Brain (which guesses the specific color of the hidden tile).

With DINOv2: The "Context Brain" had an easy job. Because DINOv2 tokens are full of high-level ideas (like "this is a cat's face"), the robot could figure out the big picture very quickly, even if only a few tiles were visible. The difficulty shifted entirely to the Detail Brain. The robot needed a very powerful, wide "Detail Brain" to figure out exactly how to paint those 768-dimensional idea-tokens into a real image.
With Raw Pixels: The "Context Brain" struggled. Looking at a few random colored dots doesn't tell you much about the whole cat. The robot had to work incredibly hard to guess the big picture. The difficulty was stuck in the Context part, and the robot needed a lot more time and computing power to learn.

2. The "One-Step" Magic
One of the coolest findings was about how fast the robot could finish the job.

The DINOv2 robot could generate a recognizable image in just one single step (like taking a guess and getting it right immediately).
The Pixel and SD-VAE robots fell apart if you tried to do it in one step. They needed many, many steps to slowly refine the image.
This suggests that DINOv2's language is so organized that the robot can jump straight to the answer, while the other languages force the robot to take a slow, winding path.

3. The Guidance Trap
The team also tested a technique called Classifier-Free Guidance, which is like giving the robot a gentle nudge or a hint ("Make it more like a cat!") to improve the picture.

SD-VAE and Pixels: These robots loved the hints. Without them, the pictures were blurry or weird. With strong guidance, they got much better.
DINOv2: This robot actually got worse with hints. It was already doing such a good job on its own that the extra nudging confused it.
This shows that you can't just use the same "training wheels" for every robot. What helps one language hurts another.

What This Means for the Future

The paper concludes that we can't just pick a language based on how "compressed" it is or how "faithful" it is to the original image.

Compression (making the data smaller) didn't guarantee it would be easier to learn.
Reconstruction Quality (how well you can rebuild the image) didn't predict how well the robot could create new ones.
Token Size (how many numbers describe a piece of the image) didn't matter either; DINOv2 and Pixels both used 768 numbers, but DINOv2 was way faster.

Instead, the choice of language redistributes the workload. It decides whether the robot has to work hard on understanding the big picture or on painting the tiny details. The "best" language is the one that moves the hard work to the part of the robot that is best at handling it.

In short, if you want a robot to learn to paint, don't just give it the most accurate dictionary. Give it a language that tells the story of the image in a way that lets the robot's brain focus on the right kind of thinking. For this specific setup, that language was DINOv2, which turned out to be the secret sauce for fast, high-quality image generation.

Technical Summary: Where Does Generative Difficulty Reside? An Empirical Study of Target Representations

Problem Statement

Generative models rely on a target representation to define the distribution they must learn. While this representation dictates output dimensionality, the extent to which it determines the difficulty of the learning process remains under-explored. A common assumption in the field is that target representations are interchangeable interfaces; once an encoder and decoder are available, a generative architecture should transfer across spaces with minimal modification.

This assumption is particularly questionable for continuous masked generators. These models decompose image generation into conditional token predictions, where a transformer aggregates visible tokens for contextual inference, and a local network models the conditional distribution of each missing token. The central question of this work is: Where does generative difficulty reside for different target representations? Specifically, does the choice of representation (e.g., raw pixels, compressed latents, or semantic features) shift the burden of learning between contextual modeling, local denoising, and inference-time control?

Methodology

The authors introduce a unified masked autoregressive rectified-flow model to isolate the effects of target representations. The architecture separates the generative process into two distinct components:

Context Network: A bidirectional transformer (based on MAE-style encoder-decoder) that processes visible tokens to produce contextual features ( $z_i$ ).
Local Denoiser: A residual MLP (conditioned via adaLN-Zero) that predicts the rectified-flow velocity for each masked token independently, using the noisy token, timestep, and context features.

The study evaluates four target spaces under a shared ImageNet 256×256 training budget (503k steps / 200 epochs) and identical token layouts (16×16 grid, 256 tokens):

Raw Pixels: Non-overlapping 16×16 RGB patches (768-dimensional tokens).
SD-VAE: Compressed latents from the Stable Diffusion VAE (16-dimensional tokens).
DINOv2: Features from a self-distilled vision transformer (768-dimensional tokens).
MAE: Features from a reconstruction-trained vision transformer (768-dimensional tokens).

The experiments systematically vary masking ratios, prediction parameterizations (velocity vs. clean-data prediction), denoiser capacity, context fusion strategies, and classifier-free guidance (CFG) schedules (linear vs. progressive).

Key Contributions

Unified Framework: The authors instantiate masked autoregressive rectified flow across raw pixels, compressed VAE latents, and two representation-autoencoder feature spaces within a common generative factorization.
Controlled Comparison: They compare optimization efficiency and inference behavior under a shared dataset, token layout, and training budget, documenting the specific modeling choices required for each space to converge.
Decoupling Representation Properties: The study demonstrates that standard representation properties (compression, reconstruction fidelity, dimensionality, and semantic clustering) do not individually predict generative behavior. Instead, the target representation redistributes the "difficulty" of generation across different stages of the pipeline.

Key Results

Optimization and Convergence

DINOv2 Superiority: DINOv2 converges significantly faster than both SD-VAE and raw pixels in terms of both training iterations and accumulated computation (GFLOPs). Without guidance, DINOv2 achieves an FID of 6.43, compared to 15.98 for SD-VAE and 43.60 for pixels.
Pixel Optimization: Raw pixel generation is substantially slower to optimize. Even with a larger parameter count (221.6M vs. ~165M for others), the pixel model shows no signs of convergence by the end of the shared budget.
MAE vs. DINOv2: Despite MAE providing superior reconstruction quality (lower LPIPS and Reconstruction FID) and clear semantic clustering, it produces significantly worse generations than DINOv2 under the same generator configuration.

Architectural Sensitivity

Context vs. Denoiser: DINOv2 performs well with a shallow (2-layer) context encoder but requires a wider local denoiser and direct context fusion to achieve optimal performance. This suggests that while DINOv2 provides globally informative context, predicting high-dimensional tokens from noise remains a difficult local task.
Pixel Requirements: Pixel generation requires specific interventions to become viable, including clean-data prediction ( $x$ -prediction), a bottleneck layer, a less aggressive masking distribution, and a stronger local denoiser.

Inference and Guidance

Guidance Dependence: The representations respond differently to classifier-free guidance (CFG).
- SD-VAE and Pixels: Benefit strongly from CFG. SD-VAE improves from FID 15.98 to 3.19 with progressive guidance; Pixels improve from 43.60 to 8.70 with linear guidance.
- DINOv2: Performs best without guidance (FID 6.43). Applying guidance degrades performance, indicating that the model's internal distributional control is already sufficient.
Precision-Recall Trade-offs: DINOv2 achieves high precision but low recall without guidance, whereas SD-VAE and MAE require guidance to balance these metrics.

Failure Modes

DINOv2: Produces sharp, natural-looking images but struggles with global coherence and multi-object separation (e.g., merging two animals into one structure).
Pixels: Preserve class identity and coarse layout but remain blurry with limited fine detail, even after architectural improvements.

Significance and Claims

The paper argues that the choice of target representation fundamentally alters where generative difficulty resides. It is not merely a matter of dimensionality or reconstruction quality. Instead:

Compressed spaces (SD-VAE) shift difficulty toward the need for strong guidance to recover high-fidelity details.
Semantic spaces (DINOv2) simplify contextual inference (allowing fast convergence and single-step generation) but place heavy demands on the local denoiser to handle high-dimensional token prediction.
Raw pixels impose a heavy burden on both contextual modeling and local denoising, requiring significantly more compute and specific architectural tweaks.

The authors conclude that compression, reconstruction fidelity, token dimensionality, and visible semantic clustering are individually insufficient to predict masked generative behavior. The target representation acts as a mechanism that redistributes the modeling burden across contextual inference, conditional token generation, flow integration, and distributional control. Consequently, a "one-size-fits-all" generative architecture is insufficient; the model design must be co-adapted to the specific properties of the target representation.