💻 computer science

Scaling Properties of Text Conditioning in Visual Generation

This paper reveals that diffusion loss scales with the amount of structured language in prompts, leading to a system that enhances both diffusability and promptability to outperform open-weight models and rival top closed-weight models on compositional and reasoning benchmarks.

Original authors: Zilong Chen, Chaorui Deng, Kunchang Li, Hongyi Yuan, Haoqi Fan

Published 2026-08-03

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Zilong Chen, Chaorui Deng, Kunchang Li, Hongyi Yuan, Haoqi Fan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot artist how to paint. For a long time, the secret sauce seemed to be giving the robot more and more words to read. The logic was simple: if you describe a scene with a short sentence, the robot might miss details. So, people started writing longer, more flowery paragraphs, hoping that more words would equal a better picture. This field of study is called "text-to-image generation," where computers turn written descriptions into visual art. But there's a catch: the robot doesn't just read words; it tries to match those words to pixels it has seen before. If the words are vague or just repetitive, the robot gets confused, no matter how many words you use. The big question researchers have been asking is: does the length of the description matter, or does the amount of useful information matter?

This paper dives into that exact mystery. The researchers discovered that simply adding more words to a prompt is like trying to fill a bucket with a leaky hose; eventually, you just get wet feet without filling the bucket. They found that the real magic happens when you stop writing long stories and start writing structured, organized "blueprints." Instead of a paragraph, they use a list of specific facts, like "a red ball at coordinates (10, 20)" or "a cat sitting on a blue chair." By organizing the information this way, the robot artist can actually see the picture in its mind before it starts painting. The paper suggests that this structured approach allows the computer to learn much faster and create much better images, especially for complex scenes with many objects.

The "More Words" Myth vs. The "Better Map" Reality

For years, the AI art world operated on a simple assumption: if you want a better picture, write a longer description. It seemed logical. If you tell a human, "Draw a dog," they might draw a generic dog. If you say, "Draw a fluffy golden retriever with a red collar running on a sunny beach," they draw something much more specific. So, researchers and hobbyists started feeding AI models massive paragraphs, hoping that more text would force the AI to pay attention to more details.

But the authors of this paper, working at ByteDance Seed, decided to test if this "more is better" rule actually holds up. They set up a clever experiment using a "reconstruction probe." Imagine you have a single, perfect photograph of a living room. You then ask an AI to describe that room in four different ways: a short sentence, a medium paragraph, a long essay, and a super-long novel. Crucially, all these descriptions talk about the exact same things; they just get wordier and more repetitive.

When they fed these descriptions back into the AI to see if it could redraw the room, something surprising happened. As the descriptions got longer, the quality of the redrawn room didn't get better. It stayed exactly the same. The AI hit a wall. It turned out that once the AI knew the basic facts (there's a sofa, there's a lamp), adding more adjectives and flowery prose didn't help it understand the image any better. The extra words were just noise. The paper explicitly rules out the idea that simply increasing the token count (the number of words) improves the image quality. In fact, for many existing models, making the prompt too long actually made the results worse.

The Secret Weapon: Structured Prompts

If long paragraphs don't work, what does? The authors propose a shift from "storytelling" to "blueprinting." They introduced something called a Structured Prompt (SP). Instead of writing a sentence like, "There is a silver pickup truck driving down a street with some boxes in the back," the structured prompt breaks the image down into a rigid, organized list of facts, almost like a computer code or a JSON file.

Think of it like the difference between giving a chef a vague story about a meal versus giving them a precise recipe with exact measurements.

Natural Language (The Story): "A silver pickup truck transports cardboard boxes, a wrapped cylinder, and Coca-Cola cans on a street."
Structured Prompt (The Blueprint):
- Vehicle: Silver pickup truck
- Cargo: Cardboard boxes, wrapped cylinder, Coca-Cola cans
- Location: Street
- Position: [Specific coordinates for the truck]
- Depth: [How far away the truck is]

The paper found that when the AI was trained on these structured blueprints, the quality of the images improved dramatically. The more detailed the blueprint (adding things like depth, specific positions, and relationships between objects), the better the AI got at drawing the scene.

To prove this wasn't just a fluke, the researchers measured the "information" in the prompts using two clever tools:

GPG (Grounded Perplexity Gain): This measures how much the image helps the AI understand the text. It's like asking, "Does seeing the picture make the sentence make more sense?"
ED (Effective Detailness): This checks how many specific facts in the text actually match the picture.

They discovered a direct link: the more "information" (facts) the prompt had, the lower the AI's training error became. It's a straight line. More facts = better learning. More words (without new facts) = no change.

The Two-Step Dance: The Writer and The Painter

The paper breaks the whole process down into two distinct roles, which they call Diffusability and Promptability.

Diffusability (The Painter's Skill): This is about how well the AI "painter" can read the instructions. The researchers found that if you give the painter a structured blueprint, they can learn the job much faster and make fewer mistakes. They trained their AI model on thousands of images converted into these blueprints. The result? The model became incredibly good at following complex instructions, like drawing five chairs where only the second and fifth have people sitting on them.
Promptability (The Writer's Skill): This is the challenge of turning a user's simple request ("Draw a cool robot") into that perfect structured blueprint. Since users don't speak in blueprints, the team built a special "Prompter" (a large language model) to do the translation.
- First, they taught this Prompter how to write blueprints by showing it examples (Supervised Fine-Tuning).
- Then, they used a "Cold-Start" method where the AI learned to imagine the details it couldn't see, using logic to fill in the gaps.
- Finally, they used a "Verifier" system. The AI would write a blueprint, the painter would draw it, and a judge would look at the result. If the drawing was bad, the judge would tell the writer exactly what was wrong (e.g., "You forgot the depth of the robot's arm"), and the writer would try again. This process, called Reinforcement Fine-Tuning, made the Prompter incredibly sharp.

The Result: A New Standard

When they combined the super-organized blueprints with the super-smart Prompter, the results were impressive. Their system beat almost every other open-source AI model on tests that required complex reasoning, like counting objects, understanding spatial relationships (left vs. right), and following detailed instructions.

For example, when asked to draw a scene with specific lighting and multiple objects, their system got the details right far more often than models that just relied on long, messy paragraphs. They even showed that you could edit specific parts of the image easily. If you wanted to change the "material" of a chair from wood to metal, you just changed that one field in the blueprint, and the AI redrew the chair without messing up the rest of the room.

What This Means for the Future

The paper suggests that the future of AI art isn't about writing longer essays. It's about organizing information better. By treating image generation like a structured engineering problem rather than a creative writing exercise, we can get much more reliable and detailed results.

The authors also tested what happens if you let the AI "think" longer. They created a loop where the AI could generate a draft, get criticized, and fix it. They found that while a few rounds of fixing helped, the trained system was so good at getting it right the first time that it didn't need many tries. This suggests that training the AI to think in structures is more powerful than just giving it more time to guess.

In short, the paper argues that we've been trying to teach AI artists by shouting more words at them. The real breakthrough comes from teaching them to read a clear, organized map. It's not about how much you say; it's about how clearly you say it.

Technical Summary: Scaling Properties of Text Conditioning in Visual Generation

Problem Statement

Current text-to-image (T2I) generation systems often struggle with information-dense prompts involving complex layouts, object relationships, and visual coherence. While Large Language Models (LLMs) have advanced through scaling model size, data, and compute, T2I models face a fundamental asymmetry: they learn from image-caption pairs where the caption acts as a conditioning signal. If a caption ambiguously binds visual content or omits details, that information fails to reach the model effectively, even if present in the pixels.

The paper investigates whether increasing the information content of text conditioning, rather than merely increasing the token count, can improve visual generation. Empirical observations suggest that extending natural language (NL) prompts often degrades performance or leads to early saturation, as verbosity does not necessarily equate to increased image-grounded supervision. The authors posit that the limiting factor is not the length of the prompt, but the amount of structured, image-grounded information it exposes to the diffusion model.

Methodology

The proposed framework addresses this limitation through two complementary factors: Diffusability (how effectively the caption representation exposes information to the diffuser) and Promptability (how effectively an LLM prompter translates user requests into that representation).

1. Quantifying Caption Informativeness

To move beyond token counts, the authors introduce two metrics to measure the "image-grounded information" in a caption:

Grounded Perplexity Gain (GPG): A white-box metric measuring the increase in a caption's likelihood under a frozen Vision-Language Model (VLM) when the paired image is revealed. It sums the log-likelihood gain over content tokens.
Effective Detailness (ED): A black-box metric measuring the precision and recall of caption attributes against an image-grounded reference set, using an F0.5 score that emphasizes precision.

2. Scaling Properties of Text Conditioning

Through controlled training sweeps on a fixed diffusion backbone (BAGEL) and fixed image data, the authors establish empirical scaling laws:

Diffusion loss does not scale with NL length: Increasing the token count of natural language captions yields negligible improvements in converged diffusion loss.
Diffusion loss scales with structured information: Converged diffusion loss decreases approximately linearly with GPG and follows a power law with ED.
Structured Prompts (SP): The authors propose a typed JSON schema that organizes image-grounded variables into named fields (Global Scene, Per-Element properties/geometry, and Cross-Element relations). This format allows for the systematic exposure of information (e.g., bounding boxes, depth, specific attributes) that NL prose often obscures.

3. Raising Diffusability (Training the Diffuser)

To improve the diffuser's ability to learn from structured information:

Image-to-SP Annotation Pipeline: A five-stage pipeline uses a VLM combined with frozen domain experts (Sapiens for pose, DepthAnything V2 for depth, SAM 2.1 for segmentation) to generate full-schema SPs from training images.
Controlled Field Ladder: The authors create a hierarchy of SP levels (L5–L10), deterministically masking field groups (e.g., removing bounding boxes, then depth, then scene context) to isolate the contribution of specific information types to training loss.
Training: The diffuser is trained on these structured prompts, learning to map the explicit fields to image generation.

4. Raising Promptability (Training the Prompter)

Since no paired image exists at inference time, an LLM prompter must infer the necessary visual details to fill the SP schema:

Scaling: Zero-shot evaluation shows that larger LLMs (up to 397B parameters) and chain-of-thought reasoning improve the quality of generated SPs.
Training Pipeline: The prompter is refined through a three-stage process:
1. Supervised Fine-Tuning (SFT): Teaches the model the target SP distribution using (user prompt, SP) pairs.
2. Cold-Start Distillation: Bootstraps image-free reasoning by training on traces generated by an image-conditioned teacher, filtered to ensure the reasoning is grounded in the prompt rather than leaking image details.
3. Reinforcement Fine-Tuning (RFT) with On-Policy Self-Distillation (OPSD): The prompter generates rollouts which are rendered by the diffuser. A verifier gates high-quality trajectories, and the prompter is updated via KL-divergence minimization against an image-conditioned teacher on these accepted trajectories.
Inference-Time Refinement: An agentic loop allows for iterative "refine–render–judge" cycles, where a verifier critiques the output and the prompter targets specific field edits to correct failures.

Key Results

Scaling Properties

The study confirms that information, not token count, is what scales. Natural language prompts saturate in information gain, while structured prompts continue to provide new, image-grounded facts as fields are added.
The relationship between caption informativeness (GPG/ED) and converged diffusion loss is strong ( $r \approx -0.98$ for GPG, $r \approx -0.97$ for ED), allowing the prediction of training outcomes based on caption quality before full training runs.

End-to-End Performance

The resulting system (Qwen-Image backbone with structured prompter and diffuser) outperforms all evaluated open-weight models (including FLUX.1, Hunyuan 3.0, BAGEL, Emu3) on nearly every benchmark:

GenEval2: Achieves 90.6/72.5 (AM/GM), significantly outperforming the official Qwen-Image prompt enhancer (82.4/52.8).
CoReBench: Scores 85.2, surpassing the next best open model (76.1) and matching or exceeding strong closed-weight models.
DPG-Bench & WISE: Shows broad gains in dense prompt following and world-knowledge generation.

Ablation Studies

Matched NL Control: Retraining the same architecture and data with a free-form NL interface fails to reproduce the gains of the structured system, confirming the benefit comes from the representation, not just additional training.
Prompter Training: The SFT–Cold-start–RFT pipeline yields substantial improvements in structure and GSB (Good/Same/Bad preference) over zero-shot baselines.
Inference Scaling: While iterative refinement improves results, the trained prompter internalizes most corrections, showing diminishing returns after ~2 rounds of refinement.

Significance and Claims

The paper claims to establish empirical scaling properties for text conditioning in visual generation, shifting the focus from prompt length to prompt information content. Its primary contributions are:

Measurement: The adaptation of GPG and ED to quantify caption informativeness and demonstrate their predictive power over diffusion loss.
Diffusability: Showing that structured prompts (SPs) organize image-grounded content into addressable fields, lowering diffusion loss without architectural changes to the backbone.
Promptability: Demonstrating that scaling and training an LLM prompter to instantiate these structured representations significantly improves generation quality, particularly for compositional and reasoning-heavy tasks.
End-to-End System: Proving that combining these factors yields a system that leads open-weight models and competes with closed systems, with the widest margins on tasks requiring spatial reasoning and complex composition.

The authors conclude that while iterative inference provides additional gains, the primary driver of performance is the training-time alignment of the caption interface (diffusability) and the prompter's ability to generate high-fidelity structured representations (promptability). The work suggests that future scaling in T2I should prioritize the richness and structure of the conditioning signal over mere token expansion.