The Plan, Not the Decoder: Diagnosing and Repairing Compositional Failure in Reasoning-Augmented Text-to-Image Generation
This paper diagnoses compositional failures in reasoning-augmented text-to-image models by demonstrating that the decoder faithfully executes the generated plan while the planner itself is the bottleneck due to phrasing-dependent biases and geometric clutter, proving that editing the explicit plan rather than retraining the model effectively resolves these errors.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Modern computers have become remarkably skilled at turning written descriptions into pictures. If you ask one to draw a "red ball next to a blue box," it usually succeeds. But when the request becomes more complex, asking for specific relationships like "a cat sitting on top of a sofa" or "a dog to the left of a tree," these systems often stumble. They might draw the animals correctly but swap their positions, or place them in the wrong order. To fix this, researchers have begun teaching these models to think before they draw. Instead of jumping straight to the image, the computer first writes out a detailed plan: it lists the objects, describes their colors, and assigns each one a specific location on the canvas, like a set of coordinates. Only after this plan is written does the computer begin to generate the actual picture. This two-step process was designed to ensure the final image matches the user's intent, but it raised a new question: when the final picture is wrong, is the fault in the written plan, or is the fault in the part of the computer that turns the plan into pixels?
A researcher at Carnegie Mellon University set out to answer this by treating the computer's plan as a separate, testable object. They used a model that generates these textual plans and then creates images based on them. Their goal was to determine whether the computer was failing because it wrote a bad plan, or because it was a poor artist who could not follow a good plan. To find the truth, they did not just look at the final pictures; they intervened directly in the process. They took the computer's own plans, made specific changes to them, and watched what happened to the resulting images. For instance, they would take a plan that correctly described a cat on the left and a dog on the right, but then swap the location instructions in the text so the plan now said the cat was on the right. When they fed this altered plan back into the computer, the resulting image flipped perfectly: the cat appeared on the right and the dog on the left. This simple test proved that the part of the system responsible for drawing the image was actually very good at following instructions. It was not confused or inefficient; it was faithfully executing exactly what it was told.
The real problem, the researcher discovered, was not the artist but the planner. In nearly half of the cases where the final image was wrong, the written plan itself contained the error. The computer would write down a plan that contradicted the user's request, often due to a subtle bias in how it processed the order of words. For example, if a user asked for "a dog to the left of a cat," the computer might correctly understand the words but still write a plan that placed the dog on the right, simply because it had a habit of placing the first-mentioned object in a specific corner of the mental canvas. The researcher found that this bias was so strong that changing the wording of the same request—asking for "a cat to the right of a dog" instead—would cause the computer to write a correct plan 98 percent of the time, whereas the original phrasing led to errors in nearly half the attempts. The drawing engine was innocent; it was faithfully reproducing the mistakes made by the planning stage.
To confirm that the drawing engine was not the bottleneck, the researcher tried feeding it plans that it had never seen before, created by humans or other systems. They tested plans that were written in very different styles, some using short, blunt sentences and others using long, detailed descriptions. They also tested plans with different types of location instructions, some that were messy and overlapping and others that were clean and well-spaced. The results were clear: the computer ignored the style of the writing and the familiarity of the plan. It cared only about the geometry. When given a plan with clear, well-separated location instructions, the computer produced accurate images 13 percent more often than when it used its own, messy plans. Even when the plan was written in a style that looked nothing like the computer's usual output, the drawing engine followed it perfectly. This proved that the system was not "co-adapted," meaning it had not become so used to its own specific way of writing plans that it could no longer understand anything else. It was a general-purpose follower of instructions, limited only by the quality of the instructions it received.
The study also revealed that previous methods for testing these systems were misleading. Many researchers had been using a type of automated question-asking tool to judge whether the images were correct. The researcher found that this tool was blind to the actual layout of the picture. It could be tricked into giving a high score to an image that was completely wrong, simply because the objects were present, even if they were in the wrong places. By switching to a method that measured the actual positions of the objects with a precise detector, they uncovered the true extent of the errors. This new, more accurate measurement showed that the computer's ability to follow a plan was actually very high, around 94 percent, once the plan itself was correct.
The implications of these findings are practical and immediate. Since the drawing engine is reliable, the solution to bad images is not to retrain the entire system or to make the drawing engine smarter. Instead, the fix lies in improving the plan before the drawing begins. The researcher showed that by simply checking the plan for logical errors and fixing the location instructions before the image is generated, they could significantly improve the quality of the final picture without any extra training. They could even replace the computer's own plan entirely with a better one, and the system would produce a superior image. The only danger is if the plan contains internal contradictions, such as text that says one thing but location instructions that say another; in those cases, the computer gets confused and produces strange, fused images. But as long as the plan is consistent and clear, the system works as intended. The lesson is that for these reasoning-based image generators, the bottleneck is not the ability to draw, but the ability to write a good plan. Fixing the plan is the key to unlocking the full potential of the machine.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.