GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learning

GoT-R1 is a novel framework that leverages reinforcement learning with a dual-stage multi-dimensional reward system to enhance the semantic-spatial reasoning capabilities of multimodal large language models, significantly improving their ability to generate images from complex prompts involving precise object relationships and attributes.

The Gist

Imagine you are an artist who has just been hired to paint a very specific scene based on a client's description. The client says, "Paint a blue butterfly sitting on the left side of a red candle."

In the past, AI artists (like the ones we have today) were like improvisational jazz musicians. They heard the words and immediately started painting. They were great at making things look realistic, but if the prompt was complex, they often got the details wrong. They might paint the butterfly on the right side, or make it green instead of blue, because they were just guessing the layout as they went along.

Then, researchers introduced a system called GoT (Generation Chain-of-Thought). This was like giving the artist a strict script to follow before painting. The AI would first write down a plan: "Okay, I need a red candle at coordinates (100, 100) and a blue butterfly at (50, 50)." Then, it would paint based on that script. This helped, but the script was written by humans using fixed templates. The AI couldn't think outside the box; if the script was slightly off, the painting was still wrong.

Enter GoT-R1.

Think of GoT-R1 as taking that artist and putting them through a high-intensity "Trial and Error" boot camp using a very smart, critical art critic (an AI called an MLLM).

Here is how it works, step-by-step:

1. The "Brainstorming" Phase (Reinforcement Learning)

Instead of just following a script, the AI is told: "Here is the prompt. Now, try to write 16 different plans (reasoning chains) for how to paint this."

Some plans will be terrible (butterfly on the wrong side). Some will be okay. One might be perfect.

2. The "Critic" Phase (The Reward System)

This is where the magic happens. In previous systems, the AI only got a grade at the very end: "Did the final painting look good?"

GoT-R1 introduces a Dual-Stage Critic. The smart AI critic looks at the process in two ways:

• The Plan Check: Does the written plan actually match what the client asked for? (e.g., Did the plan say "left" when the client said "left"?)
• The Execution Check: Did the final painting match the plan? (e.g., If the plan said "left," did the paint actually go on the left?)

The critic gives a score for every single step. If the AI writes a plan that puts the butterfly on the right, it gets a low score immediately, even before it paints a single pixel. If the plan is good but the painting is messy, it gets a low score there too.

3. The "Self-Improvement" Phase

The AI looks at all 16 attempts. It sees which ones got the highest scores from the critic. It then learns: "Ah! When I put the butterfly on the left and describe the candle first, I get a high score. When I mix them up, I get a low score."

It uses a method called GRPO (Group Relative Policy Optimization). Imagine a group of students taking a test. Instead of just telling the teacher "I got an A," the teacher compares all the students. The AI learns by seeing which strategies worked better than the others in the group, rather than just trying to hit a perfect score.

Why is this a big deal?

• From Scripted to Creative: Before, the AI was like an actor reading a script word-for-word. Now, it's like a method actor who understands the intent of the scene and figures out the best way to act it out on its own.
• The "Visual" Advantage: One cool trick the researchers used is that the AI critic is bad at reading numbers (like "x=100, y=200"). So, they turned the numbers into visual boxes on a blank canvas. It's like showing the critic a drawing of where the objects should be, rather than a list of numbers. The critic can then "see" if the plan makes sense spatially.

The Result

The paper shows that GoT-R1 is much better at following complex instructions. If you ask for "a cat on a dog's head, next to a tree," the old AI might put the cat next to the dog. GoT-R1 understands the spatial relationships and the attributes (colors, shapes) much better because it learned to think before it draws, and it learned how to think by being graded on both its thinking and its drawing.

In short: GoT-R1 teaches AI to stop guessing and start planning, using a smart critic to grade both the plan and the final artwork, resulting in images that actually look like what you asked for.

Technical Summary

1. Problem Statement

While modern text-to-image (T2I) models have achieved high fidelity in generating realistic images, they struggle with complex compositional prompts. These prompts require precise spatial relationships (e.g., "a butterfly on the left of a candle") and strict attribute binding (e.g., "a red car and a blue bike").

• Limitation of Current Methods: Existing models often map text embeddings directly to visual features without explicit reasoning, leading to spatial misalignment and attribute swapping.
• Limitation of GoT (Generation Chain-of-Thought): The previous GoT framework introduced an intermediate reasoning step (decomposing prompts into object descriptions with coordinates) but relied on Supervised Fine-Tuning (SFT) with human-defined templates. This constrained the model to predefined patterns, preventing it from autonomously discovering more effective reasoning strategies. Furthermore, SFT often produced reasoning chains that were syntactically correct but semantically unfaithful to the prompt.

2. Methodology: GoT-R1 Framework

The authors propose GoT-R1, a framework that applies Reinforcement Learning (RL) to enhance the semantic-spatial reasoning capabilities of autoregressive visual generation models. The system is built upon a unified Multimodal Large Language Model (MLLM) architecture (specifically Janus-Pro) that generates text reasoning chains followed by image tokens.

A. Training Strategy

The training process consists of two stages:

1. Supervised Fine-Tuning (SFT): The base model is fine-tuned on the GoT dataset to establish a baseline capability for generating templated reasoning chains and images.
2. Reinforcement Learning (RL): The model is further optimized using Group Relative Policy Optimization (GRPO). Unlike standard RL which requires a separate critic model, GRPO samples a group of $N$ outputs (reasoning chains + images) for a single prompt, evaluates them, and updates the policy based on relative performance within the group.

B. Core Innovation: Dual-Stage Multi-Dimensional Reward Framework

The most significant technical contribution is the design of a comprehensive reward system evaluated by an MLLM-based Reward Model. Instead of relying solely on the final image quality, the system evaluates the entire generation pipeline through four distinct reward signals:

1. Prompt-to-Reasoning Semantic Reward ($R_{sem}$): Evaluates if the generated reasoning chain captures all concepts, attributes, and logical consistency from the input prompt.
2. Prompt-to-Reasoning Spatial Reward ($R_{spa}$): Evaluates if the spatial coordinates in the reasoning chain match the prompt's spatial relationships (e.g., "left," "top").
   • Innovation: To overcome the poor spatial understanding of text-based LLMs regarding coordinates, the system renders the bounding boxes as visual images on a blank canvas before feeding them to the MLLM for evaluation.
3. Reasoning-to-Image Reward ($R_{RI}$): Measures the alignment between the planned reasoning (bounding boxes) and the actual generated image. It calculates the Intersection over Union (IoU) between the planned coordinates and the grounded object locations in the generated image.
4. Prompt-to-Image Reward ($R_{PI}$): Evaluates the overall fidelity, composition, and aesthetic quality of the final generated image against the original prompt.

Total Reward Calculation:
The total reward is a product of these components (with semantic and spatial rewards averaged):
$$R_{total} = R_{PI} \times \frac{(R_{sem} + R_{spa})}{2} \times R_{RI} \times R_{HPS}$$
(Where $R_{HPS}$ is a human preference score metric).

3. Key Contributions

1. Autonomous Reasoning Discovery: GoT-R1 enables autoregressive models to move beyond fixed templates and autonomously discover superior reasoning strategies for complex scenes through RL.
2. Dual-Stage Reward System: A novel framework that provides explicit supervision for both the intermediate reasoning process and the final output, solving the "misalignment" problem where good reasoning leads to bad images (or vice versa).
3. Visualized Spatial Evaluation: An innovative method to improve spatial reasoning evaluation by converting textual bounding box coordinates into visual representations for the MLLM reward model.
4. State-of-the-Art Performance: The framework successfully transfers sophisticated reasoning capabilities from language models to the visual generation domain.

4. Experimental Results

The authors evaluated GoT-R1 on T2I-CompBench and GenEval, comparing it against diffusion models, two-stage layout models, and other autoregressive approaches.

• T2I-CompBench: GoT-R1-7B achieved State-of-the-Art (SOTA) results, outperforming all baselines. It showed a 15% improvement in the "Complex" composition category over the baseline GoT-finetuned model.
• GenEval: The model achieved an overall score of 0.75 (SOTA). Notable gains were observed in:
  • Two-Object Generation: Improved from 0.69 to 0.94.
  • Attribute Binding: Improved from 0.43 to 0.68.
• General Quality: On the COCO 2014 validation set, GoT-R1-7B showed significant improvements in CLIP Score, Aesthetic Score, and Human Preference (77% preference over baselines).
• Ablation Studies:
  • Removing the reasoning reward ($R_{PR}$) caused significant performance drops, proving that supervising the reasoning process is critical.
  • The visualized spatial reward ($R_{spa}$) was shown to be superior to direct text-coordinate evaluation.
  • Self-explored reasoning chains from GoT-R1 were overwhelmingly preferred by GPT-4o over the predefined templates used in standard GoT.

5. Significance

GoT-R1 represents a paradigm shift in text-to-image generation by treating image generation as a sequential reasoning task rather than just a direct mapping task.

• Bridging the Gap: It effectively bridges the gap between the reasoning capabilities of Large Language Models (LLMs) and the generative capabilities of visual models.
• Scalability: By using GRPO and MLLM-based rewards, the framework avoids the need for expensive human annotation of reasoning templates, allowing models to self-improve.
• Future Impact: This approach opens new avenues for creating highly controllable, context-aware, and compositionally accurate visual content, which is essential for applications requiring precise scene composition (e.g., design, robotics, and storytelling).

The code and checkpoints are made available to the public, facilitating further research in autoregressive visual generation and reinforcement learning.