🤖 AI

Latent Action Control for Reasoning-Guided Unified Image Generation

This paper proposes Latent Action Control (LAC), a method that enhances unified image generation by representing reasoning as hidden continuous actions within a shared backbone, thereby enabling models to effectively translate inferred knowledge and spatial relations into high-quality visual outputs without generating intermediate reasoning tokens or images.

Original authors: Fuxiang Zhai, Sixiang Chen, Yingjin Li, Shuaibo Li, Jianyu Lai, Tengjun Huang, Lei Zhu

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fuxiang Zhai, Sixiang Chen, Yingjin Li, Shuaibo Li, Jianyu Lai, Tengjun Huang, Lei Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant architect (the AI) who is incredibly good at reading blueprints and understanding what a house should look like. However, when this architect tries to actually build the house, they often get the details wrong. They might understand that the prompt asks for "a red house with a blue door on the left," but the final building ends up with a green door on the right.

This paper, titled "Latent Action Control for Reasoning-Guided Unified Image Generation," proposes a new way to fix this gap between "understanding" and "building." They call their solution LAC (Latent Action Control).

Here is how it works, broken down into simple concepts and analogies:

The Problem: The "Silent Gap"

Current AI models that can both understand images and create them often suffer from a disconnect. They can "think" about the right answer (e.g., "I know I need a cat on a mat"), but that thought doesn't automatically translate into the correct pixels in the final image. It's like a chef who knows exactly how a dish should taste but keeps forgetting to add the salt while cooking.

The Solution: The "Internal Rehearsal"

Instead of making the AI write out a long list of instructions (like "Step 1: Draw a cat. Step 2: Draw a mat..."), LAC gives the AI a secret internal rehearsal space.

Think of LAC as a director on a movie set who whispers instructions directly into the actor's ear while the scene is being filmed, rather than handing them a script to read beforehand.

The AI goes through four specific "roles" or stages of this internal rehearsal, all happening in a hidden, invisible space (the "latent" space):

Plan: The AI figures out the big picture. (e.g., "Okay, we need a sunset, a beach, and a dog.")
Draft: The AI creates a rough, invisible sketch in its mind. It doesn't show this to the user; it's just a mental hypothesis to check if the idea makes sense.
Diagnosis: The AI checks its own mental sketch. (e.g., "Wait, the dog is too big for the beach, and the sun is in the wrong spot.")
Refine: The AI makes tiny, invisible adjustments to fix those errors before the final picture is even started.

How It Learns: The "Teacher's Cheat Sheet"

Since the AI is doing this thinking in a secret, invisible space, the researchers couldn't just tell it, "Here is the correct thought process." Instead, they used a clever training trick:

The Teacher: They used a "teacher" model to create perfect, structured notes (like a script) for how to solve a problem.
The Translation: They turned these text notes into visual images (like a flowchart or a diagram) that the AI could "see" during training.
The Alignment: The AI learned to mimic the feeling of those visual notes by generating its own invisible "actions."
The Result: Once training was done, the teacher's notes were thrown away. The AI now knows how to generate these invisible "actions" on its own to guide the painting process.

The "Final Polish": Learning from the Result

After the AI learns the basics, they use a method called LF-GRPO. Imagine a game where the AI tries to paint a picture, gets a score based on how good it looks, and then uses that score to adjust both the final painting and the invisible "rehearsal" steps it took to get there. This ensures that the internal thinking process is actually helping the final result.

What They Found (The Results)

When they tested this new system (called LAC) against other top AI models:

Better Details: It got much better at following complex instructions, like "a red car next to a blue tree."
Spatial Awareness: It was much better at getting positions right (left vs. right, top vs. bottom).
World Knowledge: It understood real-world facts better (e.g., knowing that a cat is smaller than a dog, or that the sun rises in the east).

The Proof: "Turning Off the Switch"

To prove that this invisible "rehearsal" was actually doing the work, the researchers did a cool experiment. They took the trained AI and randomized or deleted the invisible actions during the generation process.

The Result: The image quality dropped significantly.
The Conclusion: This proved that the AI wasn't just "thinking" for fun; it was actively using those invisible steps to control how the image was built.

In Summary

LAC turns the AI's "thinking" into a set of invisible, continuous actions that guide the image creation process from the inside out. Instead of just understanding a prompt and then blindly trying to draw it, the AI now has a structured, internal "rehearsal" (Plan, Draft, Diagnose, Refine) that ensures the final image matches the prompt perfectly. It's like giving the artist a set of invisible hands to guide the brush, ensuring the final painting is exactly what was imagined.

Technical Summary: Latent Action Control for Reasoning-Guided Unified Image Generation

Problem Statement

Unified multimodal models have successfully integrated visual understanding and image generation within shared backbones. However, a persistent "understanding–generation gap" remains: models may correctly infer objects, relations, or constraints during the understanding phase but fail to instantiate them faithfully in the generated image. Existing approaches to bridge this gap typically externalize reasoning as natural language plans, textual checklists, or iterative edit instructions. These methods treat reasoning as artifacts outside the generator's hidden-state dynamics, requiring reinterpretation as conditioning signals. Alternatively, latent reasoning methods move deliberation into continuous spaces but often fail to formulate these states as role-specific, executable actions that directly steer synthesis. Consequently, there is a missing control interface to convert inferred constraints into generation-time conditioning signals before visual synthesis begins.

Methodology: Latent Action Control (LAC)

The authors propose Latent Action Control (LAC), a framework that transforms reasoning into an internal control interface for unified image generation. Instead of decoding reasoning into text, LAC emits continuous latent actions that are written back into the model's hidden stream to condition the generation process.

1. Role-Structured Latent Action Control

LAC operates by rolling out a latent action trajectory $Z$ organized into four distinct, ordered roles:

Plan: Global intent planning.
Draft: Formation of an internal visual hypothesis (single-step, no intermediate image emitted).
Diagnosis: Failure diagnosis.
Refine: Targeted refinement.

These actions are continuous vectors ( $z \in \mathbb{R}^d$ ) injected into the hidden stream of a unified generator (instantiated on BAGEL-7B-MoT). The trajectory length is variable, determined by a learned halting head, and the process concludes with a sentinel token. The model conditions the subsequent flow-based image generation on the hidden states resulting from this latent rollout.

2. Prior-Guided Variational Latent Action Alignment

Since ground-truth action trajectories are unavailable, LAC employs a prior-guided variational latent action alignment strategy during training:

Teacher Signals: Structured teacher records (for plan, diagnosis, refine) are rendered as compact, document-style images and encoded as dense target embeddings. For the "draft" role, targets are derived from visual features of imperfect candidate images.
Variational Objective: The model learns an autoregressive policy $q_\theta$ to sample actions. A fixed-variance Gaussian prior is constructed from the teacher signals ( $E$ ). The policy is trained to align its sampled rollout with these priors using a variational loss ( $L_{var}$ ), ensuring the actions are executable without directly injecting teacher signals as inputs during inference.
Supervised Fine-Tuning (SFT): The total objective combines variational alignment, a lexical anchor loss (to ground latent states in semantics), a supervised halting loss, and an image flow-matching loss.

3. Outcome-Aligned Latent-Flow GRPO (LF-GRPO)

To optimize the latent actions based on final image quality rather than just local teacher priors, the authors introduce Latent-Flow GRPO (LF-GRPO).

Mechanism: For a given prompt, a group of rollouts is sampled. Each rollout consists of a latent action trajectory and a flow-generation trajectory.
Reward: A terminal image reward is computed for the final generated image.
Credit Assignment: Using Group-Relative Advantage (GRPO), the terminal reward is assigned as a shared advantage signal to both the continuous latent actions and the flow-generation transitions. This allows image-level feedback to shape the hidden action trajectory, aligning the planning, drafting, diagnosis, and refinement steps with the final visual outcome.

Key Contributions

Formulation of Reasoning as Control: The paper formulates reasoning in unified image generation as Latent Action Control, where role-specific continuous actions convert inferred constraints into generation-time conditioning signals, keeping deliberation within the hidden continuous space.
Prior-Guided Alignment & LAC-15K: The authors introduce a training methodology using prior-guided variational alignment and construct LAC-15K, a role-structured dataset providing rendered semantic priors, visual draft targets, and supervised halting signals.
LF-GRPO Objective: A novel outcome-level objective that jointly optimizes hidden latent actions and flow-generation transitions under shared terminal image rewards, enabling the generator to learn how to use latent reasoning to improve synthesis.

Experimental Results

The method was evaluated on GenEval (compositional generation) and WISE (world-knowledge grounded generation), using BAGEL-7B-MoT as the base.

GenEval: LAC improved the overall score from 0.77 (BAGEL baseline) to 0.82. The gains were most significant in constraint-sensitive categories: Position (+0.11), Colors (+0.13), and Color Attribute (+0.05). With LLM-rewritten prompts, the score reached 0.90.
WISE: LAC improved the overall weighted score from 0.49 to 0.54, with notable gains in Biology (+0.18) and Chemistry (+0.07).
Ablation Studies:
- Training Pathway: Removing rendered semantic priors or visual draft grounding (Latent-SFTNR) resulted in no improvement over the baseline, confirming that the specific nature of the priors is crucial.
- Latent Intervention: Replacing learned latent actions with zero vectors or random vectors caused significant performance drops (e.g., -7.92 on GenEval), proving that the generator actively consumes these actions. Shuffling the role order also degraded performance, validating the functional necessity of the plan-draft-diagnose-refine structure.
- Role Removal: Removing the "draft" role caused the largest drop, highlighting the importance of internal visual hypotheses for attribute binding and object tracking.

Significance and Claims

The paper claims that unified image generation benefits significantly when understanding is not merely encoded but made actionable during the generation process. By converting inferred constraints into an ordered, hidden action trajectory, LAC provides a direct control path from reasoning to synthesis. The results suggest that the "understanding–generation gap" can be narrowed by treating reasoning as an internal, executable control interface rather than an external textual artifact. The authors note that while the approach shows consistent gains, it currently relies on teacher-derived priors and requires further work to reduce teacher dependence and improve reinforcement learning efficiency.