💻 computer science

Planning with Unified Multimodal Models

The paper introduces Uni-Plan, a novel planning framework leveraging unified multimodal models to simultaneously function as policy, dynamics, and value functions, while employing self-discriminated filtering to mitigate hallucinations and achieve superior performance in embodied decision-making without requiring expert demonstrations.

Original authors: Yihao Sun, Zhilong Zhang, Yang Yu, Pierre-Luc Bacon

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yihao Sun, Zhilong Zhang, Yang Yu, Pierre-Luc Bacon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to play a complex board game, like moving a red pentagon to a specific spot on a table.

Most current AI robots are like blindfolded chess players. They can read the rules and talk about the game very well (using text), but they can't actually see the board in their mind as they plan. They have to guess what will happen if they move a piece, and often, they guess wrong because they can't visualize the result.

This paper introduces Uni-Plan, a new way to teach robots to think and plan. Instead of being blindfolded, Uni-Plan gives the robot a "mental movie projector."

Here is how it works, broken down into simple parts:

1. The "Swiss Army Knife" Robot Brain

Usually, building a robot planner requires three different experts:

The Planner: Decides what move to make next.
The Predictor: Guesses what the world will look like after that move.
The Judge: Decides if that future looks good or bad.

Uni-Plan is special because it uses one single brain (a Unified Multimodal Model) to do all three jobs at once. It can read your instructions, imagine the future picture, and judge if that picture is a win—all in one go.

2. The "Mental Movie" (Dynamics Model)

The core of Uni-Plan is its ability to generate images. When you tell it, "Move the red block to the blue star," it doesn't just say "Okay." It actually draws a picture of what the table looks like after the move.

Think of it like a child playing with LEGOs. Before they actually move a brick, they close their eyes and imagine where it will land. Uni-Plan does this with images. It creates a "mental movie" of the future to see if the plan makes sense.

3. The "Self-Correcting" Filter (Self-Discriminated Filtering)

Here is the tricky part: Sometimes, the robot's "mental movie" is wrong. It might hallucinate (imagine) that a block disappeared or moved to a place it couldn't possibly reach. This is called a "hallucination."

To fix this, the authors gave the robot a second brain that acts as a strict editor.

Step 1: The robot imagines a future (e.g., "I moved the block here").
Step 2: The "Editor" looks at that future picture and asks, "If I saw this picture, what move would have caused it?"
Step 3: If the Editor says, "That picture would only happen if you moved the block left, but you told me you moved it right," the robot knows the picture is fake. It throws that bad prediction away.

This is like a writer who writes a story, then reads it backwards to check if the plot holes make sense. If they don't, they delete that version and try again.

4. The Results: Why It Wins

The researchers tested this on six different tasks, from navigating mazes to rearranging objects on a real table.

Better than Text-Only: Robots that only use text (like standard chatbots) failed often because they couldn't visualize the physical world. Uni-Plan succeeded much more often because it could "see" its mistakes before making them.
Learning from Mistakes: Usually, robots need to be taught by experts showing them the perfect moves. Uni-Plan learned just as well (and sometimes better) using "non-expert" data—basically, it learned by watching people make random moves and figuring out the patterns, without needing a perfect teacher.
Speed: Because it uses one model to do everything, it is much faster than systems that try to stitch together different tools.

The Big Takeaway

The paper argues that to make smart robots, we shouldn't just make them better at talking; we need to make them better at visualizing. By letting the AI generate images of the future and then checking if those images are logical, the robot becomes much more reliable at solving physical problems.

In short: Uni-Plan is a robot that doesn't just think about the future; it draws the future, checks if the drawing makes sense, and then acts on the best version.

Technical Summary: Uni-Plan – Planning with Unified Multimodal Models

Problem Statement

Recent advancements in Large Language Models (LLMs) and Vision-Language Models (VLMs) have spurred interest in using them for decision-making and planning. However, existing approaches predominantly rely on text-based reasoning, where visual inputs are often processed only at the initial stage. This unimodal limitation restricts the model's ability to accurately represent the current state and reason about spatial or visual dynamics throughout the planning process. While some works attempt to integrate images by invoking external tools (e.g., code execution for plotting or separate vision models for segmentation), these methods lack adaptability and generalization. Furthermore, Unified Multimodal Models (UMMs), which natively support both multimodal inputs and outputs, have shown promise but suffer from a "curse of the horizon." Specifically, they struggle to act as faithful dynamics models, often failing to predict accurate next-state observations, particularly in stochastic environments or when precise causal effects are required.

Methodology

The authors propose Uni-Plan, a planning framework that leverages a single Unified Multimodal Model (UMM) to simultaneously serve four distinct roles: (i) a policy, (ii) a dynamics model, (iii) a self-discriminator, and (iv) a value function. The framework is built upon the BAGEL model, a state-of-the-art open-source UMM.

Core Components

UMM as a Dynamics Model: The system requires the model to predict the next visual observation ( $o_{h+1}$ ) conditioned on the current observation ( $o_h$ ) and a textual action ( $a_h$ ). While standard UMMs can perform basic image editing, they often fail to maintain causal consistency in complex or stochastic scenarios (e.g., object rearrangement with multiple valid outcomes).
Self-Discriminated Filtering: To address the inaccuracy of dynamics predictions, the authors introduce a novel filtering mechanism. The UMM is jointly trained for inverse dynamics inference ( $P^{-1}_{UMM}(o_h, o_{h+1}) \to a_h$ ). During planning, the model generates multiple candidate next observations. It then operates in inverse dynamics mode to infer the action that would lead from the current state to each candidate. Candidates are retained only if the inferred action matches the actual executed action. This process, combined with object-count consistency checks, effectively filters out hallucinated or invalid transitions.
Planning Algorithms:
- Open-Loop Planning: Used for fully observable environments. The policy (a prompted UMM) proposes candidate actions, and the dynamics model rolls them out. A beam search is performed in the imagined state space, where successor states are filtered via self-discriminated filtering and scored by a value function (a finetuned UMM predicting remaining steps to the goal).
- Closed-Loop Planning: Used for partially observable environments. Inspired by the "proposal, simulation, and revision" paradigm, the policy generates multiple candidate action sequences. The dynamics model simulates these sequences to their horizon, and the value function selects the most promising sequence. The agent executes only the first $L$ actions, observes the new state, and replans, enabling continual feedback correction.

Training Strategy

Unlike many VLM-based planners that require expert demonstrations for fine-tuning, Uni-Plan is trained on non-expert trajectories. The forward and inverse dynamics models are fine-tuned on transition data, while the policy component requires no fine-tuning. The value function is trained on labeled data to estimate the number of steps to a goal.

Key Contributions

Self-Discriminated Filtering: A novel technique where the generative model acts as its own discriminator to filter invalid dynamics predictions, significantly improving the reliability of the dynamics model without requiring external verification tools.
Unified Planning Framework (Uni-Plan): A system where a single UMM integrates the roles of policy, dynamics model, self-discriminator, and value function, enabling integrated planning through generated visual content.
Superior Performance and Scalability: The framework demonstrates substantial improvements over VLM-based methods in embodied decision-making tasks. It achieves high success rates without expert demonstrations and exhibits strong data scalability, outperforming VLMs even when trained on the same amount of non-expert data.

Experimental Results

The authors evaluated Uni-Plan on six embodied decision-making tasks spanning 2D navigation, manipulation, and 3D navigation (including FrozenLake, Mini-BEHAVIOR, Language Table, Real World, Active Recognition, and Image-Goal Navigation).

Performance: Uni-Plan achieved nearly 30% higher success rates compared to open-source VLM-based planning methods (including CoT variants) across all tasks. It also matched the performance of the powerful, closed-source GPT-5-Thinking model.
Ablation Studies:
- Dynamics Model: Fine-tuned BAGEL served as a highly generalizable dynamics model, significantly outperforming models trained from scratch on Out-of-Distribution (OOD) layouts.
- Filtering: Self-discriminated filtering was shown to be critical; removing it led to a significant drop in planning success rates, confirming that better search algorithms cannot compensate for poor transition modeling.
- Data Efficiency: Uni-Plan reached strong performance with just 500 trajectories, whereas VLM baselines failed to achieve competitive results even with four times that amount of data.
Efficiency: By using a single model to generate images rather than videos (as in Video Language Planning), Uni-Plan achieved roughly a 10× speedup in inference time compared to video-based planning baselines.

Significance and Claims

The paper posits that Uni-Plan represents a shift from using generated images merely for the visualization of reasoning traces to using them as intermediates for counterfactual reasoning. By simulating multiple possible futures under different actions, the model actively evaluates alternative trajectories rather than just expositing its thought process.

The authors claim that this work lays a foundation for future research in reasoning and decision-making with UMMs. They highlight that the ability to unify policy, dynamics, and value functions within a single model, reinforced by self-discriminated filtering, offers a more robust and data-efficient approach to embodied AI than current text-centric or tool-dependent paradigms. The work suggests that the inherent multimodal generation capabilities of UMMs, when properly harnessed, can significantly strengthen visual reasoning and planning performance.