💻 computer science

DreamTrajectory: Trajectory-Guided Action Generation with World Model Alignment for Mobile Manipulation

DreamTrajectory is a trajectory-guided framework for mobile manipulation that improves upon existing Vision-Language-Action policies by jointly predicting end-effector trajectories and whole-body actions to guide coordinated motion, while employing a lightweight world model for test-time refinement to align executed actions with planned trajectories, thereby significantly boosting success rates in both simulation and real-world contact-rich tasks.

Original authors: Zheng Yang, Wenjie Zhang, Xiangyu Chen, Wenxuan Song, Xianpeng Wang, Yihang Kang, Wen Chen, Lujia Wang, Renjing Xu, Xiaowen Chu

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zheng Yang, Wenjie Zhang, Xiangyu Chen, Wenxuan Song, Xianpeng Wang, Yihang Kang, Wen Chen, Lujia Wang, Renjing Xu, Xiaowen Chu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot that isn't just stuck on a table or a factory floor, but can roll around your living room, pick up a coffee mug, and hand it to you. This is the dream of mobile manipulation. It's a tricky corner of robotics because the robot has to do two things at once: drive its wheels to get to the right spot and move its arm to grab the object, all while the camera on its head sees the world spinning and shifting. Think of it like trying to juggle while riding a unicycle; if you focus only on the juggling, you might fall off the bike, but if you only focus on the bike, you drop the balls.

For a long time, robots learned to do this by looking at a picture and a command like "pick up the orange" and immediately spitting out a list of motor instructions for every wheel and joint. But this is like trying to write a novel by guessing every single letter at once without a plot outline. The robot often gets lost in the sheer number of choices, leading to clumsy movements or crashes. Furthermore, once the robot decides what to do, it just does it blindly, hoping for the best. If the floor is slippery or the object is heavier than expected, the robot doesn't realize its plan has failed until it's already dropped the mug. Scientists are trying to fix this by giving robots a better way to plan and a way to double-check their work before they move.

Enter DreamTrajectory, a new approach that acts like a smart director for a robot's performance. Instead of just guessing the final motor commands, this system first sketches out a "dream" path for the robot's hand (the end-effector) to follow. It's like a choreographer drawing a dance routine on a whiteboard before the dancers even start moving. The robot then generates the specific wheel and arm movements needed to trace that sketch. But here's the clever part: before the robot actually moves, it runs a quick mental simulation. It asks, "If I try this specific movement, will my hand actually end up where I planned?" It tests a few different versions of the move, picks the one that matches the "dream" path best, and only then executes it.

The researchers tested this idea in two ways. First, they ran it in a computer simulation called MS-HAB, where robots practice on virtual tables and fridges. They found that without this extra planning and checking, the robots only succeeded about 32.3% of the time. By adding the "dream" path, success jumped to 47.5%. When they added the mental simulation step to double-check the moves, the success rate climbed even higher to 54.8%. The improvement was especially huge for tricky tasks involving contact, like opening a fridge door or closing a counter, where the robot has to feel its way through the movement.

They didn't stop at the computer screen. They also tried this on a real physical robot, an ARX LIFT, in the real world. The results were even more impressive. On tasks like picking fruit and opening drawers, the robots went from succeeding 63.3% of the time to 81.7% with the path planning, and finally to 90.0% when they added the mental double-check. The system works by using a lightweight "world model"—a small, fast brain that predicts what happens next—to score different action choices. It doesn't need to be retrained every time; it just plugs in to help the main robot make smarter decisions on the fly.

The paper suggests that this method solves two big problems: it stops the robot from guessing blindly in a huge space of possible moves, and it stops the robot from executing bad ideas by checking them against a plan first. The authors note that the system is very efficient, adding only a tiny amount of extra computing power (about 11.75 milliseconds of delay per step) to get these big gains. While the results are promising, they are based on specific simulations and a limited set of real-world tasks, suggesting that while the approach is effective, it is a specific tool for this type of mobile robot, not a magic fix for every robotic problem in existence.

Technical Summary: DreamTrajectory

Problem Statement

Mobile manipulation requires robots to coordinate base locomotion and arm dexterity within unstructured environments, operating under continuously changing viewpoints and contact conditions. This creates an action space significantly larger and more sensitive to execution errors than fixed-base manipulation. The paper identifies two primary limitations in existing Vision-Language-Action (VLA) policies for this domain:

Implicit Motion Planning: Current VLAs map observations directly to whole-body action chunks without an explicit task-space motion plan. This lack of an intermediate representation makes coordinated base-arm prediction imprecise, as the intended motion remains implicit within the high-dimensional action space.
Open-Loop Execution: Existing policies execute predicted action chunks open-loop. They lack mechanisms to verify if the commanded actions will realize the intended motion before execution. Consequently, control errors, collisions, and unmodeled contacts accumulate, creating a gap between planned and realized motion that often leads to failure, particularly in contact-rich tasks.

Methodology: DreamTrajectory

The authors propose DreamTrajectory, a trajectory-guided framework that addresses these limitations through two core components: a joint generation policy and a test-time refinement procedure.

1. Trajectory-Guided Action Generation

Instead of mapping observations directly to actions, DreamTrajectory jointly predicts a planned end-effector trajectory and a whole-body action chunk within a single action expert.

Representation: The future end-effector motion is represented as a sequence of 7D poses (position and orientation) in a chunk-local frame anchored at the current base pose.
Architecture: The model uses a dual-stream Transformer trained via conditional flow matching.
- Trajectory Stream: Generates the intended end-effector path.
- Action Stream: Generates the base and arm commands to realize that path.
Group-Causal Attention: An asymmetric attention mask is employed where action tokens attend to the complete trajectory stream, but trajectory tokens do not attend to action tokens. This ensures the trajectory serves as an explicit, task-space guide for action generation without being influenced by the action generation process.

2. Test-Time Refinement via Trajectory World Model

To bridge the gap between planning and execution, DreamTrajectory introduces a lightweight action-conditioned trajectory world model and a search–predict–score procedure.

World Model: A recurrent model (specifically a GRU) predicts the end-effector trajectory that a candidate action chunk would induce, given the current observation and state. Unlike analytical models, this is learned from interaction data to capture deviations caused by contacts, self-collisions, and controller errors.
Refinement Procedure:
1. Sampling: The system samples $N$ candidate action chunks around the initial policy proposal (using Gaussian perturbations).
2. Prediction: The world model predicts the induced trajectory for each candidate in parallel.
3. Scoring: Candidates are ranked based on a score function that balances trajectory agreement (alignment between the predicted induced trajectory and the planned trajectory) and motion smoothness.
4. Selection: The highest-scoring candidate is executed. This process acts as a plug-in stage requiring no retraining of the main policy.

Key Contributions

The paper outlines three primary contributions:

Joint Generation Framework: Formulating mobile manipulation as the joint generation of a planned end-effector trajectory and a whole-body action chunk. This uses a group-causal attention mask to make the trajectory an explicit guide for coordinated base-arm control.
Lightweight Refinement Mechanism: Introducing an action-conditioned trajectory world model and a search–predict–score procedure. This operates in compact trajectory space rather than pixel space, allowing for efficient, test-time refinement without policy retraining.
Empirical Validation: Demonstrating consistent performance improvements on both simulated (MS-HAB) and real-world mobile manipulation tasks, showing that trajectory guidance and alignment significantly outperform action-only baselines.

Experimental Results

The authors evaluated DreamTrajectory on the MS-HAB benchmark (simulated) and a physical ARX LIFT platform (real-world).

Simulation (MS-HAB)

Evaluated on six set_table subtasks (e.g., picking objects, opening/closing fridges and counters):

Baseline ( $\pi_{0.5}$ ): 32.3% average success.
With Trajectory Guidance (no refiner): 47.5% average success.
Full DreamTrajectory (with refiner): 54.8% average success.
Notable Gains: The largest improvements were observed in contact-rich articulated-object tasks (e.g., "Open Fridge" improved from 5.0% to 51.0%; "Close Counter" from 38.0% to 80.0%).

Real-World Experiments

Evaluated on three tasks: fruit pick-and-place, drawer opening, and drawer closing.

Baseline ( $\pi_{0.5}$ ): 63.3% average success.
With Trajectory Guidance: 81.7% average success.
Full DreamTrajectory: 90.0% average success.
Specifics: Fruit pick-and-place improved from 45.0% to 80.0%; drawer opening from 60.0% to 90.0%.

Efficiency

The framework introduces minimal computational overhead:

The trajectory head adds only 0.017M parameters.
The world model adds 49.02M parameters (approx. 1.40% of the underlying 3.5B parameter VLA).
Total inference latency increase is approximately 11.75 ms per replanning step on an RTX 4090.

Significance and Claims

The paper claims that DreamTrajectory provides an effective and computationally lightweight approach to coordinated mobile manipulation. Its significance lies in:

Explicit Intermediate Representation: By introducing an explicit end-effector trajectory, the system decouples what needs to be done (task-space motion) from how to do it (whole-body control), improving precision in high-dimensional action spaces.
Consistency Mechanism: The test-time refinement allows the system to evaluate candidates against the policy's intended motion before execution, effectively filtering out actions that would fail due to unmodeled dynamics or contacts.
Complementarity: The results demonstrate that trajectory guidance and world-model refinement are complementary; guidance improves the initial generation, while refinement corrects residual plan-execution deviations, leading to robust performance in complex, contact-rich scenarios.