💻 computer science

Make Your VLA More Robust Without More Data By Interleaving Motion Planning

This paper introduces MPVI, a framework that interleaves model-based motion planning with Vision-Language-Action (VLA) models to significantly enhance robustness and task progress in long-horizon mobile manipulation without requiring additional training data.

Original authors: Dan BW Choe, Sundhar Vinodh Sangeetha, Samuel Coogan, Shreyas Kousik

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dan BW Choe, Sundhar Vinodh Sangeetha, Samuel Coogan, Shreyas Kousik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to clean your house. You give it a big, complicated instruction: "Go to the living room, find three soda cans, bring them to the kitchen, and put them in the trash can."

Current "super-smart" robots (called Vision-Language-Action models or VLAs) are like talented but easily overwhelmed students. They have read millions of books and watched thousands of videos of people cleaning. However, when faced with a long, multi-step task in a messy room, they often get lost, bump into things, or forget what they were supposed to do next. They try to do everything at once using only their "brain," and they make mistakes that pile up until the whole task fails.

The paper introduces a new method called MPVI (Motion Planner / VLA Interleaving). Think of MPVI not as a new student, but as a smart project manager who pairs the talented student with a reliable GPS navigator.

Here is how it works, broken down into simple parts:

1. The Problem: The "All-in-One" Trap

The old way was to ask the robot's brain to do everything: figure out where the trash can is, walk there without hitting the sofa, pick up the can, and put it in the bin.

The Analogy: It's like asking a brilliant chef to also drive the delivery truck, navigate through traffic, and park the car, all while cooking a complex meal. If the chef gets distracted by the traffic, the meal burns. If they get lost, the food arrives cold. The paper shows that even with huge amounts of training data, these "all-in-one" robots still fail at long tasks because they get confused by the distance and the clutter.

2. The Solution: The "Hybrid Team"

The authors built a system that splits the job into two distinct roles, switching back and forth between them:

The Navigator (The Motion Planner): This is the robot's "GPS." It doesn't need to be smart or creative; it just needs to be good at math and geometry. Its job is to get the robot from Point A to Point B without crashing. It uses a map of the house to plot a perfect, collision-free path.
The Doer (The VLA): This is the "talented student." Once the robot is right next to the object (like the soda can), the Navigator hands over control. Now, the VLA uses its vision and language skills to figure out how to grab the can and put it in the bin.

The Magic Switch: The system constantly checks, "Are we there yet?" If the robot is far away, the Navigator drives. If the robot is close, the Doer acts.

3. The Secret Sauce: "Proprioceptive Triggers"

One of the biggest problems in these systems is knowing when a step is actually finished.

The Old Way: The robot would constantly ask a super-computer (a Vision-Language Model), "Did I finish?" every few seconds. This is expensive and prone to "hallucinations" (the computer might think the job is done when it's not, just because it saw a picture that looked similar).
The New Way (MPVI): The system waits for a physical signal. It only asks, "Is this done?" when the robot's arm actually stops moving or the gripper closes.
The Analogy: Imagine a waiter. Instead of asking the chef every 10 seconds, "Is the steak ready?", the waiter waits until the chef slams the pan down and says, "Done!" The waiter then knows it's time to serve. This prevents the robot from getting confused and moving on to the next task too early.

4. The Results

The team tested this on a benchmark called BEHAVIOR-1K, which simulates 1,000 different household tasks.

They compared their "Hybrid Team" (MPVI) against the best "All-in-One" robot from a recent competition.
The Outcome: The Hybrid Team improved the robot's success rate by 113%.
Why? The Navigator handled the boring, difficult parts (walking through a cluttered room to find a hidden object), while the Doer handled the tricky parts (grasping the object). The robot didn't need to learn anything new; it just needed a better way to organize its existing skills.

Summary

The paper argues that we don't necessarily need more data or smarter AI to fix robot failures. Instead, we need to be smarter about how we combine different tools. By letting a simple, reliable planner handle the walking and letting the smart AI handle the grabbing, the robot becomes much more robust and less likely to get lost or confused in a long, messy task.

Technical Summary: MPVI – Motion Planner / VLA Interleaving

Problem Statement

Vision-Language-Action (VLA) models have demonstrated significant progress in mobile manipulation but continue to struggle with long-horizon tasks, particularly in cluttered household environments. These tasks require robots to execute extended sequences of spatially distributed subtasks while maintaining progress toward a high-level goal.

Despite finetuning on large-scale human teleoperated datasets (e.g., the 1,200 hours of data provided for the 2025 BEHAVIOR Challenge), top-tier end-to-end VLA models achieve poor success rates (e.g., 12.4% on the BEHAVIOR-1K benchmark). Systematic failure analyses reveal that these models suffer from:

Navigation failures: Inability to traverse cluttered scenes or locate distant/occluded objects.
Execution-order confusion: Errors in the sequence of subtasks.
Spatial grounding issues: Difficulty in maintaining accurate state tracking over long horizons.
Error compounding: Early execution errors rapidly degrade performance over the task horizon.

The authors posit that these issues represent fundamental architectural bottlenecks that cannot be resolved simply by scaling training data or model size.

Methodology: MPVI Framework

To address these challenges without requiring additional data or training, the authors propose MPVI (Motion Planner / VLA Interleaving). This is a hybrid framework that integrates a classical model-based motion planner with a learned VLA policy. The system is designed to be modular, treating the LLM, VLM, VLA, and motion planner as interchangeable black boxes.

The framework consists of five key modules:

Subtask Planner (LLM):
- Decomposes the natural language task instruction into a sequence of short subtasks.
- Tags each subtask as either navigation or manipulation.
- Assigns state-based completion criteria (predicates over world states, e.g., "can is in trash") rather than action-based predicates, as state predicates are more easily verifiable via single images.
Orchestrator:
- Routes subtasks to the appropriate module: the classical motion planner for navigation and the VLA for manipulation.
- Groups consecutive manipulation subtasks into a single execution unit for the VLA.
- Maintains a knowledge base of discovered rooms, fixtures, and objects to prevent redundant visual searches during subsequent navigation segments.
- Manages failure handling: if a module exceeds a time budget, the orchestrator marks the subtask as failed and proceeds to the next.
Navigation Module (Classical Motion Planner):
- Handles all base motion using A* pathfinding.
- Relies on a pre-existing traversability map with labeled rooms (a standard assumption in household robotics).
- For target object localization, it queries the knowledge base first. If the location is unknown, it employs open-vocabulary object detection (GroundingDINO) combined with frontier-based exploration to locate the object.
- The VLA is only engaged once the robot is within a specific "handoff radius" of the target object.
Manipulation Module (VLA):
- Executes manipulation subtasks using a pre-trained VLA (specifically openpi-comet based on $\pi0.5$ ).
- Receives subtask descriptions and RGB images from head and wrist cameras.
- Operates until the completion checker signals termination.
Completion Checker (VLM with Proprioceptive Trigger):
- Determines when a manipulation subtask is complete to trigger a switch back to the motion planner.
- Key Innovation: Instead of periodic VLM polling (which is computationally expensive and prone to hallucinations), the system uses proprioceptive triggers.
- A completion check is only initiated when proprioceptive signals (e.g., gripper closing or arm state) indicate that a manipulation primitive (like a pick or place) has just terminated. This significantly reduces false positives that could cause cascading failures.

Key Contributions

MPVI Framework: A novel architecture that interleaves classical motion planning with VLAs, demonstrating that a motion planner can serve as a "steering mechanism" to mitigate failures in long-horizon mobile manipulation.
Proprioception-Triggered Completion Checking: A mechanism that grounds semantic visual verification in physical robot state, preventing the cascading failures caused by VLM hallucinations during task transitions.
Empirical Validation: A comprehensive evaluation on the BEHAVIOR-1K benchmark showing that hybrid approaches can outperform state-of-the-art end-to-end VLAs even when the VLAs are trained on large-scale whole-body datasets.

Experimental Results

The authors evaluated MPVI on the 50-task subset of the BEHAVIOR-1K benchmark using the OmniGibson simulator, comparing against openpi-comet (the second-place solution in the 2025 BEHAVIOR Challenge).

Performance Gain: MPVI achieved a 113% improvement in mean task progress (Q-Score) over the end-to-end VLA baseline.
Task Coverage: Improvements were observed in 31 out of 50 tasks, with the method matching the baseline in 16 tasks and showing regression in only 3 tasks.
Failure Mode Mitigation: The method specifically addressed navigation failures, collisions, and execution-order confusion, which were identified as primary failure modes in the baseline.
Efficiency: When comparing episodes with matching task progress (Q-scores), MPVI showed a 4.5% reduction in base distance traveled and a 9.8% reduction in normalized simulation time.
Ablation Studies:
- Removing the motion planner (relying only on subtask decomposition) reduced performance by 31% relative to MPVI.
- Removing the proprioceptive trigger (using periodic polling) reduced performance by 47%, highlighting the critical role of reliable completion checking.
- Using ground-truth object locations (Oracle) improved Q-scores by 17% over MPVI, suggesting that while perception is a factor, the primary bottleneck in the baseline was navigation and planning logic, not just object detection.

Significance and Claims

The paper claims that more data alone is not the solution to the poor performance of VLAs on long-horizon tasks. Instead, the integration of classical planning with learned policies offers a path to robustness without additional training data.

The authors argue that:

Long-horizon tasks expose architectural limitations in purely end-to-end VLAs regarding memory, spatial reasoning, and error propagation.
Classical motion planning is explicitly designed to solve problems like navigating cluttered scenes and computing collision-free paths, making it a natural complement to VLAs.
The proposed "recipe" for interleaving—specifically the use of proprioceptive triggers for module switching—provides a principled way to combine the strengths of learned policies (flexibility, semantic understanding) and classical planners (reliability, geometric precision).

The work suggests that future development in VLA scaling should focus on principled criteria for deferral (knowing when to switch to a classical component) rather than solely on increasing model capacity or dataset size.

Limitations

The authors acknowledge two primary limitations:

Dependency on Maps: The framework assumes access to a pre-mapped environment with labeled room boundaries. It does not currently handle online semantic mapping or vision-language navigation from scratch.
Manipulation Bottlenecks: The integration is limited to navigation; manipulation is fully delegated to the VLA. Consequently, the method does not improve performance on tasks where the primary bottleneck is dexterous manipulation (e.g., attaching a camera to a tripod), as the failure originates in the manipulation policy itself.