NovaPlan: Zero-Shot Long-Horizon Manipulation via Closed-Loop Video Language Planning

NovaPlan is a hierarchical framework that enables zero-shot long-horizon robotic manipulation by integrating closed-loop vision-language planning for high-level task decomposition and error recovery with video-generated kinematic priors for robust low-level action execution.

The Gist

Imagine you are teaching a robot to build a complex Lego castle, but you've never shown it how to do it before, and you can't give it step-by-step instructions. You just say, "Build this castle," and point at the pile of blocks.

Most robots would freeze. They might try to grab a block, drop it, and then get stuck because they don't know what to do next.

NovaPlan is a new way to teach robots that works like a creative director with a time machine. Instead of just guessing, the robot uses a "video generator" to imagine the future, checks if its imagination makes sense, and then acts on the best version.

Here is how it works, broken down into simple steps:

1. The "Daydream" Phase (Video Planning)

Think of the robot's brain as having a movie studio inside it.

• When the robot needs to do something (like "stack the red block"), it doesn't just calculate math. It dreams up several short video clips of what that action could look like.
• It might imagine: "What if I grab the block from the left?" vs. "What if I grab it from the right?"
• It generates these videos instantly, showing the robot's hand moving the block perfectly.

2. The "Critic" Phase (Checking the Script)

Now, the robot has a film critic (an AI that understands language and logic).

• The critic watches all the generated video clips.
• It asks: "Did the block actually move? Did it fall through the table (which is impossible)? Did the robot grab the wrong block?"
• If a video shows the block floating in mid-air or melting, the critic throws it in the trash.
• It picks the one video that looks the most realistic and physically possible.

3. The "Action" Phase (Copying the Dance)

Once the best video is chosen, the robot needs to turn that 2D movie into 3D muscle movements. This is where NovaPlan gets clever.

• The Problem: Sometimes the robot can't see the block clearly because its own hand is blocking the view (occlusion). If it tries to track the block, it gets lost.
• The Solution: NovaPlan looks at the human hand in the generated video instead.
  • Analogy: Imagine you are trying to learn a dance move, but the dancer is wearing a baggy coat that hides their feet. It's hard to copy. But if you can see their arms and shoulders clearly, you can guess where their feet are going.
  • NovaPlan tracks the human hand in the video. Even if the block is hidden, the hand is usually visible. The robot copies the hand's movement to figure out where to move its own gripper.

4. The "Safety Net" (Closed-Loop Recovery)

This is the most important part. In the real world, things go wrong. The block might slip, or the table might be wobbly.

• Old robots would try to follow the plan blindly, fail, and then stop forever.
• NovaPlan is like a GPS that recalculates.
  • After every move, the robot checks: "Did I actually get where the video said I would?"
  • If the answer is "No" (e.g., the block is still on the table), the robot doesn't panic. It goes back to the "Daydream" phase.
  • It says, "Okay, the block is still here. Let me imagine a new video where I poke the block gently to nudge it into place."
  • It generates a new "recovery video," picks the best one, and tries again.

Why is this a big deal?

• Zero-Shot: It doesn't need to be trained on thousands of hours of videos for this specific task. It can figure out how to build a new type of puzzle it has never seen before, just by using its general understanding of how the world works.
• Long-Horizon: It can handle tasks with many steps (like building a whole tower) without getting confused halfway through.
• Dexterous: It can do tricky things, like poking a stuck object with a finger to free it, rather than just trying to grab it again and again.

The Bottom Line

NovaPlan is like giving a robot a superpower: the ability to imagine the future, check if the imagination is real, and fix its mistakes on the fly without needing a human to hold its hand. It turns a robot from a rigid machine into a flexible problem-solver that can "think" before it acts.

Technical Summary

1. Problem Statement

The paper addresses the challenge of zero-shot long-horizon robotic manipulation. While Vision-Language Models (VLMs) and video generation models can decompose tasks and "imagine" outcomes, they often lack the physical grounding required for real-world execution. Existing methods face three primary limitations:

1. Temporal Inconsistency: Video models often suffer from hallucinations and temporal drift over long durations, degrading performance.
2. Execution Fragility: Real-world execution fails when robots cannot track visual plans due to occlusion, depth inaccuracies, or geometric warping in generated videos.
3. Rigid Planning: Open-loop strategies fail to balance strategic foresight (needed for coupled assembly) with reactivity (needed for error recovery).

Current approaches struggle to translate high-level video plans into precise, low-level control without task-specific training or demonstrations, particularly when dealing with complex, multi-step tasks involving occlusion or non-prehensile interactions.

2. Methodology: NovaPlan Framework

NovaPlan is a hierarchical, closed-loop framework that unifies high-level VLM reasoning with low-level geometrically grounded execution. It operates through a generate-then-verify loop.

A. High-Level: Closed-Loop Video Language Planning

The system uses a VLM as a high-level arbiter to decompose tasks into sub-goals and monitor execution.

• Task Decomposition: The VLM proposes sub-goal candidates based on semantic reasoning.
• Video Rollout: A video generation model synthesizes multiple candidate videos for each sub-goal to simulate physical outcomes.
• Validation & Selection: The VLM evaluates rollouts based on four metrics:
  1. Target: Is the correct object being manipulated?
  2. Physics: Does the motion obey laws (gravity, rigidity)?
  3. Motion: Does the flow match the language command?
  4. Result: Does the final state align with the sub-goal?
• Planning Horizon: The system dynamically selects a planning horizon ($h$). It uses strategic mode ($h > 1$) for coupled tasks requiring strict ordering (e.g., assembly) and greedy mode ($h=1$) for independent or exploratory tasks.
• Verify and Recover: After execution, the VLM compares the real-world state transition against the target video. If a failure is detected (e.g., grasp slip), the system triggers an autonomous recovery routine, synthesizing a corrective action (e.g., a "poke") to restore the state.

B. Low-Level: Hybrid Flow Mechanism

To bridge the gap between generated videos and robot control, NovaPlan extracts trajectories using a hybrid switching mechanism between Object Flow and Hand Flow.

• Object Flow: Tracks 3D keypoints of the target object. It is used when the object is visible and stable.
• Hand Flow: Tracks human hand poses (using HaMeR) as a kinematic prior.
• Dynamic Switching: The system switches to Hand Flow if the object rotation exceeds a threshold ($\theta_{max}$) or if occlusion makes object tracking unreliable. This ensures stability even when the target object is hidden by the hand.
• Geometric Calibration: A critical component is the dual-anchor calibration routine:
  1. Scale Recovery: At contact onset, the system snaps the hand fingertips to the object's metric 3D point cloud to correct scale inaccuracies in the generated video.
  2. Drift Compensation: At release, it compensates for projective drift (scale changes as the hand moves relative to the camera) to ensure the trajectory remains metrically accurate.

C. Non-Prehensile Correction

For tasks where objects get stuck, NovaPlan can generate non-prehensile recovery actions (e.g., poking). The VLM annotates the specific contact point on the object in the video prompt, and the geometric grounding procedure enforces contact between the specified finger and the object surface, resolving distortions in the generated hand shape.

3. Key Contributions

1. Closed-Loop Video Language Planning: A novel architecture integrating VLM verification with video generation to enable zero-shot long-horizon planning and autonomous error recovery.
2. Hybrid Tracking Mechanism: A dynamic switching system between object flow and hand flow, allowing the robot to maintain stable control even under heavy occlusion or depth inaccuracy.
3. Geometric Calibration: A method to ground "generated" human hands into physically executable robot trajectories by resolving scale and warping inconsistencies via dual-anchor calibration.
4. Zero-Shot Performance: Demonstration of solving complex assembly and non-prehensile error recovery tasks without any prior demonstrations or training.

4. Experimental Results

The authors evaluated NovaPlan on three long-horizon tasks and the Functional Manipulation Benchmark (FMB).

• Long-Horizon Tasks:
  • Four-Layer Block Stacking: NovaPlan achieved a 70% success rate (7/10 trials) on stacking four blocks, significantly outperforming baselines like $\pi_0.5$ (which failed beyond 2 layers) and NovaFlow (which dropped to 30% on the 4th block due to object flow instability).
  • Color Sorting: Demonstrated robustness in sorting blocks into tight-fitting containers, though failures were attributed to depth estimation errors.
  • Hidden Object Search: Achieved 100% success in finding objects in drawers and placing them, outperforming baselines that struggled with drawer opening or object retrieval.
• Short-Horizon Tasks: On the NovaFlow benchmark (Block Insertion, Water Plant, Open Drawer), NovaPlan consistently outperformed the original NovaFlow, validating that using hand flow as a reference improves stability when object tracking is noisy.
• Functional Manipulation Benchmark (FMB): NovaPlan successfully completed complex, multi-stage assembly tasks requiring millimeter precision. It also demonstrated the ability to recover from failures using non-prehensile poking, a capability absent in standard grasping-based baselines.
• Baselines: Compared against $\pi_0.5$ (VLA), MOKA (VLM planner), and NovaFlow. NovaPlan was the only system to consistently handle long-horizon dependencies and recover from runtime errors without oracle task decomposition.

5. Significance

NovaPlan represents a significant step toward general-purpose robotic manipulation. By treating video generation not as a static trajectory source but as a dynamic query within a verify-and-recover loop, it overcomes the "embodiment gap" that plagues previous video-to-robot methods.

• Robustness: The hybrid hand-object tracking mechanism solves the critical issue of occlusion, a major failure point in purely object-centric approaches.
• Scalability: The zero-shot nature of the framework means it can be applied to new tasks and environments without retraining, relying instead on the reasoning capabilities of foundation models.
• Error Recovery: The ability to "improvise" corrective actions (like poking) via non-prehensile interactions allows robots to handle complex, real-world scenarios where simple grasping fails, moving beyond standard pick-and-place primitives.

In summary, NovaPlan demonstrates that grounding video models in a closed-loop framework with geometric calibration enables robots to solve intricate, long-horizon tasks with a level of dexterity and resilience previously unattainable in zero-shot settings.