💻 computer science

Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View 3D-Grounded VLM Reasoning

This paper presents a zero-shot framework for long-horizon dexterous manipulation that leverages a vision-language model to ground language instructions into executable 3D task plans by fusing multi-view 2D keypoints into 3D space, enabling robust pick-and-place and tool-use execution on unseen objects without end-to-end training.

Original authors: Jisoo Kim, Sangwon Baik, Taeksoo Kim, Sungjoo Kim, Junyoung Lee, Mingi Choi, Hanbyul Joo

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jisoo Kim, Sangwon Baik, Taeksoo Kim, Sungjoo Kim, Junyoung Lee, Mingi Choi, Hanbyul Joo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot arm with a human-like hand that needs to clean up a messy room, but it has never seen this specific room before, and no one has taught it how to do this specific job. Most robots would need hours of training or thousands of examples to learn this. This paper presents a system that can do it zero-shot—meaning it figures it out on the fly, just by looking and listening, without any prior practice.

Here is how the system works, broken down into simple concepts:

1. The "Team of Photographers" vs. The "Single Snapshot"

Most robots look at a scene through one camera, like taking a single photo. If you try to guess where a cup is just from one photo, you might get the left-right position right, but you won't know exactly how far away it is. It's like trying to catch a ball in the dark with only one eye open; you might miss the depth.

This paper's system uses multiple cameras (like a team of photographers standing in different corners of the room).

The Problem: Each camera sees the object differently. One might see the handle of a spoon; another might see the bowl. One might be blocked by a book; another sees the whole thing.
The Solution: The system uses a "smart brain" (a Vision-Language Model) to ask every camera, "Where is the spoon?" and "Where should I grab it?"
The Magic Trick: It combines these different answers using two methods:
1. Triangulation: Like how your two eyes work together to judge distance, it mathematically calculates the exact 3D spot where all the camera views agree.
2. Ray Voting: If the cameras disagree (maybe one is blocked), the system shoots a "laser beam" from the main camera's view and asks the other cameras, "Did you see the object at this specific depth?" It picks the answer that gets the most votes. This ensures the robot doesn't grab thin air or miss the object because of a shadow.

2. The "Instruction Manual" vs. The "Muscle Memory"

Once the robot knows where the object is in 3D space, it needs to know how to move.

For Picking Things Up: The system breaks the big task ("Clean the room") into tiny, simple steps: "Grab the cup," "Move to the trash," "Drop it." It treats these like Lego blocks.
For Using Tools: This is the clever part. If the robot needs to use a broom or a kettle, it doesn't need to learn how to sweep from scratch. It has a "Bag of Atomic Actions" in its memory. Think of this as a library of pre-recorded dance moves for tools.
- If the instruction is "Sweep the floor," the robot finds the "Sweeping" dance move in its library.
- It then aligns that dance move to the current room. If the broom is on the left and the trash is on the right, it stretches and rotates the pre-recorded "sweeping" motion to fit that specific geometry.

3. The "Safety Check" and "Do-Over"

Robots often fail because they try to grab something and hit a wall, or they try to put a pot on a stove that is too high.

Affordance Regions: Instead of just grabbing a single point, the system figures out the "safe zone" on an object. For a handle, it knows the whole handle is a good place to grab, not just one pixel.
Collision Avoidance: Before moving, it simulates the path to make sure the robot hand won't crash into the table or the wall.
Closed-Loop Verification: This is the robot's "reality check." If the robot tries to pick up a cup and the camera sees it didn't move, the system doesn't just keep trying the same failed move. It stops, re-evaluates the scene, and re-plans. It's like a human saying, "Oops, I missed, let me try a different angle," rather than blindly repeating the mistake.

The Results

The authors tested this on a real robot with a dexterous hand in a real room.

Better Accuracy: It was much better at finding the exact 3D location of objects than robots that only look at one camera.
Zero-Shot Success: While other advanced robots (trained on 30 specific examples) failed completely at new tasks, this system succeeded at tasks like "throw away trash," "place a pot on a stove," and "sweep with a broom" without ever being trained on those specific tasks before.
Long-Horizon Tasks: It could string together multiple steps (like organizing a whole table) and recover if it made a mistake in the middle.

In Summary

This paper describes a robot that acts like a smart, adaptable human. Instead of memorizing every possible way to move, it uses a smart brain to understand language and 3D space from multiple angles, pulls pre-made "tool dances" from a library, and constantly checks its own work to fix mistakes on the fly. It proves you don't need to train a robot for every single job if you give it the right tools to reason about the world.

Technical Summary: Zero-Shot Long-Horizon Dexterous Manipulation via Multi-View 3D-Grounded VLM Reasoning

Problem Statement

The paper addresses the challenge of enabling general-purpose robotic systems to perform long-horizon dexterous manipulation based on high-level language instructions. While recent Vision-Language-Action (VLA) models have shown progress, they typically require extensive task-specific data collection, fine-tuning, or human demonstration retargeting, which limits scalability across diverse objects, tools, and spatial configurations. Furthermore, existing zero-shot approaches often rely on 2D intermediate representations (keypoints, visual marks) that are insufficient for the inherently 3D nature of manipulation tasks, such as determining contact points, placement targets, and tool trajectories. Small 3D grounding errors in dexterous hands can lead to unstable grasps, collisions, or incorrect tool usage. The authors argue for a modular approach that decouples semantic reasoning from physical execution but highlight the critical need to ground 2D VLM predictions into reliable 3D quantities to bridge the gap between language and physical action.

Methodology

The proposed framework is a zero-shot system that converts calibrated multi-view RGB images and language instructions into executable arm-hand plans. The pipeline consists of four primary stages:

1. Reference-Frame Semantic Grounding

A Vision-Language Model (VLM) $\Phi$ processes the multi-view images and instruction to:

Select a reference view $r$ .
Infer the manipulation mode $z$ (either pick or tool-use).
Predict a mode-specific grounding tuple $g_z$ $g_{z}$ .
- For pick-and-place: Identifies the target object and its 2D destination pixel.
- For tool-use: Identifies the tool object, the skill category (e.g., pouring, sweeping), the target object, and relevant 2D keypoints.
Generate a sequence of primitives $Q_r$ (e.g., grasp, apply action, release) with associated 2D keypoints and semantic descriptions on the reference view.

2. Multi-View Fusion-Based 3D Lifting

To convert 2D keypoints into robust 3D coordinates, the system employs a dual-strategy fusion approach to handle occlusion and depth ambiguity:

Cross-View Triangulation: The VLM is queried across all views to obtain 2D groundings for the same semantic concept. RANSAC-style triangulation is performed on view pairs, scoring candidates based on the number of views where the reprojection error is below a threshold.
Reference-View Ray Voting: To anchor estimates more firmly, the system samples depth candidates along the camera ray of the reference view. These candidates are projected into non-reference views with visual markers, and the VLM selects the indices that best match the semantic description. The candidate with the highest consensus vote is selected.
Dynamic Selection: The final 3D keypoint is chosen based on geometric consensus. If triangulation yields a high-confidence inlier, it is used; otherwise, the system falls back to the robust voting estimate.

3. Object-Centric Atomic Action Alignment (Tool-Use)

For tool-use tasks, the system utilizes a Bag of Atomic Actions, a pre-constructed library of object-centric 6D tool trajectories indexed by skill category.

Upon identifying the skill category, the corresponding atomic action (trajectory $T$ ) is retrieved.
The system aligns the stored trajectory to the current scene by computing a rigid transformation that maps the action's stored start/end anchors to the lifted 3D keypoints (apply-action and terminal keypoints) derived from the VLM.
This results in a scene-aligned 6D trajectory $\hat{T}$ for the tool.

4. Dexterous Affordance-Guided Grasp and Motion Generation

Affordance Region Estimation: The system queries the VLM across views to predict 2D graspable bounding boxes (affordances) around the grasp keypoint. These are lifted to 3D by projecting mesh vertices of the object and filtering them based on multi-view inclusion scores.
Grasp Generation: Candidate grasps are generated within the 3D affordance region using cylindrical templates (for handles) or optimization-based generators.
Collision-Aware Refinement: Grounding points for placement or tool poses are refined by searching a local vertical grid for the nearest collision-free position.
Execution: An off-the-shelf arm-hand motion generator resolves kinematic and collision constraints to produce a feasible execution trajectory.
Closed-Loop Verification: For long-horizon tasks, the system performs status verification. If a step fails (e.g., grasp failure), the VLM detects the state and triggers a replan or re-grounding of the specific subtask.

Key Contributions

Unified Zero-Shot Framework: A system that couples VLM-based task planning with multi-view 3D grounding and a library of reusable atomic primitives, enabling zero-shot execution on unseen objects and tools without task-specific training.
Multi-View VLM Grounding: A novel approach that fuses view-dependent 2D predictions into reliable 3D quantities by combining cross-view triangulation with reference-view ray voting, significantly improving robustness under occlusion and partial visibility.
Dexterous Execution Extension: The extension of multi-view grounding to dexterous hands by estimating functional 3D affordance regions and filtering candidate grasps based on trajectory-level inverse kinematics and collision feasibility.
Long-Horizon Capability: Demonstration of robust zero-shot generalization in complex, multi-step tasks (e.g., cooking, organizing) through closed-loop verification and dynamic replanning.

Experimental Results

The system was evaluated on a real-world tabletop setup with an xArm and Inspire dexterous hand, using multiple calibrated RGB cameras.

Success Rates: The method outperformed a single-view RGB-D grounding baseline and two fine-tuned VLA baselines (which required 30 demonstrations per task).
- In "Throw Away Trash" and "Place Pot on Stove" tasks, the proposed method achieved 5/5 and 4/5 success rates, respectively, compared to lower or zero success for baselines.
- In "Cluttered Precise Pick-and-Place," the method achieved 4/5 success versus 2/5 for the RGB-D baseline.
- Fine-tuned VLA models (GR00T, Being-HO) failed completely (0/5) on zero-shot tool-use tasks, while the proposed system succeeded in 8/10 trials for "Broom Clean."
3D Grounding Accuracy: Multi-view fusion reduced grasp localization error ( $L_{grasp}$ ) from 16.43 cm (RGB-D) to ~4.6 cm. Apply-action error ( $L_{apply}$ ) was also significantly reduced.
Collision Robustness: The collision-aware refinement step minimized penetration depth errors.
Long-Horizon Tasks: The system successfully completed multi-step tasks (e.g., organizing 3–4 objects, cooking) with closed-loop retries recovering from intermediate failures (e.g., joint limits, collisions).

Significance and Claims

The paper claims that a modular design, where VLMs handle semantic reasoning and primitive controllers handle execution, is a more efficient route to general-purpose manipulation than end-to-end policy learning. The authors emphasize that 3D grounding is the critical missing link for dexterous manipulation, which single-view reasoning cannot satisfy.

The significance of the work lies in demonstrating that:

Modern VLMs can provide accurate task decompositions and 2D groundings zero-shot.
By fusing these 2D predictions across multiple views, the system can recover reliable 3D geometry necessary for complex manipulation.
This approach enables robust execution on unseen objects and in novel environments without the need for extensive data collection or fine-tuning, addressing the scalability limitations of current VLA models.

The authors acknowledge limitations, noting that performance is bounded by the underlying 2D VLM's reasoning reliability and that physical execution relies on off-the-shelf planners which can introduce latency or singularities. They do not claim to solve in-hand manipulation (e.g., rotating objects within the hand) but focus on object-centric manipulation and tool use.