💻 computer science

Generalizable Coarse-to-Fine Robot Manipulation via Language-Aligned 3D Keypoints

The paper proposes Coarse-to-fine Language-Aligned manipulation Policy (CLAP), a framework integrating task decomposition, VLM fine-tuning for 3D keypoint prediction, and 3D-aware representations to achieve superior generalization in robotic manipulation with significantly fewer training demonstrations than state-of-the-art methods.

Original authors: Jianshu Hu, Lidi Wang, Shujia Li, Yunpeng Jiang, Xiao Li, Paul Weng, Yutong Ban

Published 2026-02-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jianshu Hu, Lidi Wang, Shujia Li, Yunpeng Jiang, Xiao Li, Paul Weng, Yutong Ban

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to make a sandwich.

If you just tell the robot, "Make a sandwich," and show it one video of you doing it, the robot might get confused if the bread is on the left instead of the right, or if the knife is a different color. It's like trying to follow a map that only works for one specific day.

This paper introduces a new way to teach robots called CLAP (Coarse-to-fine Language-Aligned manipulation Policy). Think of CLAP as a smart project manager who helps a robot worker get the job done, even when things change.

Here is how it works, broken down into three simple steps:

1. The "Recipe" Breakdown (Task Decomposition)

Most robots try to memorize the whole movie of a task at once. CLAP is different. It acts like a chef breaking a complex recipe into small, simple steps.

Old Way: "Go to the fridge, get the milk, pour it, and put the cap back on." (If the fridge is in a different spot, the robot panics).
CLAP Way: It breaks the task into tiny language instructions:
1. "Find the fridge handle."
2. "Grab the handle."
3. "Pull the door."
4. "Find the milk carton."
5. "Grab the milk."

By breaking the big job into small "language steps," the robot can mix and match skills. If it knows how to "grab a handle," it can use that skill on a drawer, a door, or a cabinet, even if it's never seen that specific object before.

2. The "Spotlight" (Coarse-to-Fine)

Imagine you are looking for a tiny needle in a haystack. If you look at the whole haystack at once, it's hard to see. But if someone shines a spotlight on the exact spot where the needle is, it becomes easy.

The "Coarse" Step: The robot's "brain" (a super-smart AI model) first looks at the whole room and says, "Okay, the needle is somewhere in this pile of hay." It picks a general area.
The "Fine" Step: The robot then zooms in, like a camera lens focusing, only on that specific pile of hay. Now, it can see the needle clearly and grab it.

This saves the robot from trying to process the entire room at high detail, which makes it faster and more efficient.

3. The "Reasoning" Assistant (Language Alignment)

This is the magic trick. The paper uses a pre-trained AI (like a very smart chatbot that has read the whole internet) to help the robot.

The Problem: Usually, robots are bad at understanding why they are doing something. They just copy what they see.
The CLAP Solution: Before the robot moves, the "Assistant" (the AI) thinks out loud. It says: "The robot needs to open the drawer. First, it must find the handle. Then, it must pull."

The robot doesn't just guess; it follows a logical plan generated by the AI. If you change the task to "Open the top drawer" instead of the "bottom" one, the AI instantly understands the difference because it understands the words, not just the picture.

Why is this a big deal?

The researchers tested this on a benchmark called GemBench, which is like a "final exam" for robots to see if they can handle new situations.

The Result: CLAP was 12% better than the best existing robots at handling new objects and new tasks.
The Efficiency: Here is the kicker: CLAP learned all this with only 1/5th of the practice data the other robots needed.
- Analogy: If other robots needed to watch 100 videos to learn a trick, CLAP only needed to watch 20.
Real World: They even tested it on a real physical robot. With just 10 practice runs, the robot could handle new objects (like a different colored cup) and new tasks (like stacking blocks in a new order) that it had never seen before.

Summary

CLAP is like giving a robot a smart, talking project manager.

It breaks big jobs into small, understandable steps.
It uses a spotlight to focus on exactly what needs to be done.
It uses language to reason through new problems, so the robot doesn't have to re-learn everything from scratch every time the environment changes.

This means robots can finally move out of the lab and into our homes and factories, where things are messy, unpredictable, and always changing.

1. Problem Statement

Robot learning via imitation learning has shown promise in 3D manipulation but struggles with two critical generalization challenges:

Generalization to Environment Variations (G1): The ability to operate in new settings with different object appearances, backgrounds, or placements.
Skill Compositional Generalization (G2): The ability to tackle novel, long-horizon tasks by composing previously learned skills.

Existing coarse-to-fine policies (which predict a 3D keypoint to zoom in for fine-grained action) offer high sample efficiency and precision. However, they often fail to generalize because:

They rely on training from scratch, lacking the reasoning capabilities of pre-trained Vision-Language Models (VLMs).
They condition the entire trajectory on a single high-level task description, limiting compositional reasoning.
There is a domain shift between pre-trained VLMs (trained on internet images) and robotic 3D data, leading to poor 3D keypoint prediction.
They often lack explicit 3D spatial reasoning, relying heavily on 2D visual inputs.

2. Methodology: CLAP Framework

The authors propose CLAP (Coarse-to-fine Language-Aligned manipulation Policy), a hierarchical framework that integrates task decomposition, VLM fine-tuning, and 3D-aware representations.

A. High-Level: Coarse Task Planner

Instead of predicting a keypoint directly from a high-level task description, CLAP uses a pre-trained VLM (specifically Qwen 2.5 VL-3B) as a "Coarse Task Planner" with a novel inference pipeline:

Task Decomposition: The VLM decomposes a high-level instruction (e.g., "Open the drawer") into a sequence of step-wise language instructions (e.g., "Lower arm," "Grasp handle," "Pull back"). This promotes skill compositional generalization (G2).
Two-Round Inference Protocol:
- Round 1 (Text-only): The VLM generates the full language plan ( $L$ ) based on the task description.
- Round 2 (Visual + Text): The VLM takes the current observation and the generated plan to predict the current step instruction ( $\ell_{tk}$ ) and the corresponding 3D keypoint ( $p_{tk}$ ).
Chain-of-Thought (CoT) Reasoning: To bridge the gap between 2D VLMs and 3D robotics, the VLM is trained to perform a sequential reasoning process:
- Step 1: Localize task-related objects (predict 3D object positions).
- Step 2: Generate the specific step instruction.
- Step 3: Predict the 3D keypoint.
Auxiliary Training: The VLM is co-trained with an object position dataset (3D object detection) to enhance zero-shot generalization to novel objects and improve spatial understanding.
Short-Term Memory: For long-horizon tasks, the model receives the previous step instruction as input to maintain context and prevent repetition errors.

B. Low-Level: Fine-Grained Action Predictor

Once the coarse branch identifies the region of interest (ROI) via the 3D keypoint, the fine-grained branch predicts the final action.

Input: The specific step instruction ( $\ell_{tk}$ ) and the cropped, refined multi-view RGB-D images.
3D-Aware Representation:
- Visual-Language Alignment: Uses a pre-trained visual-language encoder (SigLIP) to align the step instruction with RGB features.
- Geometric Structure: Uses a dedicated encoder (DINOv2) for depth images to capture geometric edges/contours.
- Spatial Embedding: Generates 3D position embeddings from pixel-wise 3D coordinates to explicitly inject spatial awareness.
Fusion: All embeddings are fused via a Multi-View Transformer to predict the final action. This architecture ensures the model leverages specialized pre-trained models for both language and 3D geometry, overcoming the domain shift issues of standard VLMs.

3. Key Contributions

Novel Architecture: Introduced a coarse-to-fine policy where the high-level branch acts as a Coarse Task Planner that decomposes tasks into step-wise language instructions, enabling compositional generalization.
VLM Adaptation Pipeline: Designed a specific fine-tuning strategy for pre-trained VLMs involving Chain-of-Thought reasoning (object localization $\to$ instruction generation $\to$ keypoint prediction) and auxiliary object detection tasks to align VLMs with 3D manipulation.
3D-Aware Representation: Developed a fine-grained action predictor that fuses language-aligned visual features with explicit 3D position embeddings, significantly improving precision and sample efficiency.
State-of-the-Art Performance: Demonstrated superior generalization in both simulation and real-world settings with drastically fewer training samples.

4. Experimental Results

Simulation (GemBench)

Evaluated on GemBench, a benchmark designed to test generalization across four levels:

L1: Novel placements.
L2: Novel rigid objects (shape/color).
L3: Novel articulated objects.
L4: Novel long-horizon tasks.
Performance: CLAP achieved an average success rate of 62.0%, outperforming the previous SOTA (BridgeVLA) by 12%.
Sample Efficiency: CLAP achieved this with only 20 training trajectories per task (1/5th of the 100 trajectories used by baselines).
Long-Horizon Tasks: CLAP showed a massive gain on Level 4 tasks (31.4% vs. near 0% for many baselines), proving the effectiveness of task decomposition.

Real-World Experiments

Setup: Trained on a real robot (Intel RealSense camera) with only 10 demonstrations per task.
Generalization: Tested on visual perturbations (lighting, background, distractors) and task variations (object substitution, skill composition).
Results: CLAP achieved an average success rate of 77.3%, compared to 22.5% for the baseline RVT2. Notably, CLAP succeeded in tasks where RVT2 failed completely (e.g., 0% success for RVT2 on "put shape in cup" vs. 80% for CLAP).

5. Significance and Impact

Bridging the Gap: CLAP effectively bridges the gap between powerful, general-purpose VLMs and the specific, data-scarce requirements of robotic manipulation.
Data Efficiency: By leveraging pre-trained knowledge and decomposing tasks, CLAP reduces the data collection burden by a factor of 5–10 compared to existing methods, making real-world deployment more feasible.
Compositional Reasoning: The shift from monolithic task descriptions to step-wise language instructions allows robots to "reason" through complex, multi-step tasks, a crucial step toward truly general-purpose robots.
3D Alignment: The method provides a robust blueprint for adapting 2D-centric foundation models to 3D spatial reasoning tasks without requiring massive amounts of 3D-specific training data.

In conclusion, CLAP represents a significant advancement in robotic learning by combining hierarchical planning, language-guided decomposition, and 3D-aware representation to achieve robust generalization with minimal data.