🤖 machine learning

CLAMP: Contrastive Learning for 3D Multi-View Action-Conditioned Robotic Manipulation Pretraining

This contribution introduces CLAMP, a novel 3D pre-training framework that leverages contrastive learning on multi-view point clouds and robot actions to improve the sample efficiency and performance of robot manipulation policies in both simulated and real-world tasks.

Original authors: I-Chun Arthur Liu, Krzysztof Choromanski, Sandy Huang, Connor Schenck

Published 2026-05-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: I-Chun Arthur Liu, Krzysztof Choromanski, Sandy Huang, Connor Schenck

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to do household chores, such as placing a cup on a plate or opening a drawer. In the past, scientists tried to teach robots by showing them thousands of videos of humans performing these tasks. The robot attempted to imitate the movements but often struggled because it viewed the world in "2D" (like a flat photograph). It did not truly understand that a cup is a solid object with depth or that it needs to reach around something to grasp it.

This work introduces CLAMP, a new method for training robots that, in a sense, gives them a "3D brain" before they even begin learning specific household tasks.

Here is how it works, broken down into simple concepts:

1. The Problem: Flat vs. 3D Vision

Imagine a standard robot camera like a person looking at a painting. You can see the colors and shapes, but you cannot feel the depth. If a robot tries to pick up a screwdriver, it might miss because it does not fully perceive the 3D space around the object.

CLAMP's Solution: Instead of just looking at flat images, CLAMP creates a 3D point cloud. Imagine taking a million tiny dots to represent every surface the robot sees. This gives the robot a "solid" understanding of the world and allows it to know exactly where the edges and corners of objects lie in 3D space.

2. The "Gym" Training (Pre-training)

Before the robot attempts a specific task (like opening a drawer), it undergoes a massive "gym" session in a computer simulation. This is called pre-training.

The Workout: The robot watches millions of simulated robot movements.
The Lesson: It learns to connect three things:
1. What it sees (the 3D shape of objects).
2. What it does (the history of its arm movements).
3. What it is told (text descriptions like "place the can opener in the left compartment").
The Magic Trick (Contrastive Learning): Imagine a game of "Memory." The robot is shown an image of a scene and a list of actions. It must match the correct image to the correct action. If it matches an image of a screwdriver to the action "Grasp the screwdriver," it gets a point. If it matches the image of the screwdriver to the action "Grasp the scissors," it loses a point. Over millions of attempts, it learns the deep connection between seeing an object and knowing how to move to grasp it.

3. The "Wrist-Camera" Advantage

One of the major discoveries of this work is that robots must see things from their own hands, not just from a fixed camera mounted on the ceiling.

The Analogy: Imagine trying to thread a needle while someone else holds a flashlight on the wall. That is difficult. But if you hold the flashlight yourself (a "wrist view"), you can see exactly what you are doing.
CLAMP's Move: It simulates cameras attached to the robot's wrists. This helps the robot see objects clearly, even when its own arms block the view, which is crucial for high-precision tasks.

4. The "Head Start" (Fine-tuning)

After the robot completes its "gym" training, it is ready to learn a real task.

The Old Way: Normally, you have to start from scratch by showing the robot a few hundred examples of a new task, and it takes a long time to learn.
The CLAMP Way: Since the robot already understands 3D space and knows how actions relate to objects, it only needs to see a tiny number of examples (such as 4,500 demonstrations) to master a new task. It is like a student who has already learned to read and write; they only need to learn the specific vocabulary for a new subject, rather than relearning how to hold a pen.

5. The Results

The authors tested CLAMP on six different simulated tasks (such as placing a can opener into a compartment) and five real-world tasks (such as opening drawers or recycling cans).

The Outcome: CLAMP was significantly better and faster at learning than other top methods. In the real world, it opened drawers and placed cups on plates much more frequently than robots trained without this 3D pre-training.

Summary

CLAMP is like a robot bootcamp. Instead of just teaching a robot how to do a specific job, it teaches the robot to see the world in 3D and how its body moves within that space. Once the robot has this general "common sense" knowledge about 3D objects and movements, it can learn new, specific household tasks incredibly quickly and precisely.

Technical Summary: CLAMP

Problem Statement

While the use of pretrained 2D image representations for behavior cloning has become standard in robotic manipulation, these approaches often fail to capture the essential 3D spatial information required for precise interaction with objects and scenes. Existing robotics pretraining methods face challenges such as the limited availability of large-scale robotics datasets, uncertainties regarding optimal modalities (e.g., RGB vs. point clouds), and the complexity of high-dimensional state and action spaces. Furthermore, previous 3D pretraining efforts, such as 3D-MVP, often rely on image reconstruction via masked autoencoders without incorporating robotic action data, leaving a gap in learning the relationship between 3D perception and action patterns.

Methodology

The authors propose CLAMP (Contrastive Learning for 3D Multi-View Action-Conditioned Robotic Manipulation Pretraining), a framework designed to learn robust 3D image and action representations through contrastive learning on large volumes of simulated robot trajectories.

1. Data Representation and Rendering

Point Cloud Construction: The system constructs a scene point cloud from RGB-D images and camera extrinsics, omitting RGB colors to improve appearance generalization.
Multi-View Re-Rendering: Instead of standard orthogonal views, CLAMP re-renders the scene into four-channel images (depth, X-, Y-, Z-coordinates) from five virtual viewpoints: three fixed views (top-down, front-left, back-right) and two dynamic wrist views (attached to the gripper). The dynamic wrist views are specifically designed to reduce occlusions and provide clearer observations for high-precision tasks.
Text Input: Text inputs include high-level task descriptions, object names, normalized global positions, and integers for task progress. These are derived from simulator ground truths and used exclusively during pretraining to anchor the representations.

2. Model Architecture

CLAMP employs three encoders trained via contrastive learning:

Image Encoder: A Vision Transformer (ViT) utilizing STRING relative positional encoding. Unlike standard positional encodings, STRING uses 3D coordinates (centroid of visible points) derived from the point cloud to correlate tokens across different views. This enables the model to implicitly learn 3D structures and correlate tokens from different 2D views that correspond to similar 3D regions.
Action Encoder: A Transformer encoder that processes a history of previous action blocks.
Text Encoder: A CLIP-based text encoder that processes task and object descriptions.

3. Contrastive Pretraining Objective

The encoders are trained to minimize a SigLIP-like loss over three modality pairs: image-text, image-action, and text-action. The objective pulls together embeddings of matching triplets $(I, T, A)$ while pushing apart non-matching triplets. This fosters the learning of action patterns capable of inferring future states, as well as image representations capable of inferring action history, conditioned on spatial and temporal task descriptions.

4. Diffusion Policy Pretraining and Fine-tuning

Parallel Pretraining: A Diffusion Policy (visuomotor controller) is pretrained in parallel with the encoders on the same simulated dataset. This policy is initialized with the pretrained weights to improve sample efficiency in downstream tasks.
Fine-tuning: For downstream tasks, the policy is fine-tuned on a limited number of task-specific demonstrations. In this phase, the CLAMP encoders are frozen to extract image and action embeddings, which are concatenated with RGB features (from ResNet-50) and proprioceptive data. These are fed into a Transformer encoder-decoder to predict noise for the next 50 action steps.

Key Contributions

Novel 3D Pretraining Framework: Introduction of CLAMP, which leverages robotic actions and dynamic wrist views for 3D contrastive pretraining.
STRING for 3D Coordinates: The first application of STRING relative positional encoding using 3D coordinates derived directly from point clouds (rather than depth maps) to correlate tokens across multi-view observations.
Significance of Dynamic Views: Demonstration that dynamic wrist views are critical for high-precision manipulation tasks.
Joint Encoder and Policy Pretraining: Evidence that pretraining both the encoders and the diffusion policy yields significantly better results than pretraining the encoders alone.

Results

The authors evaluated CLAMP across six simulated tasks and five real-world tasks using the ALOHA robot platform.

Simulation: CLAMP significantly outperformed state-of-the-art baselines (including ALOHA Unleashed, ACT, and VLA backbones) on all six tasks. For example, on the "Can Opener in Caddy" task, CLAMP achieved a success rate of 94.0%, compared to 70.0% for Pretrained ALOHA Unleashed and 7.3% for the VLA backbone.
Real World: In real-world experiments, CLAMP outperformed the baselines on four out of five tasks, showing significant improvements in opening/closing drawers and placing cups on plates.
Ablation Studies:
- Removing virtual wrist views led to significant performance drops (e.g., -28.6% on "Mug on Plate").
- Replacing the ViT with a PointNet-like backbone (DP3) degraded performance.
- Removing the text encoder had minimal impact in simulation but a noticeably negative effect in real-world environments, suggesting that text anchoring is more valuable when dealing with visual noise.
- It was shown that pretraining the policy in parallel with the encoders is essential for high sample efficiency.

Significance

The work claims that CLAMP substantially improves learning efficiency and policy performance on unseen tasks by effectively leveraging 3D geometric information and robotic action patterns. By pretraining on large simulated datasets and initializing the policy with these learned representations, the framework reduces the sample complexity required for fine-tuning on limited real-world demonstrations. The work concludes that integrating 3D perception with action-conditioned contrastive learning represents a viable and superior approach for pretraining robotic manipulation compared to existing 2D or reconstruction-based methods.