CLAR: Learning 3D Representations for Robotic Manipulation by Fusing Masked Reconstruction with Multi-Level Contrastive Alignment

Technical Summary of CLAR: Learning 3D Representations for Robotic Manipulation

Problem Statement

Robotic manipulation requires robust representation modules that can effectively ground policies in the physical 3D world. Existing approaches face significant limitations:

1. 2D Foundation Models: While powerful in semantic understanding, 2D vision models lack inherent 3D spatial awareness. They struggle with scene geometry and suffer from multi-view ambiguity, where an object's relative position (e.g., "left" vs. "right") is inconsistent across different camera viewpoints. This forces policies to learn brittle, camera-centric behaviors that do not generalize to new setups.
2. Existing 3D Pre-training: Current 3D pre-training methods often exhibit a trade-off. Masked Autoencoding (MAE) approaches excel at capturing spatial-geometric features but lack semantic richness. Conversely, contrastive learning methods can distill semantics from 2D foundation models but often fail to capture the fine-grained local details necessary for manipulation. Furthermore, standard contrastive learning in robotics often relies on global feature alignment, which loses local details, and point cloud cropping introduces contextual mismatches between partial 3D views and full 2D images.

Methodology

The authors propose CLAR (Contrastive Learning and Alignment with Reconstruction), a novel 3D pre-training framework designed to synergize global understanding with fine-grained local alignment. The framework operates within a unified robot-centric coordinate system to eliminate view ambiguity.

1. Global Contrastive Learning and 3D Reconstruction:

• 3D Reconstruction (MAE): To enhance spatial perception, CLAR employs a point cloud Masked Autoencoder. It randomly masks a high ratio (60%-80%) of point patches and trains an encoder-decoder architecture to reconstruct the masked geometry. The loss is computed using the Chamfer Distance between the reconstructed and ground truth point patches.
• Global Semantic Alignment: To transfer semantic knowledge from powerful 2D foundation models (e.g., CLIP) without requiring massive 3D datasets, CLAR uses cross-modal contrastive learning. It aligns the 3D point cloud feature space with both image and text feature spaces from pre-trained encoders using a contrastive loss function.

2. Adaptive Fine-Grained Local Alignment:
To address the failure of global alignment in capturing manipulation-critical details and the "contextual mismatch" caused by cropping point clouds, CLAR introduces a novel Adaptive Local Alignment Mechanism:

• Deformable Attention: Instead of forcing a static global match, the mechanism uses deformable attention to adaptively match local 3D patches with corresponding 2D image regions.
• Process: The 3D point cloud patch centers serve as queries, while the full 2D image feature map serves as values. The model predicts sampling offsets and attention weights to sample features only from valid, corresponding regions in the 2D image. This allows the model to handle the variable spatial scales of 3D patches and resolve the mismatch between cropped point clouds and full RGB images.
• Loss: A local alignment loss is computed between the aligned 3D local features and the sampled 2D local features.

3. Training Objective:
The total loss is a weighted sum of the MAE reconstruction loss, global 3D-to-image/text contrastive losses, and the local alignment loss.

Key Contributions

1. CLAR Framework: A new representation learning framework tailored for robotic manipulation that integrates 3D reconstruction for spatial perception and global contrastive learning for semantic understanding.
2. Adaptive Local Alignment: A novel mechanism using deformable attention to preserve fine-grained details essential for manipulation. This overcomes the limitations of global alignment and resolves contextual mismatches introduced by standard point cloud cropping.
3. State-of-the-Art Performance: Extensive validation demonstrating that CLAR outperforms existing methods in visuomotor policy learning across simulation and real-world tasks.

Experimental Results

The authors evaluated CLAR on MetaWorld, RLBench, and real-world tasks using a Franka Emika robot arm.

• Simulation Benchmarks:
  • MetaWorld: CLAR achieved a mean success rate of 82.6%, outperforming the previous best 3D method (Lift3D at 76.8%) and all 2D-based baselines.
  • RLBench: CLAR achieved 82.0% mean success rate, surpassing Lift3D (77.0%) and other baselines.
• Real-World Experiments:
  • On five real-world tasks (e.g., pick-banana, open-drawer), CLAR achieved an 83.0% success rate, significantly outperforming Lift3D (61.0%) and 2D methods like VC-1. The authors attribute the failure of baselines to poor geometry perception (2D methods) or modality mismatches (Lift3D).
• Multi-Task & Perspective Variation:
  • In multi-task instruction-following experiments, CLAR achieved 73.3% success, demonstrating effective semantic transfer.
  • In perspective variation tests (training on one camera view, testing on another), CLAR showed high robustness (84% success vs. 27.3% for CLIP), confirming the benefit of a unified spatial coordinate system.
• Ablation Studies: Removing the MAE loss caused the largest performance drop (from 83% to 73%), highlighting the critical role of spatial awareness. However, the local alignment module provided a consistent 6% gain, proving its necessity for fine-grained manipulation.

Significance and Claims

The paper claims that CLAR establishes a new state-of-the-art for robotic manipulation by successfully bridging the gap between spatial-geometric and semantic representation learning. Its primary significance lies in:

• Synergy: Demonstrating that combining MAE (for geometry) and contrastive learning (for semantics) is crucial for robust manipulation.
• Local Precision: Proving that adaptive local alignment is necessary to overcome the limitations of global alignment in tasks requiring fine-grained control.
• Generalization: Showing that a unified robot-centric coordinate system allows policies to generalize across different camera setups and viewpoints, a key requirement for real-world deployment.

The authors acknowledge limitations, primarily the scarcity of large-scale 3D robotics datasets with precise camera and depth information, which restricts the pre-training scale compared to 2D foundation models. Additionally, current experiments are confined to tabletop manipulation, with performance in mobile navigation environments remaining unexplored.