Technical Summary: CLAM - Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrations

Problem Statement

Learning robot control policies from demonstrations typically relies on expert data where actions are explicitly labeled (e.g., via teleoperation). Collecting such data is expensive, time-consuming, and difficult to scale, particularly for fine-tuning models to diverse real-world environments. While human video demonstrations are abundant and easier to collect, they lack the ground-truth robot control signals required for standard imitation learning (IL). Furthermore, converting human videos to robot-compatible observations does not inherently provide the missing action labels.

The paper addresses a practical setting where:

1. Unlabeled Expert Data: A large corpus of robot video demonstrations exists (either converted from human video or existing logs) but lacks action labels.
2. Task-Agnostic Play Data: A small dataset of robot transitions with action labels is available, but these are "play" data (undirected, non-expert) rather than task-specific expert demonstrations.
3. Goal: Learn a high-performance policy for a specific task using the unlabeled expert videos and the small amount of task-agnostic play data, without ever collecting new action-labeled expert demonstrations.

Methodology: Continuous Latent Action Models (CLAM)

CLAM is a two-stage framework designed to infer continuous latent actions from observation sequences and ground them into executable motor commands.

1. Latent Action Model (LAM) Training (Stage 1)

The core of CLAM is a self-supervised latent action model that learns to infer actions between consecutive observations without ground-truth labels.

• Components:
  • Latent Inverse Dynamics Model (IDM): $f_\phi(z_t | o_{t-H}, \dots, o_t, o_{t+1})$ infers a continuous latent action $z_t$ between observations.
  • Latent Forward Dynamics Model (FDM): $g_\psi(o_{t+1} | o_{t-H}, \dots, o_t, z_t)$ predicts the next observation conditioned on the current history and the inferred latent action.
• Self-Supervised Objective: The model is trained to minimize the reconstruction error of the future observation ($L_{recon} = MSE(\hat{o}_{t+1}, o_{t+1})$). This loop forces the latent action $z_t$ to encode the necessary state-dependent transition information.
• Continuous Latent Space: Unlike prior works that use Vector Quantization (VQ) to discretize actions, CLAM utilizes a continuous latent action space. This is critical for high-dimensional, continuous control tasks where discrete approximations fail.
• Joint Training with Action Decoder: To ensure the latent actions are executable, CLAM jointly trains a latent action decoder $p_\omega(a_t | z_t)$ using the small task-agnostic play dataset ($D_{labeled}$).
  • The decoder maps latent actions $z_t$ to real motor commands $a_t$.
  • The training objective combines reconstruction loss and action decoder loss: $L_{CLAM} = L_{recon} + \beta L_{action-decoder}$.
  • Crucial Insight: Joint training regularizes the latent space, ensuring it remains decodable. The decoder is not conditioned on observations, forcing the latent action to be a low-level action code.

2. Latent Action Policy Training (Stage 2)

Once the LAM is pretrained:

• Relabeling: The trained IDM annotates the unlabeled expert demonstrations ($D_{unlabeled-expert}$) with pseudo-latent actions, creating a dataset of $(o_t, z_t)$ pairs.
• Policy Learning: A latent action policy $\pi_\theta(z_t | o_t)$ is trained via standard imitation learning (behavior cloning) on this relabeled dataset.
• Inference: During execution, the policy predicts a latent action $z_t$ given an observation, and the pre-trained action decoder $p_\omega$ converts $z_t$ into a physical motor command.

Key Contributions

1. Continuous Latent Relabeling: The authors demonstrate that latent action relabeling becomes practical for continuous robot control only when using continuous latent spaces combined with joint grounding (training the decoder simultaneously with the LAM).
2. Efficient Data Utilization: The framework shows that task-agnostic play data is sufficient for grounding latent actions. This dramatically reduces the cost of data collection, as it eliminates the need for expensive, task-specific expert teleoperation for the action-labeling phase.
3. Real-World Deployment: CLAM is successfully deployed on a physical WidowX robot arm across four manipulation tasks, achieving performance competitive with policies trained on privileged expert action labels, despite never seeing such labels during training.

Experimental Results

The paper evaluates CLAM on DMControl (locomotion), MetaWorld (manipulation), and a real WidowX robot.

• Performance Gains:
  • On DMControl (state-based), CLAM improves average normalized returns by over 2× compared to the best baseline (VPT).
  • On MetaWorld (image-based), CLAM achieves 2–3× higher task success rates than baselines like LAPO, LAPA, and DynaMo.
  • CLAM approaches or even exceeds the performance of BC-Expert (behavior cloning trained on ground-truth expert actions), despite BC-Expert having access to privileged labels while CLAM does not.
• Ablation Studies:
  • Continuous vs. Discrete: Discrete latent actions (using VQ) perform poorly on continuous control tasks. Continuous latent actions alone provide a ~1.5× improvement.
  • Joint Training: Jointly training the action decoder with the LAM is essential. It boosts performance by roughly 3× when combined with continuous latent actions.
  • Data Scaling: Performance scales with the amount of unlabeled expert data ($|D_{unlabeled}|$). The method also scales with the amount of play data ($|D_{labeled}|$) used for grounding, outperforming standard BC which plateaus quickly with non-expert data.
• Real Robot Results: On the WidowX, CLAM solves manipulation tasks (e.g., assembly, bin picking) with success rates significantly higher than baselines, often matching the performance of the privileged BC-Expert baseline.

Significance and Claims

The paper claims that CLAM represents a scalable solution for learning robot policies from action-less data. Its primary significance lies in:

• Removing the Bottleneck of Expert Labeling: It demonstrates that effective robot policies can be learned without collecting expensive, task-specific, action-labeled expert demonstrations.
• Leveraging Cheap Data: By utilizing abundant, unlabeled robot videos and cheap, task-agnostic play data, CLAM lowers the barrier to entry for robot learning.
• Architectural Necessity: The work establishes that for high-dimensional continuous control, the combination of continuous latent spaces and joint training of the action decoder is not just beneficial but essential for success, distinguishing it from prior latent action models designed for discrete domains.

The authors note that while CLAM is effective in single-robot, single-task settings, future work is needed to address challenges in generalizing to "in-the-wild" internet videos (handling artifacts, embodiment gaps) and scaling to cross-embodiment learning.