Dynamics-Aware Meta-Imitation for Generalization to Unseen Robotic Manipulation

Technical Summary: Dynamics-Aware Meta-Imitation for Generalization to Unseen Robotic Manipulation

Problem Statement
Imitation Learning (IL) enables robots to acquire complex manipulation skills from demonstrations but faces significant challenges regarding data scarcity and generalization to out-of-distribution (unseen) tasks. Existing methods often overfit to specific in-domain tasks or rely on dense demonstrations, leading to poor performance when encountering novel tasks or distribution shifts. While recent approaches like In-Context Imitation Learning (ICIL) and 3D diffusion policies show promise, they struggle to infer task dynamics from sparse data or rely heavily on visual similarity rather than semantic action understanding. Furthermore, directly fine-tuning high-capacity 3D diffusion policies on few-shot data often results in overfitting to support trajectories rather than learning transferable task logic.

Methodology: DAMI Framework
The authors propose Dynamics-Aware Meta-Imitation (DAMI), a framework designed to bridge the generalization gap by integrating meta-learning with an expressive 3D diffusion policy. DAMI conditions the policy on three inputs: the current observation (3D point cloud and robot state), a language instruction, and a complete reference demonstration. The core architecture consists of the following components:

1. Meta-Learning Objective: DAMI employs Model-Agnostic Meta-Learning (MAML) to learn a shared policy initialization. This initialization is optimized to be rapidly adapted to novel tasks using only a few demonstrations (few-shot fine-tuning), rather than learning each task from scratch.
2. Visual-Motor Trajectory (VMT) Module: To capture intrinsic task dynamics rather than static visual cues, VMT jointly encodes visual features and motor action sequences from the reference demonstration. It uses a Transformer encoder with positional embeddings to model spatio-temporal dependencies, creating a motion-centric context that represents "what changes under what action."
3. Unpaired Unified Task (U2T) Block: This module addresses the challenge of fusing unstructured multimodal inputs (text instructions, reference demonstrations, and current observations) that are not temporally aligned. U2T utilizes a Transformer-based fusion mechanism to align demonstration and text tokens with the observation-timestep condition specifically at the first downsampling block of the diffusion U-Net.
4. Task-Conditioned Feature Modulation (TCFM): To effectively inject the fused multimodal conditions into the policy, TCFM applies Feature-wise Linear Modulation (FiLM) to the low-level 3D features of the U-Net. This allows the network to modulate early geometric grounding layers based on the specific task context derived from the demonstration and text.

The training process involves an inner loop that adapts the policy to a support set of a specific task and an outer loop that updates the shared initialization based on the query set performance across a batch of tasks.

Key Contributions

• DAMI Framework: A novel meta-imitation framework that constructs a shared skill space for rapid few-shot adaptation and cross-task generalization.
• VMT Module: A mechanism to capture complex spatio-temporal dynamics in a compact latent space, enabling the policy to model task logic rather than memorizing static appearance.
• U2T and TCFM: A unified task encoder and feature modulation mechanism designed to fuse unstructured multimodal observations and modulate low-level 3D features effectively.
• Comprehensive Evaluation: Extensive experiments demonstrating that the approach outperforms state-of-the-art baselines in both direct inference on seen tasks and adaptation to unseen tasks.

Experimental Results
The authors evaluated DAMI on the Meta-World (ML10 and ML45 benchmarks), RLBench (FS25 setting), and real-world robotic manipulation tasks using a UR10e manipulator.

• Simulation (Meta-World): On base tasks, DAMI achieved an average success rate of 87.23% on ML10 and 79.05% on ML45, significantly outperforming baselines like DP3, Mamba, and FlowPolicy. Crucially, on novel (unseen) tasks, DAMI achieved 56.80% on ML10 and 37.53% on ML45, outperforming or tying the best baselines while maintaining superior base-task proficiency.
• Simulation (RLBench): On the FS25 novel task setting, DAMI achieved the highest average success rate of 10.80%.
• Real-World: In physical experiments involving seven tasks (five base, two novel), DAMI achieved an average success rate of 82.86%, compared to 55.71% for the DP3 baseline. Notably, DAMI demonstrated robust generalization to irregular object geometries (e.g., sweeping irregular cylinders) and unseen object types (e.g., Rubik's cubes), whereas baselines suffered from "Action Misinterpretation" (e.g., executing a pressing motion instead of a sweeping motion).
• Efficiency: The adaptation process for real-world tasks required less than two minutes (approx. 74.73 seconds for 500 steps), with only a marginal increase in computational latency compared to baselines due to the additional fusion modules.

Significance and Claims
The paper claims that DAMI successfully addresses the critical challenges of data scarcity and limited generalization in robotic imitation learning. By learning the underlying task logic from random complete reference demonstrations rather than memorizing static cues, the framework ensures effective generalization to unseen tasks. The authors assert that their method is most applicable when a complete reference demonstration, reliable 3D observations, and a small task-specific demonstration set are available. While the method requires task-specific fine-tuning and incurs slightly higher computational costs than some baselines, the substantial gains in task success rates and semantic consistency justify the overhead. The work highlights the potential of combining 3D diffusion policies with meta-learning and dynamics-aware representation to create more adaptable and robust general-purpose robotic agents.