Planning from Observation and Interaction

1. Problem Statement

The paper addresses the challenge of Learning from Observation (LfO) in real-world robotics, specifically under the most restrictive setting: Inverse Reinforcement Learning from Observation (IRLfO).

• Constraints: The agent has access only to visual observations of a task demonstration (e.g., video or images). It does not have access to:
  • The demonstrator's actions (no action labels).
  • Hand-designed reward functions.
  • Prior knowledge, pre-training, or domain-specific state estimators.
• Goal: The agent must learn to perform a manipulation task (like pushing a block or picking and placing an object) by interacting with the real world, inferring the reward, and planning its own actions based solely on observing the expert's trajectory.
• Current Limitations: Existing IRL methods often require massive amounts of online interaction (sample inefficiency), rely on state-based dynamics rather than visual inputs, or fail to scale to complex real-world manipulation tasks without extensive pre-training.

2. Methodology: MPAIL2

The authors propose MPAIL2 (Model Predictive Adversarial Imitation Learning 2), an off-policy, planning-based algorithm that combines world modeling with adversarial imitation learning.

Core Components

The system learns a latent world model consisting of four main components, updated via off-policy learning:

1. Encoder ($e_\omega$): Maps raw observations ($o_t$) to a latent state ($z_t$).
2. Dynamics Model ($f_\psi$): Predicts the next latent state ($\hat{z}_{t+1}$) given the current latent state and action ($z_t, a_t$). This is a deterministic model trained on transition sequences.
3. Inferred Reward ($r_\theta$): A discriminator trained adversarially to distinguish between expert transitions (from the demonstration dataset $D$) and learner transitions (from the replay buffer $B$). It uses a Gradient Penalty (Wasserstein GAN formulation) for stability.
4. Value Function ($Q_\zeta$) & Policy ($\pi_\phi$):
   • The Value function estimates the return of state-action pairs in an entropy-regularized, off-policy manner (similar to Soft Actor-Critic).
   • The Policy is a multi-step policy that outputs a sequence of actions (a plan) rather than a single action. This reduces online computation costs.

The Planning Loop (MPPI)

Instead of executing the policy directly, MPAIL2 uses Model Predictive Path Integral (MPPI) control for online decision-making:

1. Observation: The agent observes the current state and encodes it to $z_t$.
2. Sampling: It samples $N$ candidate action sequences (plans). A small fraction ($M/N$) is sampled from the current policy $\pi_\phi$, while the rest are random.
3. Rollout: The dynamics model $f_\psi$ predicts the trajectory of latent states for each sampled plan.
4. Evaluation: The inferred reward $r_\theta$ and value function $Q_\zeta$ are used to calculate the total return for each predicted trajectory.
5. Optimization: The algorithm re-weights the sampled plans based on their returns (using an exponential weighting scheme) to update the distribution of the next plan (mean $\mu$ and covariance $\Sigma$).
6. Execution: The first action of the optimized plan is executed. The resulting observation is stored in the replay buffer for offline model updates.

Key Innovations over MPAIL (Previous Work)

• Off-Policy Learning: Unlike the original MPAIL which was on-policy, MPAIL2 learns from a replay buffer, significantly improving sample efficiency.
• World Modeling: It learns a latent dynamics model from raw images, allowing it to predict future states and plan ahead.
• Multi-Step Policy: The policy network acts as a "seed" for the planner, outputting full trajectories to reduce the computational burden of online planning.

3. Key Contributions

1. First Real-World IRLfO from Scratch: This is the first work to demonstrate IRL from observation purely in the real world without any prior modeling assumptions, pre-training, or access to action labels.
2. Superior Sample Efficiency: MPAIL2 achieves consistent success in real-world manipulation tasks (Block Push, Pick-and-Place) in under 40 minutes of interaction. In contrast, baseline methods (RL with demonstrations, Behavior Cloning) failed to achieve success even after over an hour of training.
3. Online Transfer Learning: The learned world model representation enables online transfer learning. The agent can adapt to a new task (e.g., pushing a block in the opposite direction) purely in the real world by leveraging the previously learned dynamics and reward structures, without retraining from scratch.
4. Robustness to Observation Noise: By planning in latent space and using iterative optimization (MPPI), the method is robust to the noise and delays inherent in real-world robotic systems.

4. Experimental Results

Experiments were conducted on a Franka Emika arm (Push task) and a Kinova Gen3 arm (Pick-and-Place task) in both simulation and the real world.

• Baselines Compared:
  • IRL Ablations: Variants of MPAIL2 without planning (MAIRL), without model-based methods (DAC), and without off-policy replay (AIRL).
  • RL with Demonstrations (RLPD): Uses dense rewards and action labels.
  • Behavior Cloning (Diffusion Policy): Trained offline on demonstration data.
• Performance Highlights:
  • Real-World Success: MPAIL2 achieved 62%–82% success rates on real-world tasks.
  • Failure of Baselines:
    • RLPD: Achieved 0% success in real-world tasks despite having access to dense rewards and action labels, highlighting the difficulty of real-world RL without a world model.
    • Behavior Cloning (Diffusion): Achieved high success in simulation but dropped significantly in the real world (e.g., 12% on Pick-and-Place) due to a lack of generalization and inability to recover from errors.
    • Ablations: Removing the planner or the dynamics model caused performance to collapse, proving the necessity of both components for real-world stability.
• Transfer Learning: When transferred to a new task (reversed direction), MPAIL2 adapted significantly faster than training from scratch, whereas BC and non-planning methods failed to transfer effectively.

5. Significance and Conclusion

This work represents a significant step toward autonomous robot learning in unstructured environments.

• Practicality: It demonstrates that robots can learn complex manipulation tasks from scratch using only visual observation, removing the need for expensive teleoperation data or hand-crafted reward functions.
• Efficiency: The combination of world modeling and planning drastically reduces the number of physical interactions required, making real-world learning feasible within human timescales (minutes rather than days).
• Generalization: The ability to transfer learned representations to new tasks suggests a path toward lifelong learning for robots, where they can continuously adapt to new objectives by observing new demonstrations and interacting with the world.

In summary, MPAIL2 bridges the gap between theoretical IRL and practical robotic deployment by introducing a robust, sample-efficient, planning-based framework that learns directly from visual observations in the real world.