Multi-Camera View Scaling for Data-Efficient Robot Imitation Learning

1. Problem Statement

Imitation learning (IL) for robotic manipulation relies heavily on the diversity of expert demonstrations to ensure generalization. However, collecting diverse demonstrations across varied environments is costly, time-consuming, and labor-intensive.

• The Bottleneck: Existing datasets often utilize only a single camera view (plus an in-hand camera), wasting the inherent rich visual information contained within a single expert trajectory.
• The Challenge: How to maximize the diversity of training data and improve policy generalization without requiring a proportional increase in human data collection efforts or complex environment reconfiguration.

2. Methodology

The authors propose a framework called Camera View Scaling, which leverages synchronized multi-camera setups during data collection to generate "pseudo-demonstrations" from a single expert trajectory.

A. Core Concept: Pseudo-Demonstrations

Instead of collecting new trajectories, the system installs multiple cameras at different perspectives around the workspace. During data collection, a single expert demonstration is recorded simultaneously by $V$ cameras.

• Expansion: One expert trajectory $\tau_i$ is expanded into $V$ pseudo-demonstrations $\{\tau^v_i\}_{v=1}^V$.
• Training Strategy: During training, the policy is randomly sampled from these different camera views. This artificially increases the diversity of the training distribution, teaching the model to be invariant to viewpoint changes.

B. Action Space Transformations

A critical component of the methodology is how robot actions are represented relative to the camera view. The paper analyzes three action spaces:

1. Robot Base Space: Actions are defined in a fixed robot base coordinate system. The action remains identical ($a^v = a$) regardless of the camera view.
2. End-Effector (EEF) Space: Actions are defined relative to the end-effector's pose. Similar to Base space, the action is view-invariant.
3. Camera Space (Proposed): Actions are transformed into the specific coordinate system of the observing camera ($v$).
   • Mechanism: The translation and rotation vectors are rotated by the camera's extrinsic parameters ($R_{ext}^v$).
   • Benefit: This creates distinct action targets for the same physical movement depending on the camera view, significantly increasing the diversity of the training data (both visual and action inputs).

C. Multi-View Action Aggregation (Inference)

While the primary goal is to train a single-view policy (to avoid complex sensor setups during deployment), the framework offers a "free-lunch" capability for deployment:

• Mechanism: If multiple cameras are available during inference, the model can predict actions independently for each view. These predictions are aggregated using a probabilistic composition method derived from diffusion models.
• Formula: The aggregated action distribution is proportional to the product of individual view distributions: $p(a_{aggr}|\{o\}) \propto \prod p(a|o^v)$.
• Implementation: This is achieved by modifying the denoising step in the diffusion sampling process to sum the gradients (noise predictions) from all views.

3. Key Contributions

1. View Scaling Strategy: Identification that single-view datasets waste inherent scene diversity. The paper proposes a "free-lunch" strategy to scale camera views during collection, generating pseudo-demonstrations with minimal extra human effort.
2. Action Space Analysis: A deep analysis of how action spaces interact with view scaling. The authors demonstrate that Camera Space actions yield the highest diversity and performance gains, though Base Space is also robust.
3. Aggregation Algorithm: Development of a multiview action aggregation method that allows a single-view trained policy to leverage multiple cameras during inference without architectural changes.
4. Empirical Validation: Extensive evaluation in both simulation (RoboMimic) and real-world settings (water pouring task).

4. Experimental Results

The authors evaluated their approach against single-view baselines using Diffusion Policies.

• Performance Gains (Q1 & Q2):
  • Multi-camera training significantly outperformed single-view training across all tasks (Square, Can, Lift).
  • In some settings, success rates doubled compared to single-view baselines.
  • Action Space: The Camera Space action representation generally outperformed Base and EEF spaces, particularly when using 5 camera views. EEF space performed poorly due to the moving origin of the coordinate system.
• View Selection (Q3):
  • Views must be somewhat consistent with the inference view. Training with views that are too dissimilar (e.g., top/side views when inference is front-view) provided diminishing returns due to distribution shifts.
  • Moderate angular differences (e.g., $\pm 15^\circ$ to $\pm 30^\circ$) yielded the best results.
• Generalization (Q4):
  • Policies trained with multi-view pseudo-demonstrations generalized robustly to unseen camera angles during inference, whereas single-view trained policies failed completely when the camera angle changed.
• Real-World Validation (Q6):
  • In a real-world water-pouring task with a FANUC robot, training with two views improved performance over single-view training.
  • Inference Aggregation: Using the multiview aggregation method during inference further boosted performance, validating the algorithm's practical utility.

5. Significance and Impact

• Data Efficiency: The framework offers a scalable solution to the data scarcity problem in imitation learning. It allows researchers to achieve the generalization benefits of diverse environments without the logistical burden of physically changing scenes or collecting thousands of new trajectories.
• Hardware Agnostic: The method requires minimal additional hardware (just extra cameras) and integrates seamlessly with existing algorithms (e.g., Diffusion Policies).
• Deployment Flexibility: It bridges the gap between training and deployment. A policy can be trained with rich multi-view data for robustness but deployed on a single camera, or optionally leverage multiple cameras at inference for higher precision.
• Paradigm Shift: The paper suggests that scaling camera views is a more practical and effective lever for improving IL generalization than simply scaling the number of demonstrations.