Technical Summary: See2Act – Diffusion Models for Active Perception in Robot Imitation

Problem Statement

Current imitation learning (IL) methods for visuomotor control typically assume full observability and fixed viewpoints, conflating the abilities to "see" and "act." In realistic environments, objects are frequently occluded, requiring robots to actively search for task-relevant information before manipulating it. Existing approaches face a fundamental limitation: demonstrations entangle viewpoint selection and manipulation, yet standard IL treats them separately or assumes the required visual information is already visible.

• Fixed-view policies fail when task-relevant regions are occluded.
• Multi-view or passive-view selection methods (e.g., selecting the best view from a preset set) do not teach the policy how to move the camera to resolve occlusions dynamically.
• Data inefficiency: Simply augmenting demonstrations with additional viewpoints increases data requirements combinatorially without teaching the policy which views are useful for specific actions.

The core challenge is learning a policy that jointly determines where to see and how to act from limited demonstrations, particularly under severe occlusion.

Methodology: See2Act

The authors propose See2Act, a diffusion-based imitation learning framework that couples action denoising with viewpoint refinement within a single generative loop. The central insight is that the diffusion trajectory serves as a trajectory over both actions and viewpoints.

1. Training Strategy

The policy is trained entirely in simulation using a digital twin, leveraging offline manipulation datasets.

• Keyframe Actions: Demonstrations are decomposed into target keyframe actions ($a_0$), representing the end-effector poses for pick/place operations.
• Viewpoint Anchoring: Instead of a fixed world frame, actions are represented in the camera frame. For each diffusion timestep $t$, a camera pose $C_t$ is generated.
  • Anchors: The trajectory is anchored between a broad global view ($C_T$) and a target-centric view ($C_0$) derived directly from the keyframe action $a_0$.
  • Interpolation: Intermediate poses $C_t$ are generated via linear interpolation for position and Spherical Linear Interpolation (SLERP) for orientation between $C_T$ and $C_0$.
• Noising Process: The target action $a_0$ is transformed into the camera frame $C_t$ to create a viewpoint-dependent target $\hat{a}_0$. Gaussian noise is added to create a noisy action $\hat{a}_t$.
• Objective: A visual encoder and a noise prediction network are trained to predict the noise $\epsilon$ given the observation $O_t$ (rendered from $C_t$) and the noisy action $\hat{a}_t$. The model learns to denoise the action while implicitly learning the corresponding informative viewpoint.

2. Inference: Active Viewpoint Inference

At test time, the policy performs a coupled reverse diffusion process:

1. Initialization: Start with a global view ($C_T$) and a noisy action estimate.
2. Iterative Refinement: At each denoising step $t$:
   • Capture the current observation $O_t$.
   • Predict noise and update the action estimate in the camera frame.
   • Transform the refined action estimate back to the world frame.
   • Reposition Camera: Compute a new, more target-centric camera pose $C_{t-1}$ based on the updated action estimate (using the same interpolation logic as training).
   • Capture the new observation $O_{t-1}$.
3. Execution: This loop continues until $t=0$, resulting in a final action and a sequence of camera poses that actively resolve occlusions. The policy selects the rollout with the smallest variance in camera-pose changes to ensure stability.

Key Contributions

1. See2Act Framework: A viewpoint-agnostic policy that learns the coupling of viewpoint and action by leveraging digital twins and a novel training strategy where camera trajectories are supervised entirely by action labels.
2. Active Viewpoint Inference: A test-time mechanism that iteratively refines the robot's camera pose and predicted action simultaneously, enabling the robot to autonomously resolve occlusions without separate planning stages or rewards.
3. Benchmark Performance:
   • Ravens: Achieved 91% average success rate on four new occlusion-heavy tasks, significantly outperforming fixed-view, multi-view, and zoom-only baselines.
   • RLBench: Achieved 81.1% average success rate across nine tasks, outperforming state-of-the-art 3D and multi-view policies (e.g., RVT-2, PerAct) despite using only a single actively repositioned 2D view per step.
4. Zero-Shot Sim-to-Real Transfer: Demonstrated on a physical UR10 robot with a wrist-mounted depth camera. Using only 50 demonstrations collected in a digital twin, the policy achieved a 95% success rate on real-world pick-and-place tasks with significant occlusions, without any real-world fine-tuning.

Experimental Results

• Occlusion Resolution: On Ravens tasks (e.g., bin-picking, put-within-shelf), fixed-view baselines (Conv-MLP, Diff-MV) failed (0% success). Passive view selection (Diff-MV-Entropy) achieved up to 64% but struggled with complex occlusions. See2Act achieved up to 100% success, demonstrating that iterative active refinement is necessary to uncover hidden regions.
• Search Behavior: In the bin-search task, See2Act exhibited emergent search behavior, reorienting the camera to inspect alternative bins when the current view was empty, achieving 100% success where baselines scored 0%.
• Precision Manipulation: In real-world shelf-kitting (requiring 1–2mm tolerance), See2Act achieved 95% success compared to 25% for fixed-view methods. The ability to move the camera close to the target during denoising provided the geometric detail necessary for high-precision insertion.
• Robustness: The policy showed robustness to out-of-distribution initial camera poses, with performance dropping only marginally (8–10%) when starting from viewpoints outside the training distribution.

Significance and Claims

The paper claims that See2Act addresses a critical gap in imitation learning by treating manipulation as a coupled process of action refinement and visual evidence acquisition.

• Unified Learning: It demonstrates that "where to see" and "how to act" can be learned jointly from offline data without explicit viewpoint annotations or reinforcement learning rewards for perception.
• Efficiency: The approach achieves robust performance with limited data (50 demos) and transfers zero-shot to the real world, overcoming the sim-to-real gap by using depth observations and active viewpoint refinement.
• Limitations: The authors acknowledge that requiring a new observation at every denoising step introduces latency, making execution slower than fixed-view methods. Future work is suggested to extend the framework to dense trajectory prediction for dexterous and dynamic manipulation.

The work establishes that active perception, when embedded directly into the diffusion denoising loop, enables robots to resolve partial observability in a way that fixed-view, multi-view, and passive-selection methods cannot.