ARGUS: Aligning Robot Scene Geometry Under Shifting Views with Large 3D Vision Models

Technical Summary of ARGUS: Aligning Robot Scene Geometry Under Shifting Views with Large 3D Vision Models

Problem Statement

Large-scale visuomotor policies have achieved significant success in robot manipulation; however, they often suffer from an entanglement between scene geometry and the specific camera viewpoint. Policies tend to learn the location of objects relative to the image frame rather than the task space. This limitation restricts their ability to learn from viewpoint-diverse datasets (e.g., DROID, BridgeV2) and hinders generalization to novel camera configurations not present in the training data. While training on large, diverse datasets is a common strategy to mitigate this, the paper argues that forcing models to disentangle task semantics from scene-induced variability directly is data-inefficient and costly. Existing approaches relying on explicit depth sensors face hardware scalability issues, while simulation-based data augmentation struggles with sim-to-real transfer and visual artifacts.

Methodology: ARGUS

The authors propose ARGUS (Aligning Robot Scene Geometry Under Shifting Views), an observation pre-processing pipeline designed to decouple viewpoint variation from task learning. Instead of training the policy to handle arbitrary viewpoints, ARGUS normalizes observations into a canonical viewpoint before they are fed to the downstream visuomotor policy. The process consists of three main stages:

1. Cross-View Point Cloud Estimation:
   Given RGB images from arbitrary camera viewpoints, ARGUS utilizes a large-scale, pretrained 3D vision model (specifically VGGT) to reconstruct a 3D point cloud of the scene. This model takes multi-view images as input and outputs a predicted point cloud and corresponding extrinsics. Notably, the initial reconstruction is scale-ambiguous and defined relative to one of the input camera frames.

2. Metric Scaling and World-Frame Alignment:
   To recover metric scale and align the geometry with the real world, the method leverages known real-world camera extrinsics (calibrated via AprilTags). It calculates the ratio between the inter-camera baseline in the real world and the baseline estimated by VGGT. This ratio is used to scale the VGGT point cloud. The scaled point cloud is then transformed into the robot's base frame using the known real-world extrinsic matrices.

3. Canonical Viewpoint Rendering:
   Once the point cloud is scaled and aligned to the world frame, the scene is rendered from a fixed, virtual canonical viewpoint. This viewpoint is selected to maximize the visibility of the workspace and manipulated objects. The resulting 2D RGB image is consistent regardless of the original camera placement, effectively removing viewpoint as a nuisance variable for the downstream policy.

The pipeline is designed to be modular, compatible with various visuomotor learning paradigms (e.g., Diffusion Policies), and does not require specialized depth sensors, relying instead on ubiquitous RGB cameras and learned 3D priors.

Key Contributions

• ARGUS Pipeline: A general-purpose viewpoint normalization framework that aligns observations from arbitrary camera angles to a canonical reference frame, facilitating more effective policy learning.
• Evaluation Across Diversity: Comprehensive evaluation demonstrating that ARGUS enables data-efficient learning and strong zero-shot generalization to unseen camera viewpoints, performing well in both fixed-viewpoint and highly diverse-viewpoint training regimes.
• Depth-Free Alternative: A comparison showing that learned 3D reconstruction via large vision models can serve as a competitive alternative to explicit depth sensing for viewpoint-robust visuomotor learning.

Experimental Results

The authors evaluated ARGUS on four real-world manipulation tasks (pick-and-place, stacking, unfolding) using datasets with varying levels of viewpoint diversity.

• Generalization Performance: In experiments involving novel camera configurations (up to 135° shifts), ARGUS consistently outperformed baselines, including KYC (which encodes camera rays as Plücker coordinates) and VISTA (which uses novel view synthesis for data augmentation). ARGUS demonstrated higher success rates with smaller standard errors across diverse configurations.
• Training Efficiency: When trained on highly diverse datasets, ARGUS converged to a 90% success rate 4–6× faster than previous methods (KYC and classical Diffusion Policy). It achieved a 70% success rate with only 40 demonstrations, highlighting significant improvements in sample efficiency.
• Comparison to Depth-Based Methods: When integrated with the 3D Diffuser Actor architecture, replacing raw depth sensors with VGGT-constructed point clouds, ARGUS achieved performance comparable to methods using explicit depth data, validating the efficacy of learned 3D reconstruction.

Significance and Limitations

The paper claims that ARGUS reduces the learning burden on visuomotor policies by explicitly canonicalizing observations. This allows policies to focus on task-relevant structure rather than learning to disentangle camera pose from scene understanding. The authors posit that this approach offers a practical, scalable alternative to relying solely on massive, viewpoint-diverse datasets to learn invariance.

However, the authors modestly acknowledge specific limitations:

• Spatial Precision: Performance degrades on fine-grained manipulation tasks involving very small objects (e.g., grasping a button). This is attributed to slight translational shifts caused by VGGT's extrinsic prediction errors and occlusion-induced holes in the reconstructed point cloud.
• Computational Overhead: The reliance on running VGGT at inference time introduces an average latency of approximately 0.52 seconds per prediction (on an RTX 3080). While acceptable for quasistatic tasks, this may limit applicability in scenarios requiring high-frequency closed-loop control.

In conclusion, the work suggests that leveraging large-scale 3D vision models to align scene geometry offers a robust pathway for improving the efficiency and generalization of robot learning from diverse, unstructured data.