Technical Summary: Supervise What Survives: Geometry-Guided VLA Adaptation from Synthetic Robot Videos

1. Problem Statement

Vision-Language-Action (VLA) models require large-scale paired video-action data to learn effective robot policies. However, real-world teleoperation data is scarce due to the high costs of hardware, operator training, and collection time. While human-to-robot video generation offers a scalable source of synthetic robot footage, existing adaptation methods treat these generated videos as if they contain valid action labels. These methods typically recover "pseudo-actions" (e.g., via inverse dynamics or geometric retargeting) from synthesized pixels and feed them into the VLA's action head.

The authors argue this approach creates a fundamental mismatch. Video generation optimizes for visual plausibility, not physical control laws. Consequently, while the geometry (spatial trajectories, object arrangements) of a task is preserved during generation, the underlying control signals (exact motor commands) are lost or hallucinated. Forcing a VLA to map visually plausible but physically hallucinated pixels to rigid real-world control signals degrades policy performance.

2. Core Principle: Asymmetric Preservation

The paper introduces the Asymmetric Preservation Principle, which posits that in human-to-robot video generation:

• Geometry Survives: The spatial structure (the "where" of a task) is reliably transferred.
• Control is Erased: The physical execution dynamics (the "how") are not preserved.

Based on this, the authors propose a design rule: Supervision should follow what survives. Generated videos should be used solely to supervise visual perception (geometry), while control learning must remain grounded in real demonstrations.

3. Methodology: GRA (Geometry-guided Representation Alignment)

The proposed method, GRA, implements this principle through a two-stage training protocol that decouples geometric supervision from action learning.

3.1 Geometric Waypoint Derivation

Instead of recovering actions from generated pixels, GRA derives geometric supervision targets from the source human video:

1. Extraction: Uses pose estimation (HaMeR), detection (Grounding DINO), and segmentation (SAM2) on the human video.
2. Retargeting & Simulation: Retargets human hand trajectories to a robot's end-effector (EE) frame and replays them in a physics simulator (MuJoCo) to enforce joint limits and collisions.
3. Projection: Projects the simulated 3D EE trajectory into 2D image coordinates using calibrated camera intrinsics/extrinsics.
4. Result: A set of future 2D EE waypoints ($y_t$) that are temporally aligned with the generated robot frames but originate from the reliable human source, not the synthetic pixels.

3.2 Two-Stage Training Protocol

• Stage 1: Geometric Pretraining

  • Input: Generated robot frames ($I^s_t$).
  • Objective: Train the vision backbone ($\phi_v$) and an auxiliary 2D head ($h_{2D}$) to predict the future 2D waypoints ($y^s_t$) derived from the human source.
  • Loss: $L^s_{2D}$ (MSE between predicted and derived waypoints).
  • Scope: Only the vision backbone and 2D head are updated (via LoRA); the language model and action head are untouched.

• Stage 2: Action Fine-tuning with Spatial Anchor

  • Input: Real robot demonstration frames ($I^r_t$) with real delta actions ($\Delta a_t$).
  • Objective: Fine-tune the action head ($\phi_a$) to predict real actions while maintaining the geometric grounding established in Stage 1.
  • Loss: $L = L_{act} + \lambda L^r_{2D}$.
    • $L_{act}$: L1 regression on real delta actions.
    • $L^r_{2D}$: The same 2D waypoint loss from Stage 1, now evaluated on real frames.
  • Mechanism: The auxiliary head $h_{2D}$ acts as a spatial representation anchor. It persists during action fine-tuning to prevent the vision backbone from drifting toward action-predictive features at the expense of spatial structure.

4. Key Contributions

1. Asymmetric Preservation Principle: A formalization of the structural gap in synthetic data, arguing that generated videos are reliable geometric witnesses but unreliable control traces.
2. Supervision Routing (GRA): A novel architecture that routes human-derived geometric waypoints to the VLA vision backbone via an auxiliary 2D head, while restricting action learning to real demonstrations only.
3. Representational Anchoring: Demonstration that a persistent spatial loss during action fine-tuning prevents the erosion of geometry-aware representations, which is critical for maintaining spatial grounding in the policy.

5. Experimental Results

Experiments were conducted on a 7-DoF Franka Research 3 robot with three pick-and-place tasks (cube, cup, mango).

• Setup: Compared against baselines using a matched data budget ($N_{real}=25$ real demos + $N_{syn}=75$ generated videos).
  • Baselines: Real-only (25 demos), DreamGen-style (pseudo-actions via Inverse Dynamics), MimicDreamer-style (pseudo-actions via retargeting).
  • Upper Reference: Real-only (100 demos).
• Performance:
  • GRA achieved a mean success rate of 68.9%, outperforming the matched-budget Real-only baseline (61.1%) by +7.8% and significantly outperforming pseudo-action baselines (48.9% and 54.4%).
  • Pseudo-action baselines performed worse than the Real-only baseline, confirming the hypothesis that synthetic control signals degrade performance.
  • GRA narrowed the performance gap to the 100-demo Real-only reference (75.6%) from 14.5 points (Real-only 25) to 6.7 points, despite using 4x fewer real demonstrations.
• Diagnostics:
  • Information Asymmetry: Probes showed that spatial information transfers from synthetic to real data much better than control information.
  • Action Head Quality: GRA reduced per-step position drift by 17% compared to Real-only and 30% compared to MimicDreamer-style, indicating that geometric pretraining sharpens spatial perception without introducing synthetic action noise.

6. Significance and Limitations

Significance:
The paper demonstrates that correctly routing synthetic data—using it only for what it reliably preserves (geometry)—bridges the gap between generated videos and robot policies more effectively than attempting to reconstruct erased control signals. This approach allows VLA models to leverage scalable synthetic data for visual grounding while maintaining the fidelity of control learned from scarce real data.

Limitations:

• Upstream: The asymmetry is observed empirically; characterizing it across different generators or extending "what survives" beyond geometry remains an open problem.
• Downstream: Closed-loop performance is still bounded by the scale of real action data. The method does not eliminate the need for real demonstrations, only reduces the quantity required for effective adaptation. Future work would require generators or decoders capable of producing trustworthy control signals.