Technical Summary: G3VLA – Geometric Inductive Bias for Vision-Language-Action Models

Problem Statement

Vision-Language-Action (VLA) models, such as RT-2, OpenVLA, and $\pi_0$, have achieved significant progress in generalist robot manipulation by leveraging semantic knowledge from pretrained vision-language backbones. However, a critical limitation persists: their visual tokens are grounded in 2D image coordinates rather than the calibrated geometry of the robot's cameras. This mismatch is particularly pronounced in multi-camera setups, where known intrinsics and extrinsics geometrically couple different views, yet conventional VLA formulations process these views as independent 2D images. Consequently, the models must implicitly learn camera geometry solely from action supervision, a process that is often inefficient and limits spatial precision, especially in tasks requiring accurate pose estimation, distance reasoning, and cross-view consistency.

Existing attempts to address this, such as voxelizing observations (PerAct) or lifting features to 3D fields (Act3D), often require task-specific architectures that cannot fully leverage pretrained VLM semantics. Other bridging efforts (e.g., SpatialVLA, 3D-VLA) typically require explicit 3D sensor inputs, large-scale spatial pretraining, or modifications to the action representation. The central question addressed by this work is: Can calibrated camera geometry serve as a lightweight visual-token pathway for pretrained VLAs to improve spatial generalization without modifying the backbone, action space, or imitation objective?

Methodology: G3VLA

The authors propose G3VLA, a camera-aware geometric framework that injects calibrated structure directly into the visual-token stream of a pretrained VLA. The method preserves the pretrained backbone and action formulation, introducing geometry exclusively through the visual tokens.

Core Components

The Camera-Aware Geometric Module ($F_\psi$) consists of three key components:

1. Intrinsic-Conditioned Ray Embeddings:
   To resolve the ambiguity where the same pixel $(x, y)$ maps to different physical viewing directions under different intrinsics, the method tags each ViT patch token with its back-projected viewing direction. For a homogeneous pixel coordinate $u$, the normalized projection ray is computed as $\tilde{r}^v(u) = (K^v)^{-1}u$. A learnable embedding $G_\phi$ projects this ray map to the patch grid and adds it to the encoder output:
   $$z^v_{0,p} = z^v_p + G_\phi(R^v)_p$$
   This ensures all downstream attention operates on tokens aware of their intrinsic viewing direction.

2. Projective Positional Encoding (PRoPE):
   While ray embeddings encode local viewing directions, they do not capture geometric relationships between tokens from different views. PRoPE addresses this by deriving fixed projective transforms for query, key, and value representations based on per-view intrinsics ($K^v$), camera-to-world matrices ($T^v$), and patch locations. This provides cross-view attention with camera-model-based projective relations rather than relying solely on appearance similarity.

3. Bidirectional Cross-View Fusion:
   The fusion module processes tokens in two steps:

   • Frame Attention: Processes tokens within each camera stream independently to preserve view-local structure.
   • Cross-View Attention: Flattens view and patch dimensions, allowing all valid tokens to attend bidirectionally across views, using PRoPE as the positional signal. This exchanges geometric context before tokens reach the action model.

Training Strategy: Two-Stage Curriculum & Geometry Distillation

Since the geometric module is initialized from scratch and receives only sparse task-level gradients from the action loss, the authors introduce a dense auxiliary geometry distillation objective and a two-stage training curriculum:

• Geometry Distillation Objective: An auxiliary point head attached to the fused visual tokens predicts dense per-pixel ray coordinates ($\hat{q}^v_u$) and log-z depth ($\hat{d}^v_u$). Supervision targets come from either ground-truth point maps (in simulation) or confidence-gated predictions from a visual geometry teacher ($\pi_3X$). The teacher predictions are gated by a confidence threshold ($\tau=0.1$) to filter unreliable pixels, requiring no depth sensors or manual 3D annotations.
• Two-Stage Training:
  • Stage 1 (Geometric Module Pre-training): The pretrained backbone is frozen. Only the ray embeddings, cross-view fusion layers, and auxiliary point head are updated, with the distillation loss dominating. This aligns the geometric modules with the dense teacher signal.
  • Stage 2 (Full Policy Fine-tuning): All parameters are unfrozen. The action loss becomes dominant, while the distillation loss acts as a lightweight geometric regularizer. At inference, the auxiliary head and teacher are discarded; the policy requires only RGB images, proprioception, language, and camera calibration.

Key Contributions

1. Identification of a Geometric Gap: The paper highlights the mismatch between 2D-grounded visual tokens in pretrained VLAs and the calibrated spatial structure required for precise manipulation.
2. G3VLA Framework: A lightweight, backbone-preserving pathway that injects calibrated camera geometry via ray embeddings, PRoPE, and cross-view fusion without altering the action space or imitation objective.
3. Geometry Distillation from $\pi_3X$: A two-stage procedure that supervises geometric modules using confidence-gated dense point maps from a visual geometry teacher, eliminating the need for manual 3D annotations or depth sensors.
4. Multi-Architecture Validation: The method is validated across three distinct VLA systems ($\pi_0$, $\pi_0.5$, and GR00T 1.5), demonstrating consistent gains and providing architectural insights into how geometric signals propagate to action generation.

Experimental Results

Simulation Benchmarks

• LIBERO: On the $\pi_0$ backbone, G3VLA (GT) improved the macro-average success rate from 84.6% to 88.1% (+3.5 points). The largest gains were observed in LIBERO-Object (+5.0) and LIBERO-Spatial (+4.0), confirming that tasks requiring object localization and spatial reasoning benefit most. G3VLA ($\pi_3X$) also achieved significant gains (+2.4 over baseline), proving the utility of teacher-distilled geometry.
• RoboCasa24 & RoboTwin2.0: G3VLA (GT) improved success rates on RoboCasa24 (34.2% $\to$ 37.1%) and RoboTwin2.0 (44.0% $\to$ 49.0%). Notably, G3VLA ($\pi_3X$) underperformed on RoboTwin2.0 (dropping to 41.0%), which the authors attribute to unreliable offline $\pi_3X$ point-map targets in visually clean, texture-sparse synthetic scenes.
• Backbone Compatibility:
  • $\pi_0.5$: Gains were smaller but consistent (95.9% $\to$ 97.0%), confirming compatibility with stronger, near-saturated backbones.
  • GR00T 1.5: Results were mixed. Gains were observed with $\pi_3X$ supervision but not with GT. The authors suggest this is due to GR00T 1.5's two-tower architecture, where the diffusion policy accesses visual features via cross-attention to a frozen VLM rather than consuming geometry-aware tokens directly, attenuating the geometric signal.

Real-World Results

Evaluated on a bimanual UR5 robot with two tasks (Pick-and-Place Test Tube, Pouring Nut), G3VLA showed the most significant improvements in Out-of-Distribution (OOD) settings (unseen camera viewpoints).

• On the "Pouring Nut" task, G3VLA improved OOD success rates for $\pi_0$ from ~70-75% to ~83-87.5%.
• Qualitative analysis showed that GT-based variants exhibited robust recovery behaviors (e.g., re-grasping, re-aligning) after intermediate errors, which were absent in vanilla baselines.

Ablation Studies

• Geometric Components: Removing ray embeddings caused the largest drop (-2.0%), followed by removing PRoPE (-1.1%), confirming their complementary roles.
• Training Curriculum: Replacing the two-stage curriculum with one-stage joint training reduced performance by 0.7%, highlighting the importance of pre-aligning the geometric pathway before action optimization.
• Supervision Source: Ground-truth supervision provided the strongest signal, but $\pi_3X$ distillation recovered most of the gain, serving as a practical alternative when depth is unavailable.

Significance and Claims

The paper claims that calibrated camera geometry is a lightweight and useful inductive bias for spatial precision in generalist VLA policies. The primary significance lies in demonstrating that geometric structure can be injected into the visual-token stream of pretrained models without modifying the backbone, action space, or imitation objective.

The authors modestly conclude that the effectiveness of this geometric injection depends on the architecture: it is most effective when geometry-aware tokens have direct access to the action generation pathway (as seen in $\pi_0$ and $\pi_0.5$). In architectures where the action model accesses features indirectly (e.g., GR00T 1.5), the benefit is attenuated. Furthermore, the reliance on a visual geometry teacher introduces sensitivity to domain mismatch (e.g., texture-sparse scenes), suggesting that while teacher distillation is a viable strategy, ground-truth supervision remains superior when available.

Overall, G3VLA offers a practical approach to enhancing the spatial generalization of existing VLA models, particularly for tasks sensitive to object localization and spatial relations, by leveraging known camera calibration parameters as a structural prior.