Technical Summary: GeoProp – Grounding Robot State in Vision for Generalist Manipulation

1. Problem Statement

Current robotic manipulation architectures, ranging from diffusion-based controllers to Vision-Language-Action (VLA) models, suffer from a structural decoupling between high-dimensional visual observations and low-dimensional proprioceptive feedback. Standard fusion methods typically treat proprioception as an isolated, ungrounded global state vector, fusing it via simple concatenation or cross-attention.

The authors identify a critical limitation in this paradigm: without an explicit correspondence between 3D kinematics and 2D visual feature maps, the model must implicitly learn the alignment between the robot's physical state and the scene. Empirical analysis reveals that this ungrounded approach can be counterproductive, introducing spurious correlations that cause policies to underperform even vision-only baselines. The state vector fails to anchor the robot's configuration within the visual manifold, leading to diffuse attention patterns that do not focus on the gripper or manipulated objects.

2. Methodology: GeoProp

To address this, the authors propose GeoProp, a lightweight, plug-and-play adapter that aligns proprioception with vision through explicit geometric grounding and spatial feature sampling. GeoProp operates at the interface between perception and state representation without modifying the backbone architecture. It consists of three core components:

2.1 Geometric Projection and Feature Grounding

Instead of injecting the proprioceptive state $p_t$ as a global vector, GeoProp projects the 3D end-effector position $r_t$ onto the 2D image plane using camera intrinsics ($K$) and extrinsics ($R, t$).
$$q_t = \Pi_K(R r_t + t)$$
The projected image coordinate $q_t$ is mapped to a continuous coordinate $\bar{q}_t$ on the visual feature grid. The visual feature at this location is sampled via bilinear interpolation to create a grounded state token ($\tau_t$). This token carries local scene evidence specifically at the robot's current interaction point, inheriting scene semantics within the same visual latent space.

2.2 Spatially-Aligned Feature Modulation

To inject geometric priors deeper into the visual hierarchy, GeoProp employs FiLM (Feature-wise Linear Modulation) conditioned on the proprioceptive state.

• For each intermediate feature map $F^\ell_{enc}$ at level $\ell$, the aligned grid cell $c^\ell_t$ corresponding to the projected end-effector is identified.
• A two-layer MLP predicts channel-wise modulation parameters $\gamma^\ell_t$ and $\beta^\ell_t$ based on $p_t$.
• The modulation is applied only to the feature cell geometrically aligned with the end-effector:
  $$F^\ell_{mod}[c] = \begin{cases} (1 + \gamma^\ell_t) \odot F^\ell_{enc}[c] + \beta^\ell_t & \text{if } c = c^\ell_t \\ F^\ell_{enc}[c] & \text{otherwise} \end{cases}$$
  This ensures proprioceptive conditioning remains spatially localized prior to multi-scale aggregation via a Feature Pyramid Network (FPN).

2.3 Predictive Kinematic Sampling

To capture motion intent, GeoProp augments the grounded state with a predictive token.

• Using a temporal window of recent 3D positions, a short-horizon future waypoint $\hat{r}_{t+1}$ is estimated via polynomial extrapolation.
• This waypoint is projected to the image plane and mapped to the feature grid to obtain a look-ahead coordinate $\bar{q}^{pre}_{t+1}$.
• A predictive token ($\tau^{pre}$) is sampled from the unmodulated feature map at this location. This provides a spatial preview of anticipated motion without being conditioned on the state modulation, decoupling the "look-ahead" context from the current state.

The final policy input concatenates the grounded state token, the predictive token, and global visual tokens (and language tokens if available).

3. Key Contributions

1. Empirical Analysis of Misalignment: The authors demonstrate that treating proprioception as an ungrounded global vector can degrade manipulation performance compared to vision-only baselines, motivating the need for explicit, geometry-based state–vision correspondence.
2. GeoProp Adapter: They introduce a plug-and-play module that transforms 3D proprioception into image-grounded visual tokens. This bridges the representational gap via geometric projection, localized feature modulation (FiLM), and predictive sampling of motion intent.
3. Generalizability and Efficiency: GeoProp is validated across distinct policy architectures (Diffusion Policy and VLA models like $\pi_0$) and environments (simulation and real-world). It achieves significant performance gains with negligible parameter overhead (2–3%).

4. Experimental Results

The authors evaluated GeoProp across 67 tasks, including 63 simulation tasks and 4 real-world tasks on a Mobile ALOHA system.

Simulation Benchmarks

• Diffusion Policy (DP): Across 63 tasks (MetaWorld, RLBench, RoboTwin) with both ResNet-18 and ViT-Base backbones, GeoProp improved the average success rate by 8.7% over the vanilla fusion baseline.
• VLA Model ($\pi_0$): On the RoboTwin subset, GeoProp improved the success rate of the 3B-parameter $\pi_0$ model by 4.0%, outperforming the vanilla baseline on 5 out of 7 tasks.
• Ablation Studies: Geometric grounding was identified as the primary source of improvement. Adding predictive sampling and localized FiLM modulation provided complementary gains. The method outperformed alternative spatial grounding baselines like Heatmap injection and RC-PE.

Real-World Experiments

Evaluated on a Mobile ALOHA bimanual platform with four household tasks (Paper Toss, Coffee Retrieval, Desk Clearing, Table Cleaning):

• Diffusion Policy: GeoProp achieved a 50.0% average success rate, a 11.2% absolute improvement over the vanilla baseline and a 21.2% improvement over the vision-only baseline.
• $\pi_0$: GeoProp improved the average success rate from 43.8% to 53.8% (+10.0pp).
• The gains were most pronounced in tasks requiring precise local contact and gripper-object alignment.

Robustness

GeoProp relies on camera-robot calibration. Under synthetic extrinsic perturbations, the method degraded gradually but remained superior to baselines for translation errors up to 2.5 cm and rotation errors up to 1.5°.

5. Significance and Claims

The paper claims that GeoProp provides a simple yet high-impact inductive bias for generalist embodied policies. By anchoring the robot's state to co-located visual semantics through analytical projection, GeoProp resolves the modality misalignment inherent in standard fusion methods.

The authors emphasize that this approach is framework-agnostic, improving both shallow fusion methods (Diffusion Policy) and deep cross-modality attention models ($\pi_0$). The results suggest that explicit geometric grounding is a critical component for scaling embodied AI, enabling policies to better associate proprioceptive states with relevant visual evidence without requiring complex 3D scene reconstruction or changes to the backbone architecture.

Limitations Acknowledged:

• Requires known camera intrinsics and extrinsics; performance degrades with significant calibration drift.
• Currently grounds only the 3D end-effector position, not richer kinematic structures (e.g., joints, fingertips).
• Predictive sampling assumes locally smooth motion; abrupt direction changes may lead to off-task look-ahead points.
• Gains diminish or invert when the manipulated object occludes the projected end-effector or when the baseline is already near ceiling performance.