🤖 machine learning

Geometric Action Model for Robot Policy Learning

The paper introduces the Geometric Action Model (GAM), a language-conditioned manipulation policy that repurposes a pretrained geometric foundation model by splitting it to integrate a causal future predictor, thereby enabling efficient, accurate, and robust 3D reasoning for robot control while preserving rich geometric priors.

Original authors: Jisang Han, Seonghu Jeon, Jaewoo Jung, René Zurbrügg, Honggyu An, Tifanny Portela, Marco Hutter, Marc Pollefeys, Seungryong Kim, Sunghwan Hong

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jisang Han, Seonghu Jeon, Jaewoo Jung, René Zurbrügg, Honggyu An, Tifanny Portela, Marco Hutter, Marc Pollefeys, Seungryong Kim, Sunghwan Hong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to do chores, like stacking blocks or putting a pot on a stove. For a robot to do this well, it needs to understand three things at once: what you told it to do (language), what it sees right now (vision), and how the world will change when it moves (physics/geometry).

Most current robot "brains" are like people who only look at flat, 2D photographs. They are great at recognizing objects but struggle to understand depth, distance, or how things will fall over when pushed. They have to guess the 3D shape of the world just by looking at a flat picture, which is like trying to judge the size of a mountain by looking at a postcard.

The paper introduces a new robot brain called GAM (Geometric Action Model). Here is how it works, using simple analogies:

1. The "Pre-Trained Architect" (The Foundation Model)

The researchers didn't build a robot brain from scratch. Instead, they took a massive, pre-trained "Geometric Foundation Model" (GFM). Think of this GFM as a super-architect who has spent years studying blueprints and 3D structures. This architect already knows how to turn flat photos into detailed 3D maps. It understands depth, scale, and how objects occupy space.

2. The "Split and Insert" Strategy

Usually, people use this super-architect just to look at the room and describe it, then hand that description to a separate "doer" robot to move. GAM changes the game by splitting the architect in half and inserting a new "planner" right in the middle.

The Top Half (The Eyes): The first part of the architect looks at the current camera photos and turns them into a 3D mental map.
The Middle (The Time-Traveler): Right in the middle of the architect, the researchers inserted a special "Causal Future Predictor." Imagine this as a time-traveling planner. It takes the current 3D map, listens to your instructions ("Put the cup here"), and asks: "If I move my arm like this, what will the 3D world look like one second from now?"
The Bottom Half (The Hands): The second part of the architect takes that future 3D prediction and uses it to figure out exactly how the robot should move its arm to make that future happen.

3. Why This is a Big Deal

Most other robot models try to predict the future in 2D (like a video game) or guess the 3D shape at the very end. GAM does it differently:

It thinks in 3D from the start: Because it uses the architect's 3D knowledge for everything, the robot doesn't have to guess if an object is far away or close. It knows.
It's a "One-Pass" Wonder: Other models often have to run a complex, slow process (like a diffusion model) many times to figure out a single move, similar to a painter trying to fix a mistake by repainting the whole canvas over and over. GAM is like a sketch artist who draws the whole plan in one quick, confident stroke.
It's Fast and Light: Because it's so efficient, it runs 55 times faster than some of the biggest, slowest models out there, and it uses much less computer memory.

4. The Results: Robustness

The paper tested this robot in two ways:

Simulation: In a computer world where they changed the lighting, moved the cameras, and scrambled the background.
Real Life: On an actual robot arm in a real room.

The Analogy of the "Perturbation":
Imagine you are walking through a room. If the lights flicker or someone moves a chair, a robot that only sees 2D photos might get confused and think the chair disappeared or moved to a different spot.

Old Robots: When the camera angle changed slightly, their success rate dropped dramatically (like a driver getting lost when the GPS signal flickers).
GAM: Because it understands the true 3D geometry of the room, it didn't care if the camera moved or the lighting changed. It knew exactly where the objects were in space. In tests where the camera was moved to a weird angle, GAM stayed successful while others failed.

Summary

GAM is a robot policy that takes a "3D expert" (a geometric foundation model), cuts it open, and inserts a "future planner" inside. This allows the robot to see in 3D, predict the future in 3D, and act in 3D all at the same time. The result is a robot that is faster, smaller, and much better at handling real-world messiness (like moving cameras or changing lights) than current state-of-the-art robots.

Technical Summary: Geometric Action Model for Robot Policy Learning

Problem Statement

The development of generalist robot manipulation policies requires models that can follow natural language instructions while reasoning about the complex interactions between objects, cameras, and robot actions in 3D physical space. While recent Vision-Language-Action models (VLAs) and Video World-Action models (WAMs) leverage large-scale foundation models for semantic or temporal priors, they primarily operate on 2D image frames or 2D-derived latent spaces. Consequently, critical 3D cues—such as depth, scale, and occlusion—remain implicit. This limitation hinders generalization across environmental changes, particularly regarding robot initial states and camera viewpoints. Existing attempts to incorporate 3D geometry often treat Geometric Foundation Models (GFMs) merely as static feature extractors or distill their features into VLA backbones, failing to repurpose the GFM's multi-layer geometric structure as the policy's own temporal and action-generating substrate.

Methodology: The Geometric Action Model (GAM)

The authors propose the Geometric Action Model (GAM), a language-conditioned manipulation policy that directly repurposes a pretrained Geometric Foundation Model (GFM) as a shared substrate for perception, temporal prediction, and action decoding.

Core Architecture

GAM operates by splitting a pretrained GFM (e.g., DA3) at an intermediate layer $L_s$ and inserting a Causal Future Predictor between the shallow and deep layers. The framework functions in three sequential stages:

Observation Encoding: The shallow layers of the GFM ( $E_{\le L_s}$ ) act as an observation encoder. They process multi-view RGB observations to extract latent geometric features, producing a sequence of per-timestep geometric latent states.
Causal Future Prediction: At the split layer, a causal transformer predictor ( $g_\phi$ $g_{ϕ}$ ) is inserted. It takes the current geometric latent tokens, concatenated with tokens representing task instructions (via a frozen text encoder), proprioceptive states, and action history. Using block-causal self-attention, this predictor forecasts two outputs simultaneously:
- Future latent geometric tokens ( $\tilde{Z}^{(L_s)}_{t+1}$ ).
- A predicted next action token ( $\tilde{a}_t$ ).
  This design allows the model to learn language-conditioned temporal world modeling with minimal architectural modification, as only the inserted predictor needs to learn the fusion of language, proprioception, and action history with GFM features.
Feature Propagation and Decoding: The predicted future tokens and the action token are routed through the remaining deep GFM blocks ( $D_{>L_s}$ $D_{> L_{s}}$ ). The deep blocks, originally pretrained to decode shallow features into 3D geometry, are repurposed to decode the predicted future tokens. This stage produces:
- Future Geometry: Decoded via the GFM's depth head into action-aligned future depth maps.
- Action Chunk: Decoded via a lightweight action head into executable delta-pose or joint commands.

Training Objective

The policy is trained end-to-end using a multi-task loss function:
$\mathcal{L}_{total} = \lambda_{act}\mathcal{L}_{act} + \lambda_{feat}\mathcal{L}_{feat} + \lambda_{depth}\mathcal{L}_{depth}$

Action Loss ( $\mathcal{L}_{act}$ ): $\ell_1$ regression between the decoded action chunk and expert actions.
Feature Loss ( $\mathcal{L}_{feat}$ ): Aligns predicted future latent tokens with the actual next frame's latent tokens (extracted from a frozen GFM) to anchor the predictor to temporal geometric transitions.
Depth Loss ( $\mathcal{L}_{depth}$ ): Supervises the decoded future depth maps against ground-truth future depth (or pseudo-labels in real-world settings) using scale-invariant and gradient-matching penalties.

Key Contributions

Unified Architecture: GAM introduces a single shared-backbone architecture that combines temporal world modeling, latent feature-space prediction, and a geometric foundation model substrate.
Shared Token Space Prediction: The authors demonstrate that action and geometry can be predicted within a shared autoregressive token sequence. A single forward pass produces both action tokens and future-scene tokens, decoded by lightweight heads, rather than relying on separate heads or diffusion processes.
Repurposing GFMs: Unlike prior works that use GFMs as static extractors, GAM explicitly repurposes the GFM's deep blocks to decode predicted future tokens, fully leveraging the model's geometric priors for dynamic world modeling.

Experimental Results

The authors evaluated GAM across diverse simulation (LIBERO, LIBERO-Plus) and real-world robot manipulation benchmarks.

Performance vs. Baselines: GAM achieves success rates comparable to or exceeding current foundation-model-scale baselines (VLAs and WAMs) while using significantly fewer parameters (1.4B vs. 7B+).
Robustness: GAM demonstrates superior robustness to out-of-distribution perturbations. Notably, in the camera-perturbation setting of LIBERO-Plus, GAM outperforms the best baseline by 9.7 percentage points.
Efficiency: GAM achieves a single feed-forward inference latency of 6.9 ms (approx. 145 Hz), which is approximately 55× faster than diffusion-based policies like Cosmos-Policy.
Real-World Transfer: In real-world experiments involving four manipulation tasks, GAM substantially outperformed baselines (including $\pi0.5$ and Spatial Forcing), maintaining robustness under camera perturbations where other models struggled.

Significance and Claims

The paper claims that GAM represents a significant step toward generalist robot policies by resolving the spatial ambiguities inherent in 2D-centric foundation models. By integrating geometric world dynamics directly into the policy's predictive pathway, GAM achieves a balance of high accuracy, robustness to environmental changes, and computational efficiency. The authors posit that their approach proves that action and geometry can be effectively modeled in a shared latent space, enabling a single backbone to handle perception, future-state prediction, and control.

The authors acknowledge limitations, noting that the model's language reasoning and commonsense capabilities are bounded by the frozen text encoder used, suggesting that integrating a large language model or external reasoning module is a natural direction for future work.