🤖 AI

See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models

This paper introduces robot-centric pointmaps, a representation that encodes 3D scene geometry in the robot's coordinate frame while preserving the 2D grid structure required by pretrained vision-language-action models, thereby resolving frame mismatches and significantly improving generalization across diverse camera viewpoints in both simulation and real-robot experiments.

Original authors: Byungkun Lee, Dongyoon Hwang, Dongjin Kim, Hojoon Lee, Minho Park, Jaegul Choo

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Byungkun Lee, Dongyoon Hwang, Dongjin Kim, Hojoon Lee, Minho Park, Jaegul Choo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to make a sandwich. You want the robot to grab the bread and put it on a plate. But here's the tricky part: the robot's "brain" (the part that decides where to move its arm) speaks a different language than its "eyes."

The robot's arm moves in a map based on its own body (let's call this the Robot Map). "Move 10 centimeters forward" means something very specific to the arm. But the robot's eyes see the world through a camera, which has its own map (the Camera Map). If the camera is on the robot's wrist, it sees things differently than if it's on a shelf across the room.

For a long time, robot brains were taught to look at the Camera Map and guess how to move the arm on the Robot Map. This works fine if the camera never moves. But what if you train the robot using videos from 50 different camera angles? The robot gets confused. It has to memorize a million different ways to translate "what the camera sees" into "what the arm does." It's like trying to learn a new language every time you turn your head.

The "Robot-Centric" Solution: A New Kind of Photo

The authors of this paper suggest a clever fix: Stop asking the robot to translate. Just give it the answer in the right language from the start.

They created a special kind of image called a Robot-Centric Pointmap.

Think of a normal photo as a grid of colored pixels. Now, imagine a photo where, instead of just storing colors (Red, Green, Blue), every single pixel also stores a secret code: its exact 3D location on the Robot Map.

Normal Photo: "This pixel is a red apple."
Pointmap: "This pixel is a red apple located at coordinates (X, Y, Z) relative to the robot's arm."

The magic is that this Pointmap looks exactly like a normal photo to the robot's brain. It keeps the same grid shape (height × width) that the robot is already used to processing. The robot doesn't need to learn a new way to see; it just gets a "super-photo" that already knows where things are in the robot's own space.

What They Tried (And What They Said "No" To)

The researchers didn't just guess; they tested different ways to give the robot 3D information. Here is what they found:

Don't just give the robot the camera's settings: Some might think, "Let's just tell the robot, 'The camera is tilted 30 degrees,' and let it figure out the rest." The paper shows this is a weak strategy. It's like giving someone a map and a compass but expecting them to draw the route themselves. The robot struggled more with this than with the Pointmap.
Don't throw away the photo grid: Another idea was to turn the 3D world into a "point cloud" (a scattered cloud of dots). The paper argues this is a bad idea for these specific robots. It's like taking a high-definition photo and turning it into a low-resolution sketch made of 4,000 dots. The robot loses the fine details it needs, and it has to learn a whole new way to process that cloud of dots. The Pointmap keeps the high-definition grid, which the robot's brain already loves.
Center the map on the hand, not the floor: When creating the Pointmap, you have to choose a "center point." Should the coordinates be measured from the robot's base (its feet) or its hand (the gripper)? The paper found that measuring from the hand is much better.
- Analogy: If you are reaching for a cup, the cup's position relative to your feet changes if you walk around the kitchen. But the cup's position relative to your hand is always the same right before you grab it. By centering the map on the hand, the robot sees the same "grabbing shape" no matter where the cup is in the room.

The Results: Does It Actually Work?

The team tested this idea in two places: a computer simulation called RoboCasa and on a real robot arm in a lab.

In the Simulation (RoboCasa):
They trained robots on 24 different tasks, like opening doors or picking up coffee cups.

Without the Pointmap, a standard robot model (called π0.5) succeeded 55.3% of the time.
With the Pointmap, that number jumped to 62.9%.
They also tested a smaller model (SmolVLA), which went from 37.2% to 41.4%.
The Pointmap beat other fancy methods that tried to add 3D data in different ways.

In the Real World (The Franka Robot):
This is where it gets really cool. They trained the robot with cameras in three different spots. Then, they tested it in a fourth spot that the robot had never seen before.

The "Seen" Test: When the camera was in a spot the robot knew, the Pointmap helped the robot succeed 5.0% more often than the standard version.
The "Unseen" Test: When the camera was moved to a totally new spot, the standard robot's performance crashed (dropping from 73.3% to 55.0%). But the Pointmap robot stayed strong, only dropping a little bit.
The Result: The advantage of the Pointmap grew from 5.0% to a massive 11.7% when the camera moved to a new place.

The Bottom Line

The paper suggests that if you want a robot to be good at moving its arm in a 3D world, especially when it has to learn from many different camera angles, you shouldn't make it guess the geometry. Instead, give it a "super-photo" (the Pointmap) that already translates the world into the robot's own language.

It's a simple trick: Don't make the robot do the math of "where am I?" Give it a map that already says, "You are here, and the object is there." This allows the robot to focus on the task—like grabbing a cup—rather than worrying about where the camera is standing.

Technical Summary: Robot-Centric Pointmaps for Vision-Language-Action Models

Problem Statement
Vision-Language-Action (VLA) models predict robotic manipulation actions based on visual observations and language instructions. A fundamental mismatch exists in current VLA architectures: actions are defined in the robot's 3D coordinate frame (e.g., task-space end-effector commands relative to the robot base), yet observations are typically received in the camera frame (RGB images or depth maps). While this mismatch is manageable under fixed viewpoints where the policy can memorize a single observation-to-action mapping, it becomes significantly harder when training data aggregates demonstrations from diverse camera setups. In such scenarios, the policy must generalize the mapping across viewpoints rather than simply memorizing one, leading to degraded performance when camera placements vary.

Methodology
The authors propose Robot-Centric Pointmaps as an input representation to resolve this frame mismatch. Instead of forcing the policy to infer robot-frame geometry from camera-frame observations, the method pre-computes the 3D coordinates of scene points in the robot's frame and presents them as an image.

The core mechanism involves three steps:

Lifting and Transformation: An RGB-D observation is lifted into 3D within the camera frame using intrinsics and depth. These 3D points are then transformed into the shared robot base frame using the camera-to-robot extrinsic transform.
End-Effector Centering: The resulting pointmap is re-centered on the current end-effector position. This expresses scene geometry relative to the action origin, ensuring that the displacement required to reach a target remains consistent regardless of the target's absolute position in the workspace.
Fusion: The robot-centric pointmap retains the dense $H \times W$ grid structure of the original RGB image. It is encoded by a separate visual tower (initialized from the RGB encoder) and fused with the RGB tokens via element-wise addition. This design allows the integration of metric 3D geometry into pretrained 2D VLAs without requiring point-cloud-specific encoders, voxelization modules, or additional token sequences.

Key Contributions and Design Choices
Through controlled ablation studies on the RoboCasa dataset, the paper establishes several critical design principles:

Pre-computation vs. Learning: Providing the policy with pre-computed robot-frame geometry is significantly more effective than providing raw depth and camera calibration (intrinsics/extrinsics) and asking the policy to learn the transformation.
Image-Form vs. Point Cloud: Preserving the image grid ( $H \times W$ ) is crucial. Pointmaps outperform point clouds because they allow for direct element-wise fusion with RGB tokens, leveraging the pretrained visual pathway. Point clouds, which require dedicated encoders and often result in sparse inputs, fail to maintain spatial correspondence with RGB features.
Coordinate Origin: Centering the pointmap on the end-effector rather than the robot base yields a more viewpoint-robust representation. It clusters interaction targets near the origin, providing a consistent local geometry for similar motions regardless of where they occur in the workspace.

Experimental Results
The proposed method was evaluated on the RoboCasa simulation benchmark and on a real Franka Research 3 robot.

Simulation (RoboCasa): When applied to $\pi0.5$ and SmolVLA backbones, robot-centric pointmaps improved success rates by +7.6 and +4.2 points, respectively, across 24 tasks. The method outperformed representative camera-aware baselines (e.g., KYC, OC-VLA) and 3D-augmented baselines (e.g., GeoVLA, PointVLA) that share the same backbone.
Robustness to Viewpoint Variation: As training-time camera viewpoint variation increased, RGB-only policies suffered significant performance drops. In contrast, the pointmap-based policy remained stable.
Real-Robot Generalization: In real-world experiments, the advantage of pointmaps widened when the camera was moved to a placement unseen during training. The performance gap over an RGB-only policy grew from +5.0 points at a seen camera placement to +11.7 points at a held-out placement. While a point-cloud-based policy (DP3) showed better robustness than RGB, the pointmap-enhanced VLA achieved the highest overall performance, demonstrating that combining 3D geometry with the inductive biases of pretrained 2D VLAs is highly effective.

Significance and Claims
The paper claims that robot-centric pointmaps offer a simple yet effective mechanism to inject 3D geometry into pretrained VLAs without architectural overhaul. The primary significance lies in addressing the observation-to-action frame mismatch directly: by expressing observations in the coordinate system where the robot acts, the policy can learn from viewpoint-diverse demonstrations without needing to absorb the variation of the camera itself. The authors suggest a general principle for 3D-aware VLAs: when calibration is available, observations should be expressed in the robot's action frame before being passed to the policy. This approach retains the dense image structure required by pretrained models while providing the metric 3D information necessary for robust manipulation.

The "Robot-Centric" Solution: A New Kind of Photo

What They Tried (And What They Said "No" To)

The Results: Does It Actually Work?

The Bottom Line

Technical Summary: Robot-Centric Pointmaps for Vision-Language-Action Models

More like this