💻 computer science

Point Bridge: 3D Representations for Cross Domain Policy Learning

Point Bridge is a framework that enables zero-shot sim-to-real policy transfer for robot manipulation by leveraging unified, domain-agnostic point-based representations extracted via Vision-Language Models, thereby overcoming visual domain gaps and achieving significant performance gains over prior methods.

Original authors: Siddhant Haldar, Lars Johannsmeier, Lerrel Pinto, Abhishek Gupta, Dieter Fox, Yashraj Narang, Ajay Mandlekar

Published 2026-03-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Siddhant Haldar, Lars Johannsmeier, Lerrel Pinto, Abhishek Gupta, Dieter Fox, Yashraj Narang, Ajay Mandlekar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to do chores, like putting a bowl on a plate or stacking cups. The old way of doing this is like hiring a human to sit next to the robot for months, manually guiding its arm through every single movement. This is slow, expensive, and you can only teach it a few specific tricks.

Another way is to build a perfect, photorealistic video game world (a simulation) and let the robot practice there millions of times. The problem? The robot gets really good at the game, but when you put it in your actual kitchen, it fails miserably. It's like a pilot who has flown thousands of hours in a flight simulator but has never felt the wind or the turbulence of a real plane. The "visual gap" between the game and reality is too big.

Enter "Point Bridge."

The authors of this paper built a clever "bridge" that lets a robot learn in a video game and then immediately work in the real world, without needing to be retrained or manually aligned. Here is how they did it, using some simple analogies:

1. The "Skeleton Key" Instead of the "Painting"

Most robots learn by looking at raw images (pixels), like a painting. If the lighting changes, or the bowl is a different color, or the table looks different, the robot gets confused because the "painting" looks different.

Point Bridge ignores the paint. Instead, it looks at the skeleton of the scene.

The Analogy: Imagine you are trying to recognize a friend in a crowd. If you focus on their clothes (the "pixels"), you might get confused if they change outfits. But if you focus on their height, the shape of their nose, and where their hands are (the "points"), you recognize them instantly, no matter what they are wearing.
How it works: The system uses advanced AI (called Vision-Language Models) to look at a photo of a kitchen and say, "Okay, I see a bowl and a plate." It then ignores the background, the lighting, and the texture. It simply extracts a few 3D dots (points) that represent the shape and location of the bowl and plate.

2. The "Universal Translator"

Because the robot is only looking at these 3D dots (the skeleton), it doesn't matter if the simulation looks like a cartoon or if the real kitchen is messy.

The Analogy: Think of the robot's brain as a person who only speaks "Dot Language." In the video game, the bowl is made of 100 dots. In the real world, the bowl is also made of 100 dots. Even if the real bowl is shiny and the game bowl is matte, the arrangement of the dots is the same. The robot speaks the same language in both worlds, so it doesn't need a translator.

3. The "Magic Filter" (VLMs)

How does the robot know which dots to look at?

The Analogy: Imagine you give a robot a command: "Put the bowl on the plate." A normal robot might look at everything in the room—the curtains, the toaster, the cat.
Point Bridge uses a "Magic Filter" (powered by AI like Gemini). When you give the command, the filter instantly highlights only the bowl and the plate and turns the rest of the world into a blank void. It then grabs the 3D coordinates of just those two items. This happens automatically, so you don't have to manually tell the robot what to look at for every new task.

4. The "Training Montage"

The paper shows that you can train the robot almost entirely in the video game using this "Dot Language."

Zero-Shot Transfer: You train the robot in the game, and the moment you take it to the real world, it works. It's like learning to swim in a pool and immediately being able to swim in the ocean without practicing in the ocean first.
The "Co-Training" Boost: If you do have a tiny bit of real-world data (like 45 real-life examples), you can mix it with the millions of game examples. This is like giving the pilot a few hours of real flight time. The paper shows this makes the robot even better, beating all previous methods by a huge margin (up to 66% better!).

Why is this a big deal?

Scalability: We can now generate millions of training examples in a computer game for free, and the robot will actually learn from them.
Flexibility: The robot can learn to stack bowls, fold towels, or open drawers without needing a completely new setup for each task.
Speed: It doesn't need months of human teleoperation (remote control) to learn.

In summary: Point Bridge stops trying to teach robots to "see" like humans (with all the confusing details of color and light) and instead teaches them to "feel" the 3D structure of the world using simple points. By speaking this universal language of points, a robot can learn in a video game and walk right out the door to do your chores.

1. Problem Statement

The development of generalist robotic agents is currently bottlenecked by the scarcity of large-scale, real-world manipulation datasets. While simulation and synthetic data generation offer a scalable alternative, they suffer from a significant visual domain gap between simulated environments and the real world.

Limitations of Current Approaches: Existing methods often require careful visual or object-level alignment between simulation and reality, rely on photorealistic simulators (which are expensive to build), or depend on large amounts of real-world data to bridge the gap.
The Gap: There is a need for a framework that can leverage massive synthetic datasets for zero-shot sim-to-real transfer without requiring explicit visual alignment or large real-world datasets, while also supporting multitask learning.

2. Methodology: POINT BRIDGE

POINT BRIDGE is a framework that utilizes unified, domain-agnostic point-based representations to abstract away raw visual differences (textures, lighting, backgrounds) and focus on task-relevant geometric structures. The framework operates in three main stages:

A. Data Collection & Synthetic Generation

Source: Starts with a small set of human teleoperated demonstrations ( $D_{src}$ ).
Expansion: Uses MimicGen to expand these demonstrations into large-scale synthetic datasets ( $D_{sim}$ ) by adapting trajectories to new object instances and scenes via SE(3) transformations.
Co-training Option: Allows for the inclusion of a small set of real-world demonstrations ( $D_{real}$ ) for joint training.

B. Unified Point Extraction Pipeline

The core innovation is converting raw observations into a compact set of 3D keypoints ( $P$ ) that serve as the policy input.

In Simulation: Points are sampled directly from object meshes. To mimic real-world limitations, the system projects these points through virtual camera parameters (intrinsics/extrinsics) and adds Gaussian noise to simulate sensor imperfections.
In Reality (VLM-Guided Pipeline):
1. Object Identification: A Vision-Language Model (VLM, specifically Gemini-2.5-flash) identifies task-relevant objects based on natural language instructions (e.g., "bowl", "plate").
2. Localization & Segmentation: A specialized VLM (Molmo-7B) localizes these objects, followed by SAM-2 to extract 2D segmentation masks.
3. 3D Projection: 2D points are sampled from the masks and lifted to 3D using depth maps generated by Foundation Stereo (a zero-shot stereo matching model). This is preferred over RGB-D sensors for its robustness with transparent/reflective objects.
4. Robot Representation: The robot end-effector is represented as a set of rigid keypoints relative to the gripper pose.
Result: Both simulation and real-world data are converted into a unified 3D point cloud representation in the robot's base frame, effectively bridging the domain gap.

C. Policy Learning Architecture

Architecture: A decoder-only Transformer policy (inspired by BAKU).
Input:
- Encoded 3D point clouds (Robot points + Object points) via a PointNet encoder.
- Language embeddings (for multitask conditioning) via MiniLM.
- Historical observations (action chunking).
Output: Predicts the next robot end-effector pose and gripper state using a deterministic action head.
Training: Optimized using Behavior Cloning (maximizing log-likelihood of expert actions) with Mean Squared Error (MSE) loss.

D. Inference

During deployment, the same VLM-guided pipeline extracts 3D points in real-time. The system supports multiple depth sensing strategies (Foundation Stereo, RGB-D, or multi-view triangulation) to adapt to different hardware setups.

3. Key Contributions

Unified Domain-Agnostic Representation: Introduces a framework that abstracts scenes into task-relevant 3D keypoints, enabling zero-shot transfer without explicit visual alignment.
Automated VLM Pipeline: Develops a novel pipeline using VLMs (Gemini, Molmo, SAM-2) and Foundation Stereo to automatically extract 3D keypoints from real-world images, eliminating the need for manual annotation.
Scalable Multitask Learning: Demonstrates that the point-based representation naturally supports multitask policies conditioned on language instructions, scaling effectively across diverse tasks.
Efficient Co-training: Shows that combining large-scale synthetic data with minimal real-world data (e.g., 45 demonstrations) yields superior performance compared to image-based co-training methods.

4. Experimental Results

The authors evaluated POINT BRIDGE on 6 real-world tasks (3 rigid object manipulation, 3 involving soft/articulated objects) using a Franka Research 3 arm.

Zero-Shot Sim-to-Real Transfer:
- Achieved 44% higher success rates in single-task settings and 66% higher in multitask settings compared to prior image-based baselines (which often failed entirely in zero-shot settings).
- Outperformed the strongest baseline by 39% (single-task) and 44% (multitask).
- Successfully transferred policies across diverse object instances and significant visual discrepancies (lighting, background, table appearance).
Co-training with Real Data:
- When co-trained with a small amount of real data (80% sim / 20% real), performance improved further, surpassing prior sim-and-real co-training methods by 61% (single-task) and 66% (multitask).
- Achieved up to 100% success rate in multitask settings with co-training.
Generalization:
- Successfully handled soft objects (towels) and articulated objects (drawers, ovens) without simulation data, achieving an 85% success rate on these specific real-world tasks.
- Demonstrated robustness to background distractors and held-out object instances.
System Analysis:
- Depth Sensing: Foundation Stereo proved superior to RGB-D and multi-view triangulation for handling reflective/transparent objects and maintaining robustness.
- Camera Alignment: Training with camera-aligned points in simulation (mimicking real viewpoints) significantly improved transfer performance over uniform mesh sampling.

5. Significance

Bridging the Data Gap: POINT BRIDGE provides a viable path to utilizing the massive scale of synthetic data for real-world robotics, reducing the reliance on prohibitively expensive and time-consuming real-world data collection.
Eliminating Alignment Requirements: By moving from raw pixels to semantic 3D points, the framework removes the need for "digital twin" level visual fidelity or careful scene alignment, making sim-to-real transfer more accessible.
Foundation for Generalist Agents: The ability to learn multitask policies from synthetic data with minimal real-world fine-tuning suggests a scalable paradigm for building general-purpose robotic agents.
Practical Deployment: The framework supports diverse hardware setups and handles challenging real-world conditions (occlusions, transparency) better than previous point-cloud or image-based approaches.

Limitations: The system currently operates at a lower control frequency (5 Hz) compared to image-based baselines due to the computational cost of depth estimation and VLM processing. It also relies on the robustness of underlying VLMs, which can fail in highly cluttered or ambiguous scenes.