🤖 AI

Sim2real Image Translation Enables Viewpoint-Robust Policies from Fixed-Camera Datasets

The paper introduces MANGO, a novel unpaired image translation method that leverages a segmentation-conditioned InfoNCE loss and specialized discriminator design to generate diverse, viewpoint-robust training data from fixed-camera real-world datasets and scalable simulations, significantly improving robot manipulation policies under camera viewpoint shifts.

Original authors: Jeremiah Coholich, Justin Wit, Robert Azarcon, Zsolt Kira

Published 2026-02-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jeremiah Coholich, Justin Wit, Robert Azarcon, Zsolt Kira

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to pick up a soda can. You show the robot a video of a human doing it, but there's a catch: the camera is always in the exact same spot.

Now, imagine you try to use that robot in a real kitchen, but the camera is now on the counter instead of the ceiling, or maybe it's on the robot's wrist. Suddenly, the robot is confused. It doesn't know where the can is because the "picture" looks totally different. It's like if you learned to drive a car only by looking at a map from directly above; if you get in the car and look out the windshield, you won't know how to steer.

This is the problem the paper MANGO solves.

The Problem: The "One-View" Trap

Most robot training data is boring. It's taken from a fixed camera angle because it's easy to set up. But robots need to be flexible. If they only learn from one angle, they break when the angle changes.

To fix this, scientists usually turn to Simulations (video games for robots). In a simulation, you can take pictures from 1,000 different angles easily. But there's a catch: Simulations look fake. They look like cartoons, while the real world looks like a photograph. If you train a robot on the "cartoon" data, it fails in the "real" world. This is called the Sim2Real Gap.

The Solution: MANGO (The "Style Translator")

The authors created a tool called MANGO. Think of MANGO as a super-smart photo editor or a digital translator.

The Input: MANGO takes a "cartoon" picture from the simulation (which has many cool, weird camera angles).
The Training: It looks at a tiny, boring pile of real photos taken from just one fixed angle.
The Magic: MANGO learns the "style" of the real world (the lighting, the texture of the table, the look of the soda can) and applies it to the cartoon pictures.

The Result: MANGO can take a simulation photo taken from a crazy new angle and make it look like a real photo taken from that same crazy new angle.

How Does It Work? (The Secret Sauce)

Usually, when you try to change a photo's style, the AI gets confused and changes the shape of the objects too. It might turn a soda can into a soda bottle just because the camera moved. MANGO avoids this with three clever tricks:

The "Segmentation Map" (The Outline): MANGO uses a special map that outlines the objects (like a coloring book outline). It forces the AI to keep the objects in the exact same shape, even while changing the lighting and texture. It's like telling the artist, "You can change the paint color, but don't move the lines."
The "Patch" Trick: Instead of judging the whole picture at once, the AI looks at tiny little squares (patches) of the image. It learns that a patch of "wood table" should look like a real wood table, even if it's in a weird spot.
The "No-False-Alarm" Rule: The AI is trained to be very strict about what counts as a "match." It prevents the robot from getting confused by repetitive patterns (like a checkered tablecloth) that look the same everywhere.

Why Is This a Big Deal?

The paper compares MANGO to other methods, including massive, super-complex AI models (like Diffusion models).

The Heavy Hitters: Other methods are like trying to move a mountain with a bulldozer. They require massive amounts of computer power (GPU hours) and take forever to run.
MANGO: MANGO is like a sneaky ninja. It's tiny, fast, and incredibly efficient. It does the job 2,700 times faster than the heavy competitors.

The Real-World Test

The researchers tested this on real robots doing tasks like stacking cups, closing laptops, and picking up soda.

Without MANGO: If they trained the robot on fixed-camera data and then moved the camera, the robot failed almost 100% of the time.
With MANGO: They used MANGO to generate thousands of "fake" training photos from different angles. When they trained the robot on these, the robot's success rate on new camera angles jumped by over 40%.

The Bottom Line

MANGO is a clever, lightweight tool that teaches robots to be flexible. It takes the easy, fake data from simulations, makes it look real, and teaches the robot to understand the world from any angle, not just the one it was trained on. It's the difference between a robot that can only drive on a straight, empty road and one that can navigate a busy, twisting city street.

1. Problem Statement

Vision-based robot manipulation policies have achieved success but remain brittle to distribution shifts, particularly changes in camera viewpoint.

Data Scarcity & Bias: Real-world robot demonstration data is scarce, expensive to collect, and often captured using fixed cameras to ensure consistency. This results in policies that fail when deployed with shifted viewpoints.
Simulation Limitations: While simulation allows for collecting diverse viewpoint data at scale, there is a significant visual sim2real gap (simulated images look different from real ones).
Existing Solutions:
- Standard Image Translation (e.g., CycleGAN, CUT): Often fail to generalize to unseen viewpoints when trained on fixed-viewpoint real data because they memorize background details rather than learning viewpoint-invariant features.
- Diffusion Models (e.g., ZeroNVS): While powerful, they are computationally expensive (hundreds of GPU-hours) and slow for augmenting large robot datasets (often >100k images).
- Domain Randomization: Often sacrifices performance for robustness and requires manual tuning.

Goal: Develop a method to translate diverse simulated viewpoints into realistic real-world images using only a small amount of fixed-camera real data, thereby enabling the training of viewpoint-robust policies.

2. Methodology: MANGO

The authors propose MANGO (Multiview Augmentation with Novel Generated Observations), a GAN-based unpaired image-to-image translation framework.

Core Architecture

Input: A large dataset of diverse simulated images ( $D_A$ ) and a small dataset of fixed-camera real images ( $D_B$ ).
Generator: A ResNet-based network (12M parameters).
Discriminator: A highly regularized 3-layer CNN (11M parameters).

Key Technical Innovations

To ensure the translation preserves the simulated viewpoint while adopting the real-world style, MANGO introduces three critical components:

Segmentation-Conditioned InfoNCE Loss (SegNCE):
- Problem: Standard contrastive losses (like PatchNCE) treat all features as unique. In robotics datasets, backgrounds (tables, walls) are repetitive, causing the model to confuse background patches with object patches.
- Solution: MANGO leverages ground-truth segmentation maps from the simulator. It clusters generator features by segmentation class.
- Mechanism: Features belonging to the same semantic class (e.g., "robot arm" or "object") are treated as positive samples for each other, while features from different classes are negative. This ensures object boundaries and semantic content are preserved during translation, even if the viewpoint changes.
Modified PatchNCE Loss ( $\tilde{L}_{PatchNCE}$ ):
- Problem: Standard PatchNCE uses a scoring function that can penalize features that are naturally similar (e.g., texture repetition) as "false negatives."
- Solution: A modified scoring function $\tilde{\rho}$ is introduced. If the cosine similarity between a query and a negative sample exceeds a threshold $\theta$ , the score is scaled down by a factor $\alpha$ ( $0 \le \alpha < 1$ ). This prevents the model from over-penalizing naturally similar regions, improving content preservation.
Highly-Regularized Discriminator:
- Problem: A standard discriminator trained on fixed-camera real data would memorize the specific background and force the generator to recreate those exact backgrounds, destroying the new viewpoint.
- Solution: The discriminator employs random patch sampling and per-patch random rotations. It only evaluates local patches rather than global image consistency. This forces the generator to learn the style (lighting, texture, material) of the real domain without being constrained to the specific global layout of the fixed-camera training set.

Training Objective

The total loss combines:

GAN Loss (Style matching).
Modified PatchNCE Loss (Content preservation).
SegNCE Loss (Semantic boundary preservation).
Identity Loss (Regularization).

3. Key Contributions

MANGO Framework: A novel sim2real translation method specifically designed to preserve unseen simulation viewpoints while translating to real-world style, using only a small fixed-camera real dataset.
Novel Loss Functions: Introduction of SegNCE (segmentation-guided contrastive loss) and a Modified PatchNCE to handle the specific challenges of repetitive textures and object-centric manipulation in robotics.
Efficiency vs. Diffusion: Demonstrates that a lightweight GAN approach is ~2,700x faster than state-of-the-art diffusion models (ZeroNVS) for dataset augmentation, making it practical for large-scale robot learning.
Empirical Validation: Proves that policies trained with MANGO-augmented data are significantly more robust to camera shifts compared to baselines, including domain randomization and diffusion-based methods.

4. Experimental Results

Image Translation Quality (Sim2Real)

Metric: Fréchet Inception Distance (FID).
Result: MANGO achieved the lowest FID scores across Fixed, Randomized, and Wrist view test sets.
- On the Randomized View test set, MANGO outperformed the next best method (Without SegNCE) by 23 FID points.
- Standard baselines like CUT and CycleGAN performed poorly, confirming that generic translation methods fail on low-diversity robotics datasets.

Policy Robustness (Sim2Sim & Real Robot)

Sim2Sim: MANGO-trained policies achieved the highest success rates on 5 out of 6 tasks (e.g., threading, hammering, stacking) compared to policies trained on fixed-camera data or VISTA (a diffusion-based baseline).
Real Robot Experiments:
- Tasks: Pick up coke, stack cups, close laptop, stack blocks.
- Performance: MANGO significantly improved success rates on shifted viewpoints.
  - Example: In the "Stack cups" task, the success rate on shifted cameras jumped from 1/30 (Fixed Camera only) to 13/30 with MANGO.
  - In "Close laptop," MANGO achieved 22/30 success on shifted views vs. 19/30 for VISTA.
- Comparison: While VISTA (a massive 4.5B parameter model) performed well, MANGO (35M parameters) was competitive and often superior, with the added benefit of being computationally feasible.

5. Significance and Conclusion

Bridging the Gap: MANGO effectively solves the "viewpoint brittleness" problem in robot learning without requiring expensive multi-view real-world data collection.
Scalability: By using a lightweight GAN architecture, MANGO makes it practical to augment massive robot datasets (hundreds of thousands of images) in a fraction of the time required by diffusion models.
Generalizability: The method is agnostic to the downstream policy (tested with ACT - Action Chunking Transformer), making it a versatile tool for the robotics community.
Limitations: MANGO still requires a small amount of real-world data for training and occasionally underperforms VISTA in specific real-world tasks, likely due to VISTA's pre-trained 3D geometric understanding. However, the authors suggest that integrating MANGO's loss functions into larger pretrained models is a promising future direction.

Conclusion: MANGO demonstrates that with the right architectural constraints (segmentation conditioning and patch regularization), simple GANs can outperform complex diffusion models for the specific domain of robot data augmentation, enabling robust policies from limited real-world data.