🤖 AI

Mirage2Matter: A Physically Grounded Gaussian World Model from Video

The paper introduces "Simulate Anything," a framework that reconstructs real-world environments from multi-view videos using 3D Gaussian Splatting and generative models to create physically grounded, editable simulations that enable Vision Language Action models to achieve strong zero-shot performance without requiring expensive sensors or real-world interaction data.

Original authors: Zhengqing Gao, Ziwen Li, Xin Wang, Jiaxin Huang, Zhenyang Ren, Mingkai Shao, Hanlue Zhang, Tianyu Huang, Yongkang Cheng, Yandong Guo, Runqi Lin, Yuanyuan Wang, Tongliang Liu, Kun Zhang, Mingming Gong

Published 2026-02-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhengqing Gao, Ziwen Li, Xin Wang, Jiaxin Huang, Zhenyang Ren, Mingkai Shao, Hanlue Zhang, Tianyu Huang, Yongkang Cheng, Yandong Guo, Runqi Lin, Yuanyuan Wang, Tongliang Liu, Kun Zhang, Mingming Gong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to pick up a banana or press a button. The old way to do this is to let the robot practice in the real world. But that's slow, expensive, and robots break things when they learn.

The new way is to let the robot practice in a video game (a simulator). But here's the problem: most video games look fake. If a robot learns in a cartoon world, it gets confused when it sees the real, messy, textured world. It's like teaching someone to drive on a flat, gray screen and then expecting them to handle a bumpy, rainy road.

Mirage2Matter is a new tool that fixes this. It builds a simulator that looks and acts exactly like the real world, but it's built entirely from simple videos you take with a regular camera.

Here is how it works, using some simple analogies:

1. The "Magic Photo" (3D Gaussian Splatting)

Usually, to make a 3D world, you need expensive lasers or special cameras. Mirage2Matter is different. It takes a bunch of regular photos or a short video of a room and an object (like a loaf of bread).

Think of this like taking a photo and then magically turning it into a 3D sculpture. The paper calls this 3D Gaussian Splatting. Imagine the room is made of millions of tiny, invisible, glowing balloons (Gaussians). When you look at them from one angle, they look like a photo. When you move your head, they rearrange themselves to look like a 3D object. This captures the exact look, texture, and lighting of the real room.

2. The "Ghost Shell" (Physics vs. Looks)

There's a catch: those "glowing balloons" look great, but they are too soft to push or grab. If a robot tries to push a balloon, it just squishes through it. The robot needs something solid to bump into.

So, the system does a second trick. It looks at the object in the video and uses an AI to build a "ghost shell" (a solid 3D mesh) around it. This shell is invisible to the eye but solid to the robot's physics engine.

The Analogy: Imagine a realistic wax figure of a person. It looks exactly like a human (the 3D Gaussian part), but inside, it has a hard skeleton (the mesh) so you can actually push it without it melting.

3. The "Ruler and Compass" (Calibration)

Even if the robot has a realistic look and a solid shell, the sizes might be wrong. The robot might think the table is 10 feet tall when it's actually 3 feet.

Mirage2Matter solves this with a calibration board. Before filming, you put a special board with known measurements on the floor. The system uses this as a "ruler" to stretch or shrink the digital world until it matches the real world's size perfectly. It also aligns the robot's "eyes" (camera) so that what the robot sees in the simulation is in the exact same spot as what it sees in reality.

4. The "Frankenstein Movie" (Hybrid Rendering)

Now, the robot needs to practice moving. The system runs the robot's movements in a physics engine (where the solid shells bump into each other correctly). Then, it takes the video of the robot moving and "stitches" it into the realistic background.

The Analogy: Imagine a movie where the background is a real, high-definition video of a kitchen, but the actor (the robot) is a CGI character. Usually, the CGI looks fake. But here, the system cuts out the real robot from the video and pastes it onto the realistic background, so the lighting and shadows match perfectly.

The Result: Zero-Shot Success

The paper tested this by training a robot entirely inside this "Magic Simulator." They never showed the robot the real world during training.

When they took the trained robot and put it in the real kitchen to pick up a banana or press a button, it worked almost as well as a robot trained by humans in the real world.

The Claim: The robot didn't need any extra "fine-tuning" or practice in the real world. It learned in the simulation and immediately knew what to do in reality.

Why is this a big deal?

No Special Gear: You don't need expensive lasers; a phone or regular camera is enough.
No Breaking Things: Robots can crash a thousand times in the simulator without breaking a single real object.
Perfect Match: Because the simulator is built from the actual room where the robot will work, there is no "surprise" when the robot enters the real world.

In short, Mirage2Matter turns a simple video into a perfect digital twin of reality, allowing robots to learn complex physical tasks quickly and safely before ever touching the real world.

Technical Summary: Mirage2Matter

Problem Statement

The scalability of embodied intelligence is fundamentally limited by the scarcity of high-quality real-world interaction data required to train Vision-Language-Action (VLA) models. While simulation offers a solution, existing approaches suffer from a significant "sim-to-real" gap.

Reconstruction-based simulators (e.g., AI2Thor, Habitat, SAPIEN) often rely on manually authored assets and synthetic textures, creating a visual domain gap. Recent 3D Gaussian Splatting (3DGS) extensions improve visual fidelity but frequently depend on expensive sensors, precise robot calibration, or depth measurements, and often lack mechanisms to capture physical laws, requiring task-specific solvers.
Generative-based world models offer automated pipelines for synthesizing 3D assets and predicting physical properties. However, they often prioritize diversity over geometric accuracy, leading to mismatches between generated assets and real-world counterparts. These discrepancies degrade environmental fidelity and limit the utility of the resulting simulation for zero-shot deployment.

There is a critical need for a world model that simultaneously achieves high-fidelity visual reconstruction from ordinary videos and physically grounded interaction dynamics without specialized hardware.

Methodology: Mirage2Matter

Mirage2Matter is a physically grounded Gaussian world model that generates high-fidelity embodied training data from multi-view videos. The framework integrates photorealistic reconstruction with physics-based simulation through three core stages:

1. Photorealistic Scene and Object Reconstruction

The system reconstructs both the environment and manipulable objects using two parallel branches:

Reconstruction Branch (3DGS): Utilizes 3D Gaussian Splatting (3DGS) to create photorealistic representations.
- Scene: A handheld video of the environment is processed using COLMAP to obtain Structure-from-Motion (SfM) points and camera poses. A scene 3DGS is trained using a photometric objective to capture global layout and fine-grained textures.
- Objects: Object masks are segmented using SAM2 (guided by text prompts) on selected frames. Object-specific 3DGS models are optimized using only masked pixels to isolate the foreground, suppressing background clutter.
Generation Branch (Meshes): To enable physical simulation, explicit surface geometry is required. Single-view photos of objects are fed into Tripo3D to generate watertight meshes in a canonical pose. These meshes serve as collision-ready geometric proxies for the physics engine.

2. Cross-Domain Alignment

To ensure consistency between the visual representation (3DGS) and the physical simulation (meshes), the system aligns all assets to a unified robot-centric coordinate frame (the Genesis simulator world frame).

Scene Alignment: A calibration board is placed at the robot base location during video capture. Scaled Iterative Closest Point (ICP) is performed between the SfM points of the calibration board and a virtual robot workspace mesh. This similarity transform ( $S = (s, R, t)$ ) is applied to the SfM points and camera poses before training the scene 3DGS, ensuring the final model is natively aligned to the simulator.
Object Alignment: The object 3DGS and its corresponding mesh are aligned post-optimization. A similarity transform is estimated between point clouds sampled from the 3DGS and the mesh, followed by ICP refinement. The Gaussian parameters (mean, scale, rotation) are updated to match the mesh geometry, enabling unified rendering and simulation.
Sim-to-Real Camera Calibration: A ChArUco board attached to the robot's end-effector is used to solve the hand-eye calibration equation. This ensures that the robot's head-mounted camera in the simulation aligns pixel-perfectly with the real-world camera, maintaining consistency in the manipulator's relative position.

3. Data Generation and Hybrid Rendering

Once aligned, the system assembles a unified world model and generates training data:

Interactive Placement: Objects are placed within the scene using interactive transforms, merging the scene 3DGS and object 3DGS into a single Gaussian world ( $G_{world}$ ).
Physics-Based Simulation: The Genesis physics engine simulates robot interactions using the object meshes. Motion planning (IK and RRT) generates collision-free trajectories.
Hybrid Rendering: To create training videos, the system composites the output of the two branches:
1. The robot's motion and interaction are rendered from the Genesis simulator (providing kinematic/dynamic correctness and collision constraints).
2. The background and static objects are rendered from the 3DGS world model (providing photorealism).
3. A robot mask is extracted from the Genesis video and used to alpha-composite the robot foreground onto the 3DGS background, producing a final frame that is both physically consistent and visually realistic.

Key Contributions

Faithful and Photorealistic World Representation: The authors reconstruct high-fidelity visual representations of environments and manipulable objects using only ordinary multi-view videos, capturing fine-grained geometry and appearance without specialized sensors.
Physically Grounded and Robot-Centric Modeling: By leveraging generative models for collision-ready meshes and introducing a calibration pipeline for accurate scale alignment, the system ensures the simulation remains consistent with the real world.
Practical and Scalable Data Generation: The approach enables the low-cost construction of editable world models with grounded interaction dynamics. It supports large-scale data synthesis, allowing VLA models trained on this data to achieve strong zero-shot generalization.

Experimental Results

The authors evaluated Mirage2Matter by training a VLA model (PoSA-VLA) on simulated data and deploying it on a real AlphaBot 1s robot without any fine-tuning (zero-shot transfer).

Baselines: The method was compared against DISCOVERSE and RoboSimGS, as well as an upper-bound Real-World training set.
Tasks: Evaluation covered three manipulation tasks: Grasp, Press Button, and Push/Pull.
Performance:
- Grasping: On real-world objects (banana, croissant), Mirage2Matter achieved success rates of 80.0% and 86.7%, respectively. This significantly outperformed DISCOVERSE (60.0%, 66.7%) and RoboSimGS (76.7%, 76.7%), approaching the Real-World training upper bound (96.7%, 90.0%).
- Task-Level: For "Press Button," Mirage2Matter achieved 93.3% success compared to 96.7% for Real-World training. For "Push/Pull," it achieved 73.3% compared to 83.3%.
Ablation Studies:
- Replacing mesh-only objects with 3DGS-reconstructed objects improved grasping success, highlighting the importance of visual consistency between objects and the environment.
- Performance scaled positively with the number of training demonstrations, with 300 demonstrations yielding optimal results.

Significance and Claims

The paper claims that Mirage2Matter provides a practical path toward scalable embodied intelligence by bridging the simulation-to-real gap. The authors attribute the improved zero-shot performance to:

Higher Visual Fidelity: The unified 3DGS rendering reduces the perceptual gap between simulation and reality, particularly in texture, lighting, and background geometry.
Visual-Physical Consistency: Unlike prior simulators that mix mesh objects with 3DGS backgrounds (creating visual discrepancies), Mirage2Matter renders both foreground and background within a unified 3DGS world. This prevents models from learning "shortcut" behaviors based on simulator-specific artifacts.
Physical Grounding: The use of Genesis-based execution ensures that the learned policies respect kinematic and collision constraints, preventing overfitting to physically impossible behaviors.

The authors conclude that their framework enables the construction of reality-consistent world models directly from real environments, supporting the training of VLA models that can execute manipulation tasks in the physical world without additional fine-tuning.