World Simulation with Video Foundation Models for Physical AI

The paper introduces Cosmos-Predict2.5 and Cosmos-Transfer2.5, a unified family of world foundation models that leverage flow-based architectures and reinforcement learning to deliver high-fidelity, instruction-aligned video generation and world translation for advancing physical AI, robotics, and embodied intelligence.

The Gist

Imagine you are trying to teach a robot how to make a sandwich, drive a car, or walk through a busy city. If you let the robot learn by actually doing these things in the real world, it might break the toaster, crash the car, or trip over its own feet. It's expensive, dangerous, and painfully slow.

This paper introduces NVIDIA's Cosmos-Predict2.5, a new kind of "digital twin" for the physical world. Think of it not just as a video generator, but as a hyper-realistic video game engine that runs on pure imagination.

Here is the breakdown of how it works and why it matters, using simple analogies:

1. The Core Idea: The "Dream Machine"

Previous AI models were like a student who had only read a textbook about driving but never sat in a car. They could talk about traffic lights but couldn't predict what would happen if a ball rolled into the street.

Cosmos-Predict2.5 is like a student who has watched 200 million hours of real-world video. It has seen every type of weather, every kind of robot arm movement, and every car crash. It has learned the "physics" of how the world works. Now, you can ask it: "Show me what happens if a robot tries to pick up a slippery apple in a rainy kitchen," and it will generate a video of that exact scenario, complete with realistic reflections, splashes, and gravity.

2. The Secret Sauce: How It Learned

The team didn't just dump random videos into the computer. They built a giant, automated filter factory:

• The Filter: Imagine a sieve that catches only the best, most interesting clips. They threw out blurry videos, cartoons, and things that looked fake. They kept only the "good stuff" (like real robots moving real objects).
• The Teacher (Cosmos-Reason1): They added a smart "teacher" AI that reads the text prompts. If you say "The robot drops the cup," the teacher makes sure the AI understands exactly what "dropping" looks like, rather than just making the cup disappear.
• The Practice (Reinforcement Learning): After the AI generated a video, they had it "grade" itself against human preferences. If the video looked weird or the physics were wrong, the AI got a "bad grade" and had to try again. This is like a student taking practice tests until they ace the final exam.

3. The Two Main Tools

The paper introduces two main tools in this new family:

A. Cosmos-Predict2.5 (The Simulator)

• What it does: It creates new worlds from scratch based on your instructions.
• The Analogy: It's like a director who can film any movie you can imagine, but the movie is a simulation of reality. You can say, "Film a robot walking on Mars," and it generates the video.
• Why it's better: It's faster, sharper, and understands instructions much better than the previous version. It can handle 2 billion or 14 billion "brain cells" (parameters), making it incredibly smart.

B. Cosmos-Transfer2.5 (The Translator)

• What it does: It takes a rough sketch or a simple video and turns it into a photorealistic masterpiece.
• The Analogy: Imagine you have a crayon drawing of a street. This tool acts like a magical artist who takes that crayon drawing and paints a hyper-realistic, 4K video of that street, adding cars, people, and shadows, all while keeping the exact layout you drew.
• The Magic: It's 3.5 times smaller than the old version but produces higher quality. It can even make long videos without the picture getting blurry or the story making less sense (a problem called "error accumulation").

4. Why Does This Matter? (The Real-World Impact)

This isn't just about making cool videos; it's about training robots safely.

• For Robots: Instead of breaking 1,000 real robot arms to learn how to fold laundry, engineers can train the robot in this "Cosmos" simulator. The robot makes all its mistakes in the digital world, learns the perfect moves, and then steps into the real world ready to work.
• For Self-Driving Cars: You can't easily test a self-driving car in a blizzard in a desert. With Cosmos, you can generate a blizzard in a desert instantly. You can test the car's AI against millions of "what-if" scenarios (e.g., "What if a deer jumps out?") in seconds.
• For Data: It creates "fake" data that looks so real it's indistinguishable from the real thing. This helps train other AIs without needing to film millions of hours of real footage.

5. The Big Picture

NVIDIA is releasing the source code and the models for free (open source).

Think of this as NVIDIA handing out the blueprints and the raw materials for a super-powerful simulation engine to the whole world. They want researchers, students, and companies to build their own "digital worlds" to solve real-world problems.

In a nutshell:
Cosmos-Predict2.5 is a time machine and a parallel universe generator combined. It lets us fast-forward the training of robots and self-driving cars by letting them live a thousand lifetimes in a computer before they ever touch the real world. It's the ultimate safety net and the ultimate practice ground for the future of physical AI.

Technical Summary

1. Problem Statement

Training Physical AI systems (embodied agents like robots and autonomous vehicles) in the real world is often slow, costly, and dangerous, particularly during early development stages where system imperfections can cause damage. While world simulators offer a safe alternative, existing generative video models often lack:

• Physical Plausibility: They struggle to simulate realistic physics, object permanence, and long-horizon temporal consistency.
• Fine-grained Control: They lack precise control over camera views, specific actions, and spatial constraints required for robotics and driving.
• Domain Adaptation: General-purpose video models are not optimized for the specific data distributions of robotics, driving, or industrial environments.
• Scalability: Previous iterations (e.g., Cosmos-Predict1) were large, computationally expensive, and required separate models for different tasks (Text2World, Image2World, Video2World).

2. Methodology

The paper introduces Cosmos-Predict2.5 (a unified world foundation model) and Cosmos-Transfer2.5 (a control-net framework for world translation).

A. Data Curation Pipeline

The authors processed 200 million curated video clips from over 35 million hours of raw footage.

• Filtering: A multi-stage pipeline removes low-quality data (motion artifacts, text overlays, semantic artifacts) using aesthetic scoring, motion filters, OCR, and a final high-precision pass by a Vision-Language Model (VLM). Only ~4% of raw clips survive.
• Domain Specificity: Specialized datasets were curated for Robotics (e.g., AgiBot, GR00T), Autonomous Driving (7-camera surround view), Smart Spaces, Human Dynamics, and Physics.
• Captioning: Clips are captioned using Qwen2.5-VL-7B, generating multi-length, context-aware descriptions emphasizing object motion and scene semantics.

B. Model Architecture: Cosmos-Predict2.5

• Core Architecture: Built on Flow Matching (replacing the Diffusion/EDM approach of the previous version) for smoother optimization and better sample quality.
• Unified Model: A single model handles Text2World, Image2World, and Video2World generation, eliminating the need for separate architectures.
• Backbone: Based on a Diffusion Transformer (DiT) with 3D Rotary Position Embeddings (RoPE). It removes absolute positional embeddings to improve generalization to unseen resolutions and sequence lengths.
• Text Encoder: Replaces the T5 encoder with Cosmos-Reason1, a decoder-only Vision-Language Model (VLM) specialized for Physical AI. This provides richer text grounding and finer control.
• Visual Tokenizer: Uses WAN2.1 VAE (4x8x8 compression) to encode video latents.
• Scales: Released in 2B and 14B parameter versions.

C. Training Strategy

1. Pre-training: Progressive training from 256p to 720p, starting with Text2Image and moving to Video2World tasks.
2. Supervised Fine-Tuning (SFT): Domain-specific SFT on curated Physical AI datasets (Robotics, Driving, etc.).
3. Model Merging: Techniques like Model Soup, TIES, and DARE are used to merge domain-specific models into a unified model without catastrophic forgetting.
4. Reinforcement Learning (RL): A VideoAlign reward model (VLM-based) is used for post-training to align outputs with human preferences regarding text alignment, motion quality, and visual fidelity.
5. Distillation: A hybrid forward-reverse distillation framework (rCM) allows for high-fidelity generation in just 4 inference steps.

D. Cosmos-Transfer2.5 (Control-Net Framework)

A smaller (2B parameter), 3.5× smaller than its predecessor, framework designed for Sim2Real and Real2Real translation.

• Architecture: Distributes control blocks evenly throughout the network (one every 7 blocks) rather than stacking them at the start, improving integration of conditioning signals.
• Inputs: Supports multiple control modalities: Edges, Blur, Segmentation, Depth, and World Scenario Maps (for driving).
• Capabilities: Enables camera-pose controllable multi-view generation and action-conditioned video generation.

3. Key Contributions

1. Unified World Foundation Model: The first open-source model to unify Text/Image/Video-to-World generation in a single architecture, significantly reducing complexity.
2. Physical AI Specialization: Introduction of Cosmos-Reason1 as the text encoder and specialized domain datasets (Robotics, Driving, Physics) to ensure physical plausibility and instruction following.
3. Efficient Control-Net Framework: Cosmos-Transfer2.5 achieves higher fidelity and robust long-horizon generation while being significantly smaller (2B vs 7B) than previous control-net models.
4. Open Ecosystem: Full release of source code, pretrained checkpoints (2B/14B), and curated benchmarks under the NVIDIA Open Model License.
5. Advanced Training Techniques: Demonstration of Flow Matching, Model Merging, and RL-based post-training specifically for video world models.

4. Results

Quantitative Benchmarks (PAI-Bench)

• Cosmos-Predict2.5-2B (Post-trained): Achieves an Overall Score of 0.768 (Text2World) and 0.810 (Image2World).
• Comparison: Outperforms larger models like Wan2.2-27B-A14B in Image2World tasks and matches or exceeds Wan2.1-14B despite having only ~1/7th of the parameters.
• Human Evaluation: In human preference studies, the 2B model is preferred over Wan2.2-5B, and the 14B model is preferred over Wan2.1-14B and matches Wan2.2-27B-A14B.

Application-Specific Results

• Robot Policy Learning: Using Cosmos-Transfer2.5 for data augmentation, a robot policy achieved 80% success rate (24/30) in novel, out-of-distribution scenarios (e.g., new object colors, lighting, distractors), significantly outperforming baselines using standard image augmentation.
• Autonomous Driving: The multi-view driving model shows a 2.3x improvement in FVD/FID scores and up to 60% improvement in 3D object and lane detection metrics compared to previous models.
• Long-Horizon Generation: The new model exhibits significantly less error accumulation (measured by RNDS) over long video sequences compared to Cosmos-Transfer1.
• VLA Training: The model generates high-quality synthetic data for Vision-Language-Action (VLA) training, achieving the highest instruction-following scores on the DreamGen benchmark.

5. Significance

This work represents a major leap forward in Embodied AI and Physical AI:

• Simulation-First Ecosystem: It provides a scalable, high-fidelity, and controllable simulation platform that allows agents to learn perception and control skills entirely in "silicon" before real-world deployment.
• Bridging the Sim2Real Gap: By generating physically plausible, diverse, and controllable synthetic data, it addresses the data scarcity and safety issues inherent in training real-world robots and autonomous vehicles.
• Democratization: By releasing open-source models and tools, NVIDIA lowers the barrier to entry for researchers and developers to build next-generation embodied intelligence, fostering reproducibility and innovation across the Physical AI community.
• Efficiency: The ability to achieve state-of-the-art results with smaller models (2B/14B) and faster inference (4-step distillation) makes these technologies viable for real-time deployment in resource-constrained environments.