💻 computer science

NVIDIA Cosmos-H-Dreams: Real-Time Generative Physics Simulation for Surgical Robotics

The paper introduces Cosmos-H-Dreams, a real-time, controller-agnostic generative world model that leverages teacher-student distillation and Self Forcing to enable interactive, photorealistic surgical simulations at 160 FPS, bridging the gap between costly physical experiments and classical simulators for education, synthetic data generation, and decision support.

Original authors: Javier Gamazo Tejero, Lukas Zbinden, Keyur Sheth, Raghavendra K M, Nadim Daher, Diego Granero Maraña, Filip Binkiewicz, Patrick Thornycroft, Mahdi Azizian, Sean D. Huver

Published 2026-08-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Javier Gamazo Tejero, Lukas Zbinden, Keyur Sheth, Raghavendra K M, Nadim Daher, Diego Granero Maraña, Filip Binkiewicz, Patrick Thornycroft, Mahdi Azizian, Sean D. Huver

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the operating room, the margin between a successful procedure and a complication can be measured in millimeters and milliseconds. Surgeons rely on robotic systems to perform delicate tasks like stitching blood vessels or removing organs, but training for these moments has historically been a slow, expensive, and often dangerous process. To learn, a trainee might practice on animal tissue or a cadaver, resources that are finite and cannot be reused. Alternatively, they might use computer simulations, but these have long struggled to look and behave like the real thing; they often fail to capture the way soft tissue ripples, bleeds, or reacts to the heat of a surgical tool. For decades, the gap between the digital training ground and the physical operating theater has remained wide, forcing surgeons to rely heavily on intuition and limited practice hours.

A team of researchers from NVIDIA and CMR Surgical has now built a system that bridges this gap, creating a digital world where surgical robots can be trained and tested in real time. They call their creation Cosmos-H-Dreams. Unlike previous simulations that simply play back pre-recorded videos or rely on rigid physics engines that look stiff and artificial, this system generates new video frames on the fly, reacting instantly to the movements of a surgeon or a computer program. It is a generative model, a type of artificial intelligence that learns the laws of physics and the visual appearance of surgery by watching thousands of hours of real surgical footage. The result is a simulator that runs fast enough to feel like a live interaction, allowing a human to control a virtual robot with a keyboard or a headset, or letting a computer algorithm learn to tie a knot without ever touching a physical instrument.

The core of this achievement is a two-part strategy that balances the need for high-quality visuals with the need for speed. The researchers first trained a massive "teacher" model on a vast collection of surgical data from nine different types of robotic systems. This teacher learned to predict what would happen next in a surgical scene with high accuracy, but it was too slow to use for live interaction; it took too long to calculate each new frame. To solve this, they created a smaller, faster "student" model. They used a technique called distillation, where the student learns to mimic the teacher's predictions but does so in a much more efficient way. By teaching the student to generate short bursts of video frames in just two steps rather than dozens, and by optimizing the software to run on a single powerful computer chip, the team achieved a breakthrough in speed. The system can now produce a continuous stream of realistic surgical video at roughly 160 frames per second on a single workstation, fast enough to eliminate the lag that usually makes virtual reality feel disjointed.

What makes this system truly unique is its ability to accept control from any source. The researchers demonstrated that the simulator could be driven by a person typing on a keyboard, a surgeon wearing a virtual reality headset, or even a commercial surgical robot console like the Versius. It can also be controlled by an artificial intelligence policy that is learning to perform tasks on its own. In these experiments, the AI watches the video it generates, decides on a movement, and then sees the immediate visual consequence of that movement, creating a closed loop of learning. This is the first time such an interactive, real-time world model has been applied to surgery, allowing both humans and machines to act within a synthesized environment and observe the results instantly.

The researchers tested the system rigorously to see if the digital world behaved like the real one. They had the simulator run through thousands of suturing tasks and compared the outcomes to what happened when the same tasks were performed on a physical robot. The results showed a moderate overall agreement between the simulation and real-world outcomes, with a Pearson correlation of 0.696 across tasks. However, performance varied substantially depending on the specific procedure. The system showed relatively strong agreement for tasks like picking up and throwing objects, but for more complex tasks like handover and knot tying, the correlation was negative, indicating that the simulator sometimes predicted success where the real robot failed, or vice versa. The system accurately captured the physics of soft tissue and the behavior of surgical tools in most scenarios. However, the study also revealed the limits of the technology. When the simulation involved extremely fine, overlapping structures, such as a thread folding over itself during a knot-tying task, the system occasionally made mistakes, hallucinating the shape of the thread in ways that did not match reality. These errors were more common in the fast, real-time version than in the slower, more accurate teacher model, suggesting that while the system is a powerful tool for training and evaluation, it is not yet perfect for every single type of delicate manipulation.

Despite these limitations, the implications of this work are significant. By providing a safe, scalable, and photorealistic environment, Cosmos-H-Dreams offers a new foundation for surgical education. Surgeons can now practice complex procedures repeatedly without the need for cadavers or animals, and robotic policies can be trained and refined in a digital sandbox before ever being deployed in a hospital. The researchers have made the code and the model available to the public, hoping to accelerate the development of safer surgical robots and better training tools. While the system currently focuses on tabletop suturing tasks, the team envisions expanding it to cover full clinical procedures involving complex anatomy. For now, it stands as a proof of concept that the future of surgical training may not be in a physical lab, but in a real-time, generative world where mistakes are safe, and learning is endless.

Technical Summary: NVIDIA Cosmos-H-Dreams: Real-Time Generative Physics Simulation for Surgical Robotics

1. Problem Statement

Surgical robotics training and policy evaluation face significant bottlenecks. Physical experiments involving animal or cadaver labs are costly, time-consuming, and difficult to reproduce. Conversely, classical simulators struggle to capture the photorealistic appearance and complex deformable-tissue dynamics (e.g., bleeding, electrocautery smoke, specular wet surfaces) required for realistic surgical training. While World Foundation Models (WFMs) offer a data-driven alternative for synthesizing plausible futures, existing approaches often fail to jointly achieve high visual/physical fidelity, action controllability, and real-time generation speeds necessary for interactive simulation. Specifically, there is a lack of surgical world models that support live human control and closed-loop policy evaluation in real time.

2. Methodology

The authors propose Cosmos-H-Dreams, an integrated real-time surgical world-model system. The methodology follows a multi-stage pipeline involving data aggregation, model architecture design, a teacher-student distillation strategy, and a specialized inference stack.

2.1 Data Strategy

Open-H-Embodiment: A large-scale, multi-embodiment corpus aggregating 32 datasets across 9 robotic embodiments (including CMR Versius and dVRK) and approximately 22 million frames. This provides a unified 44D action space covering diverse instruments, anatomies, and lighting conditions.
dVRK Tabletop Suturing: For specific fine-tuning, the entire dVRK suturing dataset (including failure episodes) is used. This dataset comprises ~1 million frames at 10 Hz, expressed in a 20D dual-arm action format (padded to 44D).

2.2 Model Architecture

The system is built on Cosmos-Predict2.5, an action-conditioned Diffusion Transformer (DiT).

Conditioning: The model ingests a first RGB frame (encoded as clean latent tokens), a streaming KV cache for temporal context, and a chunk of robot kinematics (unified 44D action vector). Actions are injected via timestep/AdaLN modulation rather than as separate attention tokens.
Two-Stage Training:
1. Bidirectional Teacher: Initialized from the multi-embodiment Cosmos-H-Surgical-Simulator, this model is fine-tuned on specific procedure data (e.g., tabletop suturing) using a rectified-flow objective. It is trained at increasing temporal horizons (up to 73 frames) to improve long-rollout stability.
2. Causal Student (Distillation): The bidirectional teacher is distilled into a causal, autoregressive student capable of real-time streaming. This addresses the "exposure bias" problem where the model must condition on its own imperfect past generations.
  - Self Forcing: The student is trained on self-generated context via self-rollouts, bridging the train-test gap.
  - Distribution Matching Distillation (DMD): Instead of pixel-wise reconstruction, the student minimizes the distributional difference between its self-generated clips and the real data manifold using a frozen score network and a trainable fake-score network.

2.3 Inference and Serving

FlashDreams: The deployment stack utilizes the FlashDreams streaming-inference library to enable autoregressive generation on edge hardware.
Optimizations: The system employs a fixed-size streaming KV cache, local-window temporal attention with appearance sinks, lightweight latent encoders/decoders (TAEHV), and precomputed operations.
Latency Reduction: A fully GPU-resident pipeline replaces MJPEG with NVENC-based H.264 encoding, reducing downstream latency by 1.20x.
Control Interfaces: The system supports live control via WebRTC (browser keyboard), WebXR (Meta Quest headset), TCP sockets (learned policies), and direct integration with surgical robot consoles (e.g., CMR Surgical Versius).

3. Key Contributions

First Interactive Surgical World Model: The system is the first to support live control by both human operators (via VR/keyboard) and learned policies in a closed loop.
Multi-Embodiment Foundation: It introduces Cosmos-H-Surgical-Simulator, a foundation model fine-tuned across 9 robotic embodiments, providing a reusable checkpoint for downstream procedure-specific models.
Real-Time Streaming on a Single GPU: By combining Self Forcing distillation with FlashDreams, the system achieves interactive frame rates (~160 FPS) on a single NVIDIA RTX PRO 6000 Blackwell GPU, generating 12-frame blocks at 288×512 resolution.
New Evaluation Benchmark: The paper proposes evaluating surgical world models based on closed-loop action controllability and stability over long task rollouts, moving beyond static pixel metrics.

4. Results

Simulation Quality: On tabletop suturing episodes, the distilled 2-step causal student (using TAEHV) achieves a Fréchet Video Distance (FVD) of 265.36 and LPIPS of 0.1212, compared to the teacher's 170.13 and 0.0864. This quantifies the fidelity cost of real-time operation.
Rollout Stability: Progressive training with longer-horizon teachers (up to 73 frames) significantly reduces temporal drift in long autoregressive rollouts compared to short-horizon teachers.
Closed-Loop Policy Evaluation: When evaluating surgical policies (e.g., $\pi_0$ $π_{0}$ , GR00T N1) in simulation against real-world dVRK outcomes, the system shows moderate agreement (Pearson $r=0.696$ $r = 0.696$ ).
- Strong correlation was observed for "pickup" and "throw" tasks ( $r=0.67, 0.77$ ).
- Negative correlations were found for "handover" and "knot tie" tasks ( $r=-0.39, -0.24$ ), attributed to hallucinations of thin, self-interacting structures (e.g., thread geometry) in the distilled model.
Performance: The system reaches 161 FPS for 12-frame chunks at 288×512 on a single GPU. At higher resolutions (540p), inference drops to ~30 FPS, with NVENC encoding becoming less reliable and falling back to CPU.

5. Significance and Claims

The paper claims that Cosmos-H-Dreams represents a shift from offline video generators to interactive, real-time surgical simulators. Its significance lies in enabling:

Surgical Education: Providing a common foundation for training where human operators can act and observe consequences in real time.
Scalable Synthetic Data: Generating data for rare or counterfactual scenarios difficult to capture physically.
Policy Development: Offering a closed-loop environment for the training and evaluation of robotic policies.

The authors explicitly state that while the system demonstrates interactive generation and moderate sim-to-real agreement, it is not yet a substitute for offline evaluation on fine bimanual manipulation involving complex, thin structures like sutures. They identify the degradation of fidelity in such scenarios as a primary limitation and note that future work will focus on extending the pipeline to clinical procedures with richer anatomy and validating the system for intraoperative decision support. The paper positions the system as a foundational step, with Cosmos 3 identified as a natural next-generation backbone for improving fidelity while retaining throughput.