💻 computer science

SWoMo: Neuro-Symbolic World Model for Cataract Surgery Simulation

This paper introduces SWoMo, a neuro-symbolic world model for cataract surgery simulation that combines a rule-based symbolic engine for physically grounded motion dynamics with a diffusion model for realistic visual rendering, thereby achieving superior generalization and sim-to-real translation compared to existing approaches.

Original authors: Ssharvien Kumar Sivakumar, Akwele Johnson, Anirudh Dhingra, Yannik Frisch, Ghazal Ghazaei, Anirban Mukhopadhyay

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ssharvien Kumar Sivakumar, Akwele Johnson, Anirudh Dhingra, Yannik Frisch, Ghazal Ghazaei, Anirban Mukhopadhyay

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to perform delicate eye surgery, or training a new surgeon without risking a real patient. To do this, you need a "virtual reality" simulator that looks exactly like a real eye and reacts exactly like real tissue when a tool touches it.

The paper introduces SWoMo, a new kind of simulator for cataract surgery. Think of SWoMo not as a single machine, but as a collaboration between a strict physics teacher and a creative artist.

Here is how it works, broken down into simple parts:

1. The Problem: The "Uncanny Valley" of Surgery

Current simulators usually fail in one of two ways:

The Physics Experts: They get the movement right (the tool pushes the tissue, and the tissue squishes), but the video looks like a cartoon or a low-quality video game. It doesn't look real enough to train a human.
The Video Artists: They use AI to generate beautiful, realistic videos. But they can't actually control the surgery. If you tell the AI to move a tool to a specific spot, it might ignore you, or the tool might pass right through the eye like a ghost. They also can't handle situations they haven't seen before (like a tool entering at a weird angle).

2. The Solution: The "Neuro-Symbolic" Team

SWoMo solves this by splitting the job into two distinct roles that work together:

The Symbolic Brain (The Physics Teacher):
This part is a rule-based computer program. It doesn't care about how things look; it only cares about math and rules.
- Analogy: Imagine a puppet master holding strings. The puppet master knows exactly how the strings move, how the joints bend, and how the eye rotates. It creates a "digital twin" of the eye and the tools. It ensures that if you push the tool, the tissue actually moves. It creates a skeleton of the surgery.
- Key Feature: Because it follows strict rules, it can handle new situations it has never seen before (like a tool entering from a strange angle) without getting confused.
The Diffusion Model (The Creative Artist):
This is a powerful AI that is excellent at painting pictures.
- Analogy: Imagine a master painter who has watched thousands of hours of real surgery videos. They know exactly what the texture of an iris looks like, how light reflects off wet tissue, and how shadows fall.
- The Job: The "Physics Teacher" hands the "Artist" the moving skeleton (the tool positions and eye movements). The Artist then paints a hyper-realistic video over that skeleton, adding the skin, the wetness, and the lighting.

3. The Secret Sauce: "Inverse Pairing"

How do you teach the Artist to paint the Physics Teacher's work? You can't just show them random videos.

The researchers used a clever trick called Inverse Pairing:

They took real videos of actual surgeries.
They used AI to strip away the "paint" (the realistic look) and extract only the "skeleton" (the movement of the eye and tools).
They fed that skeleton into their Physics Teacher to create a perfect, rule-based simulation of that exact moment.
Now they have a pair: The original real video and the new simulated version of the same event.

They used these pairs to train the Artist. The Artist learned: "When the Physics Teacher moves the tool like THIS, the real world looks like THAT."

4. The Scene Graph: The "Instruction Manual"

To make sure the Artist doesn't get confused when the tool touches the tissue (which is the hardest part to render), SWoMo uses a Scene Graph.

Analogy: Think of this as a detailed instruction manual or a map. It tells the Artist, "This object is a tool, this is the eye, and they are touching right here."
Without this map, the AI might get confused and blur the line between the tool and the eye. With the map, the AI knows exactly where the boundaries are, even if the simulation and the real video don't align perfectly pixel-for-pixel.

5. What Did They Prove?

The paper shows that SWoMo is better than previous methods in three main ways:

It looks real: The videos are high-quality and realistic.
It follows instructions: If you move the tool in the simulation, the video shows the tool moving there. It doesn't ignore you.
It handles the unknown: Because the "Physics Teacher" part uses rules, SWoMo can simulate surgeries with tools or angles it has never seen in its training data.

Bonus Feature: Style Transfer
The system can also take the movements from a surgery video in one hospital (Dataset A) and apply them to the visual style of a different hospital (Dataset B). It's like taking the choreography of a dance from one movie and having the actors from a completely different movie perform it, looking exactly like they belong in that new movie.

Summary

SWoMo is a hybrid simulator. It uses a rule-based engine to ensure the surgery makes physical sense and a generative AI to make it look photorealistic. By combining the two, it creates a training environment that is safe, controllable, and realistic enough to help train new surgeons and build better surgical robots.

Technical Summary: SWoMo – Neuro-Symbolic World Model for Cataract Surgery Simulation

Problem Statement
Realistic surgical simulation is essential for training novice surgeons and developing autonomous surgical agents. While "world models" offer a pathway to scalable, interactive simulation by predicting future patient states based on observations and actions, current state-of-the-art approaches struggle to meet three critical criteria for clinical applicability:

Visual Realism: Simulators often lack the fidelity required to minimize the sim-to-real gap, particularly in modeling complex instrument-tissue interactions and textures.
Physically Grounded Interactions: Many methods fail to ensure that interactions adhere to physical laws, limiting the transferability of learned behaviors to real high-stakes settings.
Generalization: Existing models often degrade significantly when faced with out-of-distribution (OOD) scenarios, such as unseen tool geometries, insertion angles, or novel surgical workflows, as their behavior is tightly coupled to the training distribution.

Current methods typically excel in only one of these areas. Purely generative video models lack step-wise action conditioning and physical grounding, while traditional physics-based simulators often lack visual realism and struggle with complex deformations.

Methodology
The authors introduce SWoMo, a neuro-symbolic world model for cataract surgery that decouples motion generation from visual realism. The architecture consists of two primary components:

Symbolic Component (Motion & Dynamics):
- Inverse Pairing Strategy: The system extracts anatomical configurations ( $k_{anat}$ ) and tool kinematics ( $k_{tool}$ ) from real surgical videos. Anatomical parameters (pupil/iris ellipses, globe rotation/translation) are derived using nnU-Net and CoTracker, while tool parameters (tip position, orientation, articulation) are recovered using SASVi and SAM2.
- Rule-Based Simulator: These extracted parameters drive a rule-based simulator built on the Godot game engine. This simulator generates a digital twin of the eye and surgical tools, producing a sequence of simulated frames ( $\bar{x}$ ) and corresponding segmentations ( $\bar{m}$ ). This ensures physically grounded interactions and precise control over tool-tissue dynamics.
- Symbolic Scene Graphs: From the simulated frames and segmentations, the system constructs scene graphs ( $G$ ). These graphs encode object geometry and relational structures (e.g., centroids, spatial spreading, optical flow). The authors utilize two graph encoders (global and local) to capture structural relationships and appearance cues, which serve as conditioning signals to mitigate misalignment artifacts between simulated and real videos.
Neural Component (Visual Realism):
- Video Diffusion Model: A denoising diffusion model ( $\epsilon_\theta$ ) translates the simulated sequence ( $\bar{x}$ ) and scene graphs ( $G$ ) into realistic surgical videos ( $x$ ).
- Two-Stage Training:
  - Stage 1: The model learns image and graph-to-video generation, conditioned on the first frame and scene graphs.
  - Stage 2: A ControlNet-based approach is employed to incorporate the full simulated video sequence ( $\bar{x}_{1:n}$ ) as a conditioning signal. This allows the model to modulate intermediate features without disrupting the pretrained generative prior, ensuring high visual fidelity while adhering to the simulated motion.

Key Contributions

Decoupled Architecture: SWoMo explicitly separates the modeling of surgical motion/interaction dynamics (handled by the symbolic simulator) from visual appearance (handled by the diffusion model). This allows for precise preservation of tool and anatomical motions while learning complex textures and deformations from data.
Inverse Pairing for Training: The authors propose a novel strategy to reconstruct real surgical videos in the simulator to generate large-scale paired simulated and real video datasets. This enables the training of a video-to-video diffusion model for sim-to-real translation without requiring manual annotation of every frame.
Scene Graph Conditioning: The integration of symbolic scene graphs as conditioning inputs is shown to be crucial for maintaining object identity and reducing smoothing artifacts near tool-tissue boundaries caused by residual misalignments.
Unsupervised Style Transfer: By utilizing a shared intermediate representation (the simulated sequence) that encodes geometry and motion independently of appearance, SWoMo enables unsupervised video style transfer between different datasets (e.g., CATARACTS and Cataract-1k) without paired training data.

Results
Experiments were conducted on the CATARACTS and Cataract-1k datasets, focusing on safety-critical surgical phases.

Quantitative Performance: SWoMo outperformed multiple baselines (including StyleGAN-V, Endora, MedSora, LVDM, MOFA, and SG2VID) in terms of visual fidelity (FID, FVD, LPIPS) and conditioning adherence (Bounding Box IoU, F1 score). Specifically, it achieved an FVD of 265.4 and an F1 score of 0.412 on the CATARACTS dataset, surpassing the next best method (SG2VID) in conditioning adherence.
Ablation Studies: Removing scene graph conditioning (SWoMo-✗SG) or image conditioning (SWoMo-✗IMG) resulted in measurable performance drops, confirming the necessity of both components for high-fidelity synthesis and adherence.
Generalization: The model demonstrated the ability to generate plausible outcomes for unseen interaction geometries, including novel tool entry directions and tool combinations not present in the training data.
Downstream Utility: When used as a generative data augmentation tool for surgical phase recognition (using MS-TCN++), SWoMo improved accuracy and F1 scores compared to training on real data alone and compared to augmentation with other generative methods.

Significance
The paper claims that SWoMo addresses the trade-off between physical grounding and visual realism that has hindered the clinical applicability of surgical world models. By combining a rule-based simulator for dynamics with a diffusion model for appearance, SWoMo satisfies the three key criteria: it achieves high visual realism, ensures physically grounded interactions, and generalizes to scenarios beyond the training distribution. The authors emphasize that this approach reduces the implementation effort required for simulators (by eliminating the need to manually model complex textures) while maintaining precise control over surgical actions, thereby enabling more effective training environments and the development of autonomous surgical agents.