💻 computer science

EndoGSim: Physics-Aware 4D Dynamic Endoscopic Scene Simulations via MLLM-Guided Gaussian Splatting

The paper proposes EndoGSim, a unified framework that leverages MLLM-guided 4D Gaussian Splatting and a differentiable Material Point Method to achieve high-fidelity, physics-aware reconstruction and simulation of dynamic endoscopic scenes for robot-assisted surgery.

Original authors: Changjing Liu, Yiming Huang, Long Bai, Beilei Cui, Hongliang Ren

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Changjing Liu, Yiming Huang, Long Bai, Beilei Cui, Hongliang Ren

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to perform delicate surgery inside a human body. To do this safely, the robot needs a "digital twin" of the inside of the body—a simulation that looks real and, more importantly, feels real when the robot pushes or pulls on it.

The paper introduces a new system called EndoGSim that builds this digital twin. Here is how it works, broken down into simple concepts:

1. The Problem: The "Plasticine" vs. The "Real Thing"

Current methods for creating 3D videos of surgery are great at making things look real (like a high-definition movie). However, they are terrible at making things act real.

The Analogy: Imagine a video of a jellyfish floating. A standard 3D model might look exactly like a jellyfish, but if you poke it in the simulation, it might just pass through like a ghost, or bounce like a rubber ball. It lacks the "squishiness" and "stretchiness" of real tissue.
The Gap: Existing tools can't automatically figure out that a surgical tool is hard metal while the stomach lining is soft, squishy tissue. They often guess the wrong physics, leading to simulations that look pretty but behave incorrectly.

2. The Solution: A "Smart Painter" with a Physics Brain

The authors created a framework that combines three powerful tools to fix this:

A. The Canvas: 4D Gaussian Splatting (The "Living Painting")

Instead of building a rigid 3D model, the system uses a technique called 4D Gaussian Splatting.

The Analogy: Think of a traditional 3D model as a statue made of stone. This new method is more like a cloud of millions of tiny, glowing paint droplets (Gaussians) that move and change shape over time. Because there are so many droplets, they can blend together to create a smooth, realistic image of moving tissue, and they can stretch and squish just like real organs.

B. The Brain: The Multi-Modal Large Language Model (The "Smart Assistant")

This is the paper's big innovation. Usually, to make a simulation realistic, a human expert has to manually tell the computer: "This part is steel, Young's modulus is 200," and "This part is liver, Young's modulus is 0.05." This is slow and prone to error.

The Analogy: The authors gave the computer a "Smart Assistant" (an AI like GPT-4o). They show the AI a picture of the surgery and ask, "What is this?"
How it works: The AI looks at the image, recognizes the tools and the tissue, and instantly guesses the physical properties (how hard or soft they are). It's like handing a child a picture of a rubber duck and a rock, and the child immediately knows the duck is squishy and the rock is hard, without needing a physics textbook.

C. The Refiner: The "Differentiable Material Point Method" (The "Practice Session")

The AI's first guess isn't perfect. So, the system runs a physics simulation to see if the guess holds up.

The Analogy: Imagine the AI guesses the tissue is "medium soft." The system then runs a virtual simulation where a tool pushes the tissue. It compares the result to the actual video of the surgery.
- If the virtual tissue squishes too much, the system knows the guess was wrong.
- If it doesn't squish enough, it knows the guess was too hard.
The Magic: The system automatically tweaks the numbers (the "material parameters") over and over again until the virtual movement matches the real video perfectly. It does this by looking at two things:
1. The Look: Does the rendered image match the real photo?
2. The Motion: Does the movement match the optical flow (how pixels move from one frame to the next)?

3. The Result: A "Physics-Aware" Simulation

By combining these steps, the system creates a simulation where:

Tools behave like hard metal.
Tissues behave like soft, stretchy organs.
Everything interacts realistically.

The paper tested this on real surgical videos (including some they filmed themselves using pig organs). They found that their method was much more accurate at guessing the physical properties and creating realistic movements than previous methods. Surgeons who watched the simulations rated them as the most "realistic" compared to other techniques.

Summary

In short, EndoGSim is a system that takes a video of surgery, uses an AI to guess what things are made of, and then uses a physics engine to fine-tune those guesses until the digital simulation moves exactly like the real human body. It turns a static 3D video into a dynamic, physics-accurate playground for training surgical robots.

Technical Summary: EndoGSim

Problem Statement
In robot-assisted minimally invasive surgery, high-fidelity dynamic reconstruction and simulation of endoscopic scenes are critical for spatial intelligence, pre-operative planning, and autonomous control. While recent advancements in Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS) have achieved superior visual reconstruction and real-time rendering, existing methods primarily focus on appearance. They lack physics-based descriptions of the scene, specifically the material properties of deformable tissues and instruments. This limitation hinders physically accurate spatial perception and the generation of realistic data necessary for surgical automation. Furthermore, current approaches to physical simulation often struggle with dynamic video inputs, rely on manual initialization of material parameters, or fail to distinguish between multiple objects (e.g., tools vs. tissues) with different material properties.

Methodology
The authors propose EndoGSim, a unified framework that achieves physics-aware reconstruction and physical simulation of endoscopic scenes using Multi-modal Large Language Models (MLLMs) guided by 4D Gaussian Splatting (4DGS). The pipeline consists of four main stages:

4DGS Reconstruction: The surgical scene is reconstructed from video input using 4D Gaussian Splatting. This extends static 3D Gaussian representations to the spatiotemporal domain to handle deformable tissues. The framework integrates pre-trained models for depth estimation ( $\pi3$ ) and segmentation (Surgical-SAM-2) to generate Gaussian splats representing the scene, including distinct masks for soft tissues and surgical instruments.
Continuum Mechanics & MPM: To simulate physical behavior, the system employs the Material Point Method (MPM), specifically a differentiable MLS-MPM. This solver handles continuum equations for soft tissue deformation, modeling hyperelastic properties parameterized by Young's modulus ( $E$ ) and Poisson's ratio ( $\nu$ ). The deformation is governed by mass and momentum conservation, allowing for the modeling of elastic recoil and permanent tearing.
Object-wise Material Field & MLLM Initialization: A novel object-wise material field is introduced to estimate physical properties for different objects independently.
- Initialization: Instead of manual tuning, initial material parameters ( $\theta_{init}$ ) are inferred automatically by prompting an MLLM (GPT-4o) with segmented visual inputs (images masked for tissue and tools).
- Optimization: For deformable objects, these parameters are refined through gradient-based optimization within the differentiable MPM. The optimization is jointly supervised by rendering loss (comparing rendered images to ground truth) and optical flow loss (comparing simulated motion to observed optical flow). Rigid instruments are frozen after initialization.
Simulation: The optimized material parameters are integrated into the simulation pipeline to generate realistic 4D dynamic scenes.

Key Contributions

Unified Framework: The introduction of a system that simultaneously reconstructs deformable endoscopic scenes from video and estimates material parameters to enable realistic physical simulation.
MLLM-Guided Material Estimation: A novel approach where an object-wise material field is coarsely initialized via MLLM inference and then jointly refined with rendering and optical flow losses in a differentiable MPM, eliminating the need for manual parameter initialization.
Physics-Aware 4DGS: The integration of 4D Gaussian Splatting with continuum mechanics to handle dynamic surgical scenes, addressing the gap in existing methods that struggle with dynamic video inputs and multi-object material estimation.

Results
The framework was validated on open-source datasets (EndoNeRF, CholecSeg8K) and an in-house dataset (PorcineEndo, featuring ex-vivo porcine tissues with ground-truth material parameters derived from tensile tests).

Quantitative Performance: Compared to state-of-the-art methods (PhysGaussian, Physics3D, GIC, PhysFlow), EndoGSim achieved superior performance in estimating material properties (lower Relative Error in Young's modulus and Poisson's ratio) and capturing deformation (lower End-Point Error in optical flow).
Realism: Expert ratings from 33 surgeons indicated that EndoGSim achieved the highest physical realism score (4.02 on a 5-point Likert scale) compared to baselines ranging from 3.26 to 3.77.
Ablation Studies: Experiments confirmed that MLLM initialization followed by optimization yields significantly better material consistency and visual realism than manual initialization or MLLM initialization without optimization. The object-wise material field was shown to be critical for accurate parameter assignment and physically realistic simulations.

Significance
The paper claims that EndoGSim offers substantial potential to advance robot-assisted minimally invasive surgery by providing comprehensive perception capabilities. By bridging the gap between visual reconstruction and physical simulation, the framework enables the generation of realistic data essential for surgical automation, haptic feedback training, and robust autonomous control. The ability to automatically infer and refine material properties for both tools and tissues represents a significant step toward high-fidelity digital twins of surgical environments.