💻 computer science

Training LLMs with Reinforcement Learning over Digital Twin Representations for Reasoning-Intensive Surgical VideoQA

This paper proposes a reinforcement learning framework that trains large language models to decouple perception from reasoning by operating on hierarchical digital twin representations with uncertainty estimates, achieving state-of-the-art performance on surgical VideoQA benchmarks through a novel plausibility-aware reward and the introduction of the REAL-Colon-Reason dataset.

Original authors: Yiqing Shen, Han Zhang, Mathias Unberath

Published 2026-06-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yiqing Shen, Han Zhang, Mathias Unberath

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but impatient student (an AI) how to answer complex questions about a live surgery video.

The Problem: The "Snap-and-Go" Trap
Currently, most AI systems try to answer these questions by taking a quick snapshot of the video, turning it into a list of simple keywords (like "scalpel," "blood," "move left"), and then immediately guessing the answer. The authors argue this is like trying to understand a movie by looking at a single frame and guessing the plot. It breaks the continuous flow of time and space. If the AI needs to figure out why a tool is moving or what will happen next, this "snap-and-go" method fails because it loses the connection between the steps.

The Solution: The "Digital Twin" Workshop
The authors propose a new way to train the AI using a concept called a Digital Twin. Think of this not as a video file, but as a detailed, interactive 3D model or a "virtual replica" of the surgery that exists alongside the video.

Instead of forcing the AI to look at the raw video and think at the same time, they split the job into two distinct teams:

The Observers (Foundation Models): These are specialized AI tools that act like expert surgeons' eyes. They watch the video and build the "Digital Twin." They don't just say "there is a tool"; they say, "There is a 'Kerrison' tool at the center, moving with 95% confidence, and it is 2 millimeters deep." They also note their own uncertainty (e.g., "I'm only 60% sure about this tissue type").
The Reasoners (The Large Language Model): This is the main AI student. It doesn't look at the video directly. Instead, it looks at the Digital Twin report built by the Observers. It uses this structured, high-quality data to plan its answer, step-by-step, like a detective solving a mystery.

The Training Method: Reinforcement Learning with a "Clinical Coach"
How do they teach the AI to do this well? They use Reinforcement Learning, which is like a video game where the AI gets points for good moves and loses points for bad ones.

The Structure: The AI is forced to follow a strict script: Think about the question -> Plan how to build the Digital Twin -> Read the Twin's data -> Think about the answer -> Give the final answer.
The Reward System: The AI gets a score based on two things:
1. Did it follow the rules? (Did it use the correct format?)
2. Is the answer medically plausible? A "Clinical Coach" (another AI) checks if the answer makes sense. If the AI says, "The surgeon is cutting the heart with a spoon," it gets a huge penalty, even if the grammar is perfect. If the answer is logically sound and matches the data, it gets a high score.
3. Uncertainty Check: If the AI is very confident but the "Observers" were unsure about the data, the AI gets a lower score. This teaches the AI to be humble and accurate rather than just confident.

The Test: The "REAL-Colon-Reason" Benchmark
To prove this works, the authors created a new test called REAL-Colon-Reason. It's a collection of 2,000 questions about colonoscopy videos, ranging from easy (what tool is this?) to very hard (predict the next step based on the current motion and tool function).

The Results
The new method crushed the competition.

It outperformed existing state-of-the-art models by a significant margin (about 17% better on the hardest questions).
It proved that by separating the "seeing" (building the Digital Twin) from the "thinking" (reasoning over the Twin), the AI can handle complex, multi-step logic that previous models couldn't figure out.
It also worked well on older, existing surgery video tests, showing it's a versatile improvement.

In a Nutshell
Instead of asking an AI to stare at a blurry, fast-moving video and guess the answer, this paper teaches the AI to first build a clear, structured, and honest "virtual report" of what is happening, and then use that report to think through the problem logically. It's the difference between guessing a movie's ending based on a blurry screenshot versus reading a detailed script summary before writing your own conclusion.

Technical Summary: Training LLMs with Reinforcement Learning over Digital Twin Representations for Reasoning-Intensive Surgical VideoQA

Problem Statement

Surgical Video Question Answering (VideoQA) aims to interpret natural language queries about surgical videos to support workflow analysis and education. However, current approaches face significant limitations in handling reasoning-intensive tasks that require multi-step semantic, spatial, and temporal inference.

Existing methods typically treat VideoQA as a direct visual matching task or compress continuous video into discrete token representations. This architectural approach fragments continuous spatial-temporal relationships, restricting the model's ability to perform complex reasoning such as:

Semantic Reasoning: Understanding instrument functions and procedural logic.
Spatial Reasoning: Analyzing geometric and depth relationships.
Temporal Reasoning: Tracing motion trajectories and workflow phases.

Furthermore, general Vision-Language Models (VLMs) require costly domain-specific adaptation, while surgery-specific models often bind visual perception and reasoning within a single model. This coupling forces the model to simultaneously learn fine-grained feature extraction and high-level reasoning, introducing competing optimization objectives that necessitate large-scale annotated datasets.

Methodology

The authors propose a Reinforcement Learning (RL) framework that decouples visual perception from reasoning. Instead of processing raw video directly, a Large Language Model (LLM) plans and operates over structured intermediate representations called Digital Twin (DT) representations.

1. Structured Rollout Sequence

The LLM generates a structured output sequence to solve a query $Q$ given a video $V$ :

Initial Reasoning ( $R_0$ ): The LLM generates a chain-of-thought to decompose the query and identify required information.
DT Construction Plan ( $G$ ): The LLM outputs a Directed Acyclic Graph (DAG) specifying which surgical foundation models to invoke and their execution dependencies.
DT Representation ( $D$ ): Upon plan execution, the system appends the structured DT representation containing extracted entities, relations, and uncertainty estimates.
DT-Based Reasoning ( $R_1$ ): The LLM reasons over the constructed DT to formulate the final answer.
Final Answer ( $A$ ): The LLM generates the response.

2. Hierarchical and Probabilistic Digital Twin Representations

The DT representation is constructed by invoking specific foundation models (e.g., SurgSAM-2, DepthAnything2, RASO, OWLv2) based on the LLM's plan. Key features include:

Hierarchical Structure: To accommodate multi-scale temporal dynamics, the DT is organized into three levels:
- Frame Level: Instantaneous observations (masks, depth, tags, bounding boxes).
- Temporal Window Level: A sliding window aggregating recent frames to preserve short-term dynamics (trajectories, deformations) using tracking identifiers.
- Procedure Level: Aggregated statistics summarizing long-term context (phase transitions, cumulative usage) via an external LLM.
Probabilistic Uncertainty: To handle perceptual ambiguity, entities and relations are formulated with probabilistic attributes. Uncertainty estimates (e.g., segmentation confidence, semantic recognition confidence, depth deviation) are derived from the foundation models' inherent outputs.

3. Reward Design

The framework employs a rule-based reward function $R(Y)$ combining format validation and accuracy assessment:

Format Reward ( $R_{format}$ ): Ensures the rollout sequence contains all required token pairs in the correct order.
Accuracy Reward ( $R_{acc}$ ):
- Evaluates semantic equivalence between the generated answer and ground truth using an LLM-as-a-judge.
- Clinical Plausibility Check: Penalizes answers containing clinically implausible claims (e.g., contradictory instrument functions or unsafe sequences).
- Uncertainty-Aware Calibration: The reward magnitude is modulated by a confidence factor $\gamma$ , calculated as the average reliability of the specific instances referenced in the answer. This scales the reward based on the reliability of the visual observations.
Training: The LLM is trained using Group Relative Policy Optimization (GRPO).

4. Dataset: REAL-Colon-Reason

To support this research, the authors introduced REAL-Colon-Reason, a colonoscopic benchmark containing 2,000 question-answer pairs. The dataset is derived from the REAL-Colon dataset and features three complexity levels:

Level 1: Basic single-step inference.
Level 2: Requires two intermediate reasoning operations.
Level 3: Demands chains exceeding three steps of logical deduction.

Key Contributions

RL Framework for Decoupled Reasoning: A novel framework that trains LLMs to plan and reason over Digital Twin representations constructed by surgical foundation models, effectively separating perception from reasoning.
Hierarchical Probabilistic DT: The design of DT representations that encode multi-scale temporal structures and probabilistic uncertainty estimates to address the inherent ambiguities and temporal hierarchies of surgical videos.
Clinical Validity Reward: A reward mechanism that enforces clinical plausibility and calibrates answer confidence against visual uncertainty, eliminating the need for manually annotated reasoning chains.
REAL-Colon-Reason Benchmark: A new dataset specifically designed for reasoning-intensive surgical VideoQA, covering varying complexity levels.

Experimental Results

The method was evaluated on REAL-Colon-Reason, REAL-Colon-VQA, and EndoVis18-VQA.

REAL-Colon-Reason: The proposed method achieved state-of-the-art performance with an overall Exact Match (EM) score of 0.584 and a SMILE score of 0.646. This outperformed the second-best method (Qwen3-VL-8B) by 17.2% in EM and 17.3% in SMILE. The performance gap widened significantly on higher complexity levels (Level 3), demonstrating the method's superior capability in multi-step reasoning.
Existing Benchmarks: On REAL-Colon-VQA and EndoVis18-VQA, the method achieved the best BLEU-4 scores (75.42% and 87.20% respectively for in-template) and significantly outperformed existing methods in keyword accuracy, indicating improved factual grounding.
Ablation Study: Removing the DT representation reduced performance to a baseline of 0.242 EM. The full RL-trained model (0.584 EM) significantly outperformed a supervised fine-tuning baseline (0.325 EM), confirming the efficacy of the reward design and RL training.

Significance and Claims

The paper claims that by decoupling visual perception from reasoning and utilizing structured Digital Twin representations, the proposed framework preserves the fine-grained spatial-temporal information that token-based compression methods fragment. This approach enables LLMs to perform complex, multi-step reasoning required for advanced surgical applications, such as anticipating adverse events or providing guidance in ambiguous intraoperative situations, without relying on massive amounts of manually annotated reasoning data. The results suggest that explicit reasoning structures over probabilistic, hierarchical representations are essential for handling reasoning-intensive surgical scenarios where general-purpose VLMs and single-frame approaches fail.