💻 computer science

Reasoning Text-to-Video Retrieval for Operating Room Clips via Action-Driven Digital Twins

The paper introduces OR3, a text-to-video retrieval framework for operating room clips that leverages action-driven digital twins and LLM-based imagination to perform reasoning over implicit safety-critical queries, significantly outperforming existing methods on a new benchmark of robotic knee procedures.

Original authors: Yiqing Shen, Hao Ding, Mathias Unberath

Published 2026-06-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yiqing Shen, Hao Ding, Mathias Unberath

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a massive library that contains thousands of hours of video footage from operating rooms. Most of these videos look almost identical: the same bright lights, the same white coats, the same robotic arms moving in the background.

Now, imagine a surgeon walks in and asks for a very specific clip. They don't say, "Show me the clip where the robot arm moves left." Instead, they ask for something that requires thinking, like: "Show me the step right before the surgeon clips the artery," or "Find the moment that caused the bleeding."

This is the problem the paper tackles. Existing computer systems are like librarians who only look at the cover of a book. They see the "white coat" and "robot arm" and think, "Oh, that looks like the surgery video!" But they can't understand the story or the sequence of events. They fail to find the specific moment the surgeon is looking for because they can't reason about what happened before or after a specific action.

The authors propose a new system called OR3 (Operating Room Reasoning Retrieval) to solve this. Here is how it works, using simple analogies:

1. The "Action-Driven Digital Twin" (The Recipe Card)

Instead of treating a video clip as a blurry moving picture, OR3 turns every clip into a structured "Action-Driven Digital Twin" (ActDT).

Think of a video clip as a long, messy recipe. The ActDT is like rewriting that recipe into a clean, step-by-step list of ingredients and actions, organized by time.

Old way: "Here is a video of a surgery."
OR3 way: "From 0:00 to 0:10, the Surgeon (Subject) used a Scalpel (Object) to Cut (Action). From 0:10 to 0:20, the Nurse (Subject) handed a Gauze (Object) to the Surgeon."

By breaking the video down into these specific "Subject-Action-Object" triplets, the system can tell the difference between two clips that look identical but have different actions happening at different times.

2. "Imagination-Based Retrieval" (The Mental Movie)

Usually, computers try to match a text question directly against a video file. This is like trying to match a written description of a dream directly against a photograph of a house; they are different formats, so the match is often poor.

OR3 does something clever called Imagination-Based Retrieval.

When you ask, "Show me the step before clipping," the system doesn't just look for the word "clipping."
Instead, it uses a powerful AI (a Large Language Model) to imagine what that specific moment would look like in its "recipe card" format. It creates a hypothetical ActDT based on your question.
Now, instead of matching Text vs. Video, it is matching Recipe Card vs. Recipe Card. Since both are now in the same language (structured text), the computer can find the perfect match much more easily.

3. "Evidence-Grounded Refinement" (The Detective's Second Look)

Sometimes, the AI's first "imagination" of the answer isn't quite right because it doesn't know the specific habits of that particular surgery team.

So, OR3 plays a game of "Detective":

It finds the top few clips that look like a match.
It looks at the "recipe cards" of those top clips to see what actually happened.
It compares its original "imagination" with the reality of those clips.
If there's a difference (e.g., the AI thought the nurse handed the tool before the cut, but the top clips show the nurse handed it after), the AI rewrites its own question to be more accurate.
It searches again with this refined, smarter question.

The Results

The researchers tested this on a dataset of robotic knee surgeries. They created a "test" with 276 tricky questions that required reasoning (like "What happened right before the robot calibrated?").

Old methods (the librarians who only look at covers) got the right answer less than 16% of the time.
OR3 got the right answer 57.6% of the time for the top result, and 77.3% of the time if you look at the top 5 results.

Why This Matters (According to the Paper)

The paper claims that OR3 is the first system that can truly "reason" about these videos. It doesn't just see pixels; it understands the flow of actions. It can distinguish between two visually identical moments simply because the order of events or the specific interaction between the surgeon and the tools was different.

In short, OR3 turns a chaotic library of surgical videos into an organized, searchable storybook where you can find the exact page of the story you need, even if you only remember the plot point, not the picture.

Technical Summary: Reasoning Text-to-Video Retrieval for Operating Room Clips via Action-Driven Digital Twins

Problem Statement

Operating rooms (ORs) generate thousands of hours of video footage, yet clinical stakeholders (e.g., surgical trainees, quality teams) typically require short, specific clips lasting seconds to minutes. While clip-level text-to-video retrieval is a natural formulation for accessing these segments, existing methods face two critical limitations in the OR context:

Inability to Handle Implicit Queries: Unlike conventional retrieval where queries explicitly describe visual content, OR queries are often implicit and require reasoning (e.g., "the step right before clipping" or "the action that caused bleeding"). Current methods relying on global embeddings cannot reason over such queries.
Lack of Fine-Grained Discrimination: Clips from the same surgical procedure often appear visually near-identical, differing only in specific actions and entity interactions. Existing object-centric digital twin (DT) approaches focus on per-frame entities and visual attributes, failing to distinguish between consecutive clips that share identical objects but differ in actions and state transitions.

Methodology: OR3

The authors propose OR3 (Operating Room Reasoning Retrieval), a framework designed to handle implicit queries through temporal action reasoning. The methodology consists of three core components:

1. Action-Driven Digital Twins (ActDTs)

Instead of object-centric representations, OR3 converts each video clip into an Action-Driven Digital Twin (ActDT).

Structure: An ActDT is a sequence of action primitives serialized as structured JSON. It groups concurrent subject-action-object triplets under non-overlapping temporal intervals.
Components:
- Intervals: Defined by start and end timestamps, covering the entire clip duration without overlap.
- Triplets: Each interaction includes a subject ( $s$ ), action ( $v$ ), and object ( $o$ ).
- Attributes: The subject and object include category (from an OR-specific vocabulary), attribute (natural language summary of properties like posture or instrument state), and location (bounding box and depth).
Generation: Vision Foundation Models (Qwen3-VL, SAM-3, DepthAnythingV3) extract these triplets from RGB frames and segmentation masks.

2. Imagination-Based Retrieval

OR3 reformulates the retrieval task from cross-modal matching to intra-modal matching to bridge the semantic gap between abstract queries and concrete video representations.

Hypothetical Generation: Given an implicit query $q$ and procedural metadata $K$ , a Large Language Model (LLM) generates a hypothetical ActDT ( $\widehat{\text{ActDT}}_q$ ). The LLM predicts the action primitives the target clip would contain, rather than rephrasing the query.
Intra-Modal Encoding: Both the hypothetical ActDT and the actual clip ActDTs are serialized as JSON text and encoded by a single shared text encoder ( $f_\theta$ ). Relevance is computed via cosine similarity.
Training Strategy: The encoder is trained with a contrastive objective using two types of hard negatives tailored to ActDTs:
- Temporally Adjacent Negatives: Clips sharing entities but differing in actions.
- Temporal-Permutation Negatives: Clips where the interval order is shuffled, disrupting temporal logic while preserving action primitives.
- Field-Dropout Augmentation: Randomly masking fields in actual ActDTs to make the encoder robust to the missing visual attributes present in hypothetical ActDTs.

3. Evidence-Grounded Refinement

To address the limitation that LLMs may not capture specific procedure conventions, OR3 employs an iterative refinement loop.

Process: After an initial ranking, the top- $K$ candidates (with their actual ActDTs) are fed back to the LLM along with the original query and the initial hypothetical ActDT.
Revision: The LLM identifies discrepancies between the imagined and observed action primitives and generates a revised hypothetical ActDT.
Iteration: This process repeats for up to $T_{ref}$ rounds or until the top-ranked clip remains unchanged, allowing the system to adapt query interpretation to procedure-specific patterns.

Benchmark Construction

The authors constructed a new benchmark for OR text-to-video reasoning retrieval using the MM-OR dataset:

Data: 386 clips derived from 17 robotic knee procedure recordings, segmented by surgical phases and steps.
Queries: 276 implicit queries across four reasoning categories:
- Temporal: (e.g., "the step right before bone sawing")
- Causal: (e.g., "the action that triggered a sterility breach")
- Procedural: (e.g., "the first use of the drill after robot calibration")
- Role-based: (e.g., "what the nurse prepared while the surgeon was sawing")
Split: Training (162 queries), Validation (48 queries), and Test (66 queries).

Experimental Results

Evaluated on the benchmark, OR3 significantly outperforms existing baselines, including embedding-based methods (CLIP4Clip, X-CLIP), video-LLMs (InternVideo2, Qwen3-VL), and the previous state-of-the-art reasoning method (ReasonT2V).

Overall Performance: OR3 achieves 57.6% R@1 and 77.3% R@5, surpassing the strongest baseline (ReasonT2V) by 33.4 percentage points in R@1.
Category Breakdown:
- Role-based: Highest performance (67.6% R@1), as agent-action pairs align naturally with ActDT triplets.
- Temporal & Procedural: Showed the largest absolute gains over ReasonT2V (36.9 and 35.3 points, respectively), demonstrating the efficacy of interval-structured action primitives.
- Causal: Most challenging category, yet OR3 achieved 42.1% R@1 compared to 15.8% for ReasonT2V.
Ablation Studies:
- Replacing ActDT with object-centric DTs caused the largest drop (15.2% overall), confirming the necessity of action-driven representation.
- Removing imagination-based retrieval reduced R@1 by 12.1%, validating the intra-modal matching approach.
- Evidence-grounded refinement contributed a 7.6% overall gain, with the most significant impact on causal queries.

Significance and Claims

The paper claims that OR3 enables fine-grained discrimination between visually similar OR video clips by shifting the focus from static object attributes to temporal action reasoning.

Key contributions include:

Task Formulation: Introducing text-to-video reasoning retrieval for OR clips, a task requiring the discrimination of near-identical visuals via implicit queries.
Methodological Innovation: Proposing ActDTs to encode temporal object interaction dynamics and imagination-based retrieval to eliminate cross-modal semantic gaps.
Adaptive Refinement: Introducing evidence-grounded refinement to adapt query interpretation to specific procedural conventions.
Benchmark: Providing a curated dataset of 276 implicit queries over 386 clips to facilitate future research in this domain.

The authors conclude that their approach successfully addresses the gap between how users search for OR clips and how existing retrieval methods operate, though they note future work could explore other procedure types, multi-turn queries, and real-time retrieval during live procedures.