💻 computer science

Real-Time Multimodal Activity-Aware Error Detection in Robot-Assisted Surgery

This paper proposes a unified, activity-aware multimodal framework that integrates video, kinematics, and descriptive textual prompts to significantly improve real-time error detection in robot-assisted surgery, achieving up to 16.6% F1 score improvements over state-of-the-art baselines.

Original authors: Seyed Hamid Reza Roodabeh, Zongyu Li, Homa Alemzadeh

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Seyed Hamid Reza Roodabeh, Zongyu Li, Homa Alemzadeh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot surgeon as a highly skilled, but sometimes clumsy, assistant performing a delicate task like sewing a wound. The goal of this paper is to build a "smart supervisor" that watches this robot in real-time and instantly shouts, "Wait, you're doing it wrong!" before a mistake hurts the patient.

Here is how the researchers built this supervisor, explained through simple analogies:

The Problem: The "Blind Spot" of Current Systems

Previous systems tried to catch errors by just watching the video (like a security camera) or just looking at the robot's movement logs (like a fitness tracker).

The Video Problem: It's like trying to understand a complex play just by watching the actors move across the stage. You see that they moved, but you don't know why or if they were holding the right prop.
The Movement Problem: It's like judging a chef's knife skills only by the speed of their hand, ignoring whether they are actually cutting the onion or the table.

The researchers found that these old methods missed the "fine print"—the tiny, specific details of what the robot was actually doing with its tools and whether it was making a specific type of mistake.

The Solution: A "Multilingual" Detective

The team created a new system that acts like a detective who speaks three languages at once: Sight (Video), Motion (Kinematics), and Description (Text).

1. The "Script" (Activity Prompting)

Instead of just feeding the computer raw video, the researchers gave it a "script" or a set of descriptions. Think of this as giving the detective a cheat sheet that says:

"The surgeon is holding a needle."
"The surgeon is trying to pull the thread."
"The surgeon is dropping the needle."

They tested different versions of this script. They found that the best "cheat sheet" wasn't just listing the action (e.g., "sewing"), but combining the action with the specific error (e.g., "sewing, but the needle slipped"). It's the difference between a security guard saying, "Someone is walking," versus "Someone is walking but tripping over a rug."

2. The "Feeling" (Kinematics)

The system also listens to the robot's "muscle memory." It reads the exact speed, position, and angle of the robot's arms.

The Analogy: Imagine watching a dancer on TV (Video). Now imagine you can also feel the tension in their muscles and the exact force of their steps (Kinematics). Sometimes, a dancer looks like they are doing a perfect leap, but their muscles are straining in a way that suggests they are about to fall. The robot's "muscle data" helps the system catch errors that the camera might miss.

3. The "Smart Glasses" (Visual Embeddings)

The team also tried teaching the computer to "see" the activity directly, like giving the detective special glasses that automatically highlight "needle holding" or "thread pulling" in the video.

The Surprise: They found that the "Script" (Text Prompts) worked just as well, or even better, than the "Smart Glasses." This is huge because writing a script is much easier and cheaper than training a computer to recognize every single tiny movement from scratch.

The Results: Catching More Mistakes

When they tested this new "Multilingual Detective" on two different surgical datasets (one simulated in a lab, one from real surgeries), it worked significantly better than the current best methods:

On the Lab Data: It improved error detection by about 5%.
On the Real Surgery Data: It improved detection by a massive 16.6%.

It's like upgrading a smoke detector that only beeps when the fire is huge, to one that smells the smoke the moment a candle flickers dangerously.

Speed and Reality

The system is fast enough to run in real-time. It processes a 2-second window of surgery in about 36 milliseconds.

The Analogy: It's fast enough to be the referee in a live game, blowing the whistle instantly, rather than reviewing the tape hours later.

The Catch (Limitations)

The paper notes a few things that hold this back right now:

The "Muscle" Data is Rare: The system works best when it has both the video and the robot's movement data. But most surgical robots don't share that movement data publicly, so this "superpower" is hard to use everywhere yet.
Manual Scripting: The "Scripts" (prompts) had to be carefully written by humans. The system relies on these human descriptions to work well.
Specific Tasks: They tested this on sewing and needle-passing tasks. We don't know yet if it works as well on more chaotic or different types of surgeries.

In summary: The paper shows that if you teach a computer to "read" the story of what the robot is doing (using text descriptions) while watching it move, you can catch surgical errors much faster and more accurately than just watching the video alone.

Problem Statement

Robot-Assisted Minimally Invasive Surgery (RAMIS) enhances surgical precision but introduces complexity that necessitates robust technical error detection to ensure patient safety. While existing automated skill assessment methods utilize deep learning on video and kinematic data, they often fail to capture fine-grained contextual descriptions of activities and error types within the hierarchical structure of surgical procedures. Current executional error detection methods typically rely on predefined gesture labels or overlook complementary multimodal information, such as instrument-object interactions and specific error definitions. Furthermore, many recent approaches are limited to offline processing or fail to fully leverage kinematic data alongside visual and textual modalities for real-time inference.

Methodology

The authors propose a unified framework for real-time executional error detection that integrates three modalities: video (RGB), robot kinematics, and descriptive textual prompts. The framework is designed as a binary classification task to detect errors ( $y=1$ ) or normal execution ( $y=0$ ) within a sliding time window.

Core Components:

Activity Prompting: The framework introduces "Activity Prompting," which encodes multi-level surgical context into textual descriptions. These prompts cover:
- Gestures: High-level surgical actions (e.g., "picking-up the needle").
- Instrument-Object Interactions: Low-level context states describing which grasper holds or contacts specific objects (e.g., needle, thread, tissue).
- Error Definitions: Explicit descriptions of failure modes (e.g., "needle drop," "out of view").
- Combined Prompts: Combinations such as "Gesture + Error" or "Interaction + Error" to capture specific failure contexts.
  These prompts are processed via a CLIP text encoder to generate activity embeddings.
Activity-Aware Visual Embeddings: As an alternative to text prompts, the authors explore extracting features directly from vision encoders (ResNet50) pretrained on surgical activity labels (gestures or interaction states). This generates per-frame embeddings that inherently contain activity-level information.
Multimodal Fusion Architecture:
- Visual Feature Extraction: Video frames are processed through a ResNet50 backbone.
- Kinematics Fusion: Kinematic data (position, velocity, grasper angle) is projected into the same feature space as video embeddings and concatenated.
- Contextualization: A transformer-based cross-attention mechanism fuses the spatial features (video/kinematics) with the activity information (either text prompts or pretrained visual embeddings).
- Temporal Modeling: A Temporal Convolutional Network (TCN) summarizes the spatial input over the window, which is then fused with the refined activity embeddings to produce the final error probability.

The framework evaluates three specific setups:

(a) Activity Prompting (Img+Txt): Uses CLIP text embeddings for activity context.
(b) Activity Kinematics Fusion (Img+Txt+Kin): Adds kinematic data to the text-prompted setup.
(c) Activity-Aware Visual Embeddings (Img+Label): Replaces text prompts with embeddings from a vision encoder pretrained on activity labels.

Key Contributions

Unified Multimodal Framework: The paper presents a system that seamlessly integrates video, kinematics, and curated textual prompts for real-time error detection.
Activity Prompting and Embeddings: The introduction of activity prompting to integrate gesture, interaction, and error descriptions, alongside a comparative study of these prompts against activity-aware visual embeddings derived from pretrained models.
Kinematic Integration: Demonstrating the benefit of fusing kinematic data with visual and textual modalities to capture fine-grained motion deviations.
Performance Gains: The framework achieves significant improvements over state-of-the-art (SOTA) baselines on two major datasets.

Experimental Results

The methods were evaluated on the JIGSAWS (dry-lab) and SAR-RARP50 (in-vivo) datasets using Leave-One-Supertrial-Out (LOSO) and standard train/test splits, respectively.

JIGSAWS Dataset:
- The Activity Kinematics Fusion setup with Gesture+Error prompts achieved the highest F1 score of 0.760.
- This represents a 5% improvement over the SOTA CoG baseline and a 5% improvement over the ResNet-only baseline.
- Adding kinematic data to the best text-prompted model (Gesture+Error) improved the F1 score from 0.745 to 0.760.
- Activity Prompting (Gesture+Error) outperformed Activity-Aware Visual Embeddings, suggesting that curated text prompts can match or exceed the performance of models requiring dense frame-level activity pretraining.
SAR-RARP50 Dataset:
- The Interaction+Error prompting strategy achieved the best F1 score of 0.746.
- This corresponds to a 16.6% improvement over the SOTA SEDMamba baseline and a significant gain over the ResNet baseline (0.590).
- The results highlight that complex prompts blending fine-grained behavioral intents with error conditions are highly effective.
Efficiency:
- The proposed framework demonstrates real-time applicability. The best-performing model (Activity Kinematics Fusion) has an end-to-end latency of approximately 36 ms, enabling near-30 Hz execution. This is significantly faster than the SEDMamba baseline (368 ms) and competitive with CoG, despite CoG's lower per-frame latency.

Significance and Claims

The paper claims that combining curated textual prompts with multimodal data (video and kinematics) is a highly effective strategy for accurate surgical error detection. Key insights include:

Prompt Specificity: The quality and semantic content of prompts (specifically combining activity context with explicit error descriptions) are more critical than the quantity of prompts.
Multimodal Context: Incorporating multi-level activity information (gestures, interactions, kinematics) provides a rich contextual framework that outperforms unimodal or single-level approaches.
Training Efficiency: The study suggests that strong error detection performance does not necessarily require costly, time-intensive activity labeling and separate encoder pretraining. Textual prompts grounded in domain knowledge can achieve comparable or superior results to visual embeddings derived from pretrained models, simplifying the training pipeline.
Real-Time Feasibility: The framework addresses the gap in online applicability, offering a practical trade-off between detection granularity, precision, and real-time efficiency, unlike some SOTA models that are restricted to offline processing.

The authors acknowledge limitations, including the reliance on datasets with synchronized video and kinematics (limiting broader applicability until more such datasets are available), the dependence on manual curation of textual prompts, and the need for further validation on less structured surgical procedures beyond suturing and needle passing.