💻 computer science

ExpOS: Explainable Open-Surgery Skills Assessment Using 3D Hand Reconstruction

ExpOS is an explainable, data-driven framework that utilizes 3D hand reconstruction and temporal attention mechanisms to automatically assess open-surgery skills and provide actionable feedback, achieving strong correlation with expert ratings without relying on predefined metrics.

Original authors: Roi Papo, Idan Smoller, Shlomi Laufer

Published 2026-05-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Roi Papo, Idan Smoller, Shlomi Laufer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are learning to play the piano. In the old days, you needed a master teacher to sit next to you, listen to every note, and tell you, "That was too fast," or "Your finger placement was sloppy." This is how surgical training works today: an expert watches a student and gives a grade. But there aren't enough expert teachers for everyone, and they can't be everywhere at once.

The paper "ExpOS" introduces a digital "smart coach" that can watch a student surgeon, grade their performance, and—most importantly—explain exactly why they got that grade.

Here is how it works, broken down into simple concepts:

1. The "3D X-Ray" Vision

First, the system needs to see what's happening. It doesn't just watch a flat video; it uses a special camera setup to build a 3D skeleton of the surgeon's hands and the tools they are holding (like forceps or needles).

The Analogy: Think of it like a video game character model that perfectly mimics the real person's hand movements in 3D space, tracking every joint and tool tip.

2. The "Smart Highlight Reel" (Temporal Attention)

When a human watches a surgery, they know to pay attention to the tricky parts (like tying a knot) and ignore the boring parts (like waiting for a tool). The ExpOS system learns to do the same thing.

The Analogy: Imagine a movie editor who watches a 10-minute surgery video and automatically highlights the 30 seconds that actually matter. The system uses a "spotlight" (called attention) to say, "This specific moment where the hand moved slowly was crucial for a good score," while ignoring the rest. It tells the student: "You messed up right here."

3. The "Statistical Report Card" (Global Features)

While the "spotlight" looks at specific moments, the system also calculates big-picture statistics for the whole video.

The Analogy: This is like a fitness tracker for surgeons. It doesn't just look at one step; it counts the total steps, the average speed, and how much the person stopped moving. It calculates things like:
- Stillness: Did the surgeon hold the tool steady when they needed to?
- Speed: Was the movement too jerky or just right?
- Time: How long did the whole task take?
- Coordination: Did the left hand and right hand move in sync?

4. The "Double-Check" Explanation

The magic of ExpOS is that it combines these two views to give a grade and a reason.

The Analogy: Imagine a teacher grading a test. Instead of just writing "85/100," the teacher says: "You got an 85. Here is why: You finished the test quickly (Global Stat), but you hesitated on question 4 (Temporal Highlight), and your handwriting was shaky (Global Stat)."
The system uses a mathematical method (SHAP) to figure out which of these "fitness tracker" stats mattered most for the final score.

What Did They Test?

The researchers tested this on 221 videos of medical students performing three specific tasks:

Suturing (stitching).
Knot Tying.
Fascial Closure (closing a deep layer of tissue).

They compared the computer's grades against the grades given by real human experts.

The Results

It works: The computer's grades matched the human experts' grades very well, especially for the "Fascial Closure" task (where the computer got about 78% of the way to the human expert's accuracy).
It explains: For the "Fascial Closure" task, the system correctly identified that finishing faster and holding the forceps still were the biggest signs of a skilled surgeon. For "Knot Tying," it found that moving the right hand smoothly was key.

The Bottom Line

ExpOS is a tool that turns a video of a surgery into a clear, data-driven report card. It doesn't just say "Good job" or "Bad job." It acts like a patient, 24/7 coach that can point out exactly when a mistake happened and what specific movement caused it, allowing students to practice on their own and improve without needing a human expert standing over their shoulder.

Note: The paper focuses strictly on analyzing these specific training videos and providing feedback. It does not claim to perform surgery, replace doctors in the operating room, or be used for real-life patient care at this stage.

Technical Summary: ExpOS – Explainable Open-Surgery Skills Assessment Using 3D Hand Reconstruction

Problem Statement

Effective surgical training requires timely and transparent feedback, yet current assessment methods rely heavily on expert observation using structured rating systems (e.g., OSATS). These traditional approaches are resource-intensive, subjective, and limit opportunities for autonomous practice. While recent deep learning methods can predict overall skill levels from motion or video data, they typically function as "black boxes," providing global scores without explaining which temporal segments or behavioral patterns drive the prediction. Conversely, methods relying on expert-defined metrics offer interpretability but constrain discovery to pre-defined criteria, limiting data-driven insights. The paper addresses the need for a scalable, objective, and explainable framework that can automatically assess surgical skills and provide actionable, feedback-oriented evaluation.

Methodology

The authors propose ExpOS, an end-to-end framework that combines weakly-supervised temporal importance learning with interpretable motion statistics. The pipeline processes video data of open-surgery tasks through three primary stages:

1. Data Extraction and Feature Engineering

Input: 4K RGB video recordings (30 fps) of residents performing three tasks: suturing, knot-tying, and fascial closure.
Detection & Reconstruction:
- A YOLO model detects surgical tools (forceps, needle driver, scissors, simulator) and hands.
- RoHans pseudo-label self-training refines the hand detector, which is integrated into WiLoR for 3D hand reconstruction. This tracks 21 joints per hand.
Frame-Level Features: Each frame is encoded using bounding box information (6 objects: left/right hands, 3 tools, simulator) and reconstructed 3D hand keypoints (2 joints × 21 points). These are concatenated into a per-frame tensor. Short gaps are interpolated, and trajectories are smoothed.
Global-Level Features: Video-level statistics are computed based on temporal changes in bounding box centers, including:
- Still time (proportion of frames where objects are stationary).
- Average velocity and acceleration (during non-idle periods).
- Hands-out-of-frame ratio.
- Task completion time.
- Autocorrelation between hands.

2. The ExpOS Architecture

The framework consists of three main components:

Temporal Backbone (MS-TCN++): The per-frame features are fed into an MS-TCN++ backbone (with the classification head removed) to encode spatio-temporal dynamics. This preserves the temporal dimension, outputting a sequence of hidden representations ( $F \times T$ ).
Temporal Attention Pooling: A multi-head self-attention mechanism is applied to the hidden sequence. This module assigns importance weights to specific temporal segments, compressing the sequence into a compact representation while generating a frame-level importance map. This allows the model to identify when critical performance moments occur.
Fusion and Prediction: The temporally pooled representation is concatenated with the pre-computed global statistical features. This fused vector is passed through a Multi-Layer Perceptron (MLP) to output a final regression-based skill score. Global features are injected only at this final stage to preserve their interpretability.

3. Explainability Mechanisms

ExpOS provides multi-level explainability:

Temporal Importance: Inspired by NLP attention mechanisms, the attention weights from the pooling layer highlight specific video frames (temporal segments) that most influenced the skill prediction.
Global Feature Attribution: The authors use SHAP (Shapley Additive exPlanations) values to quantify the contribution of each global motion statistic (e.g., task time, acceleration) to the final prediction. This identifies which motion characteristics drive the score.

Key Results

The framework was trained and evaluated on 221 videos from 77 surgical residents performing three distinct tasks. A separate model was trained for each simulator using Soft Ordinal Regression (SORD) loss.

Performance Correlation: The framework achieved strong correlations with expert ratings across all tasks:
- Fascial Closure: Highest performance with Pearson correlation $r = 0.778$ and $R^2 = 0.74$ .
- Knot Tying: $r = 0.678$ , $R^2 = 0.649$ , with the lowest RMSE (1.276).
- Suturing: $r = 0.622$ , $R^2 = 0.579$ , with the lowest MAE (0.833).
Interpretability Findings:
- Fascial Closure: Task completion time was the dominant feature (shorter times associated with higher scores). Positive contributions came from higher proportions of still forceps/hands and hand autocorrelation; negative contributions came from higher average acceleration.
- Knot Tying: Higher average speed and lower acceleration of the right hand were associated with higher scores.
- Suturing: Similar to fascial closure, with still forceps usage and shorter completion times being the most influential positive factors.

Significance and Claims

The paper claims that ExpOS represents a shift from expert-defined evaluation criteria toward model-driven importance estimation. Its primary significance lies in its ability to:

Learn Discriminative Patterns: Instead of relying on pre-defined metrics, the model learns temporal patterns directly from data to identify what constitutes skill.
Provide Multi-Level Feedback: By combining attention-based temporal localization with SHAP-based global feature analysis, the system offers structured feedback on when performance deviations occur and which motion behaviors contribute to the score.
Enable Scalable Autonomous Training: The framework demonstrates that data-driven temporal modeling of 3D hand motion can capture clinically meaningful performance patterns, offering a scalable alternative to resource-intensive expert supervision.

The authors acknowledge limitations, noting that the approach depends on reliable 3D hand reconstruction and clear visibility, and that occlusions or challenging viewpoints may degrade performance. They suggest future work will focus on multimodal integration and validation on larger, more diverse cohorts.