Technical Summary: Multimodal AI for Automated Assessment of Surgical Performance

Problem Statement

Accurate and objective assessment of surgical skill is critical for patient safety, educational improvement, and scalable evaluation. Current methodologies rely heavily on subjective human ratings (e.g., OSATS, Global Rating Scale), which suffer from inter-rater variability, limited scalability, and high resource costs. While recent AI-driven approaches have utilized deep learning for automated assessment, many existing methods are limited by their reliance on a single data modality (either video or kinematics) or their dependence on specialized sensor telemetry available primarily in robotic-assisted settings. This restricts their applicability in real-world clinical environments where comprehensive sensor data may be unavailable, and where variability in camera motion and lighting is high. Furthermore, existing state-of-the-art models often depend on complex, specialized architectures or specific rubric-informed training that may not generalize well across diverse datasets and assessment settings.

Methodology

The authors propose a novel, modular multimodal deep learning framework designed to integrate complementary signals for surgical skill assessment. The system is adaptable to varying data availability, capable of operating with rich multimodal inputs (video, kinematics, events, time) or deriving motion representations directly from video when kinematic sensors are absent.

Feature Extraction Pipeline

The framework extracts and fuses four primary feature channels:

1. Spatio-Temporal Video Features (V): Extracted using a SlowFast-R50 backbone pretrained on Kinetics-400. This dual-pathway architecture captures both spatial context (slow pathway) and rapid temporal dynamics (fast pathway) from surgical videos.
2. Kinematic Motion Features (K):
   • For JIGSAWS: Utilizes synchronized 2D Cartesian coordinates of tool tips from robotic arms.
   • For Biotissue (Video-only kinematics): Tools (needle drivers) are segmented using YOLOv8 (fine-tuned with SAM2-assisted annotations for wrist, shaft, and jaw parts) and tracked using BYTE Tracker. Centroid trajectories are smoothed via a low-pass Butterworth filter.
   • Encoding: Trajectories are converted into 1D line plot images (x and y coordinates over time) and encoded using a Trajectory-to-Image Autoencoder. This autoencoder learns a compact, noise-resilient 256-dimensional latent representation. The framework also incorporates handcrafted kinematic statistics (RMS velocity, acceleration, jerk, and total path length) where applicable.
3. Semantic Density Features (D): Heatmap density plots generated from tool-tip motion trajectories are processed using ResNet-50 to derive semantic features representing spatial motion distribution.
4. Event and Time Features (E, P):
   • Event Sequences (E): For datasets with gesture annotations (JIGSAWS), sequences are one-hot encoded to capture procedural flow and transitions.
   • Task Completion Time (P): Included as an auxiliary variable where available.

Fusion and Prediction

All extracted features are concatenated, standardized via z-score normalization, and reduced using Principal Component Analysis (PCA) (retaining 99% variance) to create a compact multimodal representation. This reduced feature vector is fed into a 1D Convolutional Neural Network (1D-CNN) regression head to predict expert-assigned skill scores (Global Rating Scale for JIGSAWS, OSATS for Biotissue). The 1D-CNN architecture consists of three Conv1D layers (64, 128, 256 filters), max pooling, a dense layer (1024 units), dropout, and a final regression output.

Evaluation Protocols

The framework was rigorously evaluated on two benchmarks:

• JIGSAWS Dataset: 103 videos (Suturing, Needle Passing, Knot Tying) with synchronized kinematics and event annotations. Evaluated using 4-fold cross-validation and Leave-One-User-Out (LOUO) splits.
• Biotissue Dataset: 205 videos (Gastrojejunostomy, Pancreatojejunostomy, Hepaticojejunostomy) representing high-fidelity simulation with variable camera motion and no direct sensor kinematics. Evaluated using 5-fold cross-validation.

Key Contributions

1. Unified Multimodal Pipeline: A comprehensive framework that unifies spatio-temporal video features, kinematic motion features (derived from both sensors and video), semantic density features, and event sequences. It is designed to function flexibly across settings with varying modality availability.
2. Video-Derived Kinematics: The ability to derive robust motion representations directly from standard 2D endoscopic video using YOLOv8 tracking and autoencoder encoding, enabling assessment in settings without direct robotic telemetry.
3. Robust Benchmarking: Extensive validation on both controlled simulation (JIGSAWS) and high-variability training environments (Biotissue), demonstrating generalizability across different procedural tasks and data constraints.
4. Ablation and Architecture Analysis: Systematic evaluation of different input combinations and sequence encoders (comparing the proposed autoencoder against BiLSTM, TCN, and Transformer baselines), identifying the optimal configuration for skill prediction.

Results

JIGSAWS Dataset

• 4-Fold Cross-Validation: The multimodal model (Video + Kinematics + Events) achieved a Spearman correlation coefficient (SCC) of 0.85 (average), outperforming other multimodal approaches like MultiPath (0.81) and achieving comparable results to video-only state-of-the-art models like CoRe + GART (0.85). The model achieved the lowest Mean Absolute Error (MAE) of 2.44 among compared methods.
• Leave-One-User-Out (LOUO): The video-only configuration achieved the highest average SCC of 0.87, outperforming all previously published video-only methods (e.g., ViSA at 0.81, Contra-Sformer at 0.68). The kinematic-only (K+E) model achieved an SCC of 0.83, with exceptional performance on the knot-tying task (0.94).
• Ablation: The combination of Video, Kinematics, and Events (V+K+E) proved most robust. Adding task completion time (P) or density features (D) did not consistently improve performance over V+K+E and, in some cases (adding D), slightly degraded performance due to coordinate noise.

Biotissue Dataset

• Gastrojejunostomy (GJ): The fusion of Kinematics and Video (K+V) yielded the best performance with an SCC of 0.67 and MAE of 2.71, significantly outperforming kinematic-only (0.47) and video-only (0.57) baselines.
• Pancreatojejunostomy (PJ): The K+V model achieved an SCC of 0.69 and MAE of 2.05, again outperforming single-modality approaches.
• Encoder Comparison: The proposed Trajectory-to-Image Autoencoder consistently outperformed direct sequence encoders (BiLSTM, TCN, Transformer) in the best-performing K+V configuration, suggesting that compact trajectory-image representations provide more stable summaries of global motion patterns for this task.

Significance and Claims

The paper claims that the proposed framework establishes a new standard for multimodal surgical skill assessment by demonstrating that integrating complementary signals (video, kinematics, events) yields robust, state-of-the-art performance. Key claims include:

• Scalability and Adaptability: The framework is effective in both sensor-rich (JIGSAWS) and sensor-limited (Biotissue) environments, proving that high-quality assessment can be achieved by deriving motion cues from standard video when telemetry is unavailable.
• Generalization: The model demonstrates strong generalization to unseen surgeons, particularly when utilizing video features, achieving an LOUO SCC of 0.87 on JIGSAWS, which surpasses all previously reported video-only methods.
• Practical Utility: By reducing reliance on complex, rubric-specific modeling or specialized sensor inputs, the approach offers a more streamlined and broadly applicable alternative for automated assessment in research and clinical contexts.
• Modest Limitations: The authors acknowledge that while the Biotissue dataset represents a high-fidelity simulation, it does not yet establish generalizability to real intraoperative cases. They also note that performance in the Biotissue setting is limited by camera motion and the lack of depth information, and that the inclusion of density features (D) can introduce noise in 2D representations. The study emphasizes that task completion time (P) is not a reliable proxy for skill on its own and should be viewed as an auxiliary variable.

The work concludes that thoughtful integration of accessible data streams and multimodal deep learning can deliver expert-level skill assessment with high accuracy and practical flexibility, bridging the gap between simulation-based training and real-world clinical evaluation.