MiDAS: A Multimodal Data Acquisition System and Dataset for Robot-Assisted Minimally Invasive Surgery

This paper introduces MiDAS, an open-source, platform-agnostic system that enables non-invasive, time-synchronized multimodal data acquisition for robot-assisted minimally invasive surgery, validated by demonstrating that its external sensing approach achieves gesture recognition performance comparable to proprietary telemetry while releasing the first annotated dataset for hernia repair suturing.

The Gist

Imagine you are trying to teach a robot how to be a master chef. To do this, you need to record exactly what the chef does: how they move their hands, how hard they press the stove knobs, and what the food looks like on the plate.

In the world of robotic surgery, scientists want to do the same thing. They want to record how surgeons move their hands, press foot pedals, and operate the robot to train AI systems that can help surgeons learn or even spot mistakes.

The Problem:
Most surgical robots (like the famous da Vinci system) are like expensive, locked safes. The companies that make them keep the internal "black box" data (the exact electrical signals telling the robot how to move) secret. Researchers can't easily get this data without special, expensive permissions or hardware. It's like trying to learn how a car engine works by only looking at the outside of the car, without ever being allowed to open the hood.

The Solution: MiDAS
The authors of this paper built a tool called MiDAS (Multimodal Data Acquisition System). Think of MiDAS as a "Spy Kit" or a "Universal Translator" that sits outside the robot.

Instead of trying to break into the robot's locked safe, MiDAS uses clever, non-invasive sensors to watch what the surgeon is doing from the outside and translate it into data the computer can understand.

How MiDAS Works (The "Spy Kit" Components)

1. The "Magic Gloves" (Electromagnetic Tracking):

   • What it is: Tiny sensors placed on the surgeon's hand controls (the joysticks they hold).
   • The Analogy: Imagine the surgeon is wearing invisible, high-tech gloves that know exactly where their fingers are in 3D space. Even if the robot's internal computer is silent, these sensors shout, "Left thumb is moving up! Right finger is closing!"
   • Why it's cool: It captures the surgeon's intent perfectly without needing to touch the robot's internal code.

2. The "Eagle Eye" (3D Cameras):

   • What it is: A special camera sitting above the surgeon's console.
   • The Analogy: This is like a security camera that doesn't just see a flat picture; it sees depth. It tracks the surgeon's hands in 3D, creating a digital skeleton of their movements. It's like having a ghostly 3D model of the surgeon's hands floating in the air, mirroring their real movements.

3. The "Footprint Sensor" (Pedal Sensors):

   • What it is: Thin, pressure-sensitive pads stuck onto the surgeon's foot pedals.
   • The Analogy: Think of these as "pressure-sensitive doormats" for the foot pedals. They detect exactly when and how hard the surgeon presses their foot to activate tools or switch cameras. It's like knowing exactly when a pianist hits a key, just by feeling the vibration of the floor.

4. The "High-Def Recorder" (Video):

   • What it is: A system that records the 3D video feed the surgeon sees through the robot's eyes.
   • The Analogy: This is the "dashcam" of the surgery, recording exactly what the surgeon sees on their screen.

The Great Experiment

The team tested MiDAS in two very different kitchens:

1. The Practice Kitchen (Raven-II): An open-source, research robot where they could peek inside the hood to check if their "spy kit" was accurate.
2. The Real Kitchen (da Vinci Xi): A real, commercial hospital robot used by actual surgeons during a training bootcamp. They had surgeons practice repairing hernias on realistic fake tissue (KindHeart models).

The Results:

• Accuracy: The "spy kit" data was shockingly close to the robot's internal data. The external sensors could predict the robot's movements with high accuracy.
• The "Footprint" Success: The foot pedal sensors were great at knowing when the surgeon was pressing the buttons.
• The "Magic Gloves" vs. The Camera: The electromagnetic sensors (Magic Gloves) were much better at tracking movements than the camera alone. Why? Because cameras get confused if the surgeon's hands block the view (occlusion), but the "Magic Gloves" work even if the hands are hidden.

Why This Matters

Before MiDAS, if you wanted to study how surgeons move, you needed a very expensive, specific robot and special permission. Now, with MiDAS:

• It's Open Source: Anyone can build it.
• It's Universal: It works on any robot, old or new, because it doesn't care what's inside the robot; it just watches the outside.
• It's Safe: It doesn't change the robot or the surgery. It's like putting a sticker on a car rather than rewiring the engine.

The Bottom Line:
MiDAS is like giving researchers a universal remote control for surgical data. It allows them to collect high-quality data from any robot, anywhere, to teach AI how to understand surgery. This could lead to better training for surgeons, smarter robots that can help during operations, and safer surgeries for everyone.

The authors have even released their "Spy Kit" designs and the data they collected for free, so the whole world can start building better surgical AI today.

Technical Summary

Here is a detailed technical summary of the paper "MiDAS: A Multimodal Data Acquisition System and Dataset for Robot-Assisted Minimally Invasive Surgery."

1. Problem Statement

Robot-assisted minimally invasive surgery (RMIS) research relies heavily on multimodal data (robot kinematics, video, interaction events) for skill assessment, error detection, and training. However, significant barriers exist:

• Proprietary Lock-in: Access to internal robot telemetry (kinematics, pedal states) is restricted by proprietary systems (e.g., da Vinci), limiting cross-platform research.
• Data Scarcity & Heterogeneity: Existing large-scale datasets are often vision-only or limited to "dry-lab" tasks. Multimodal datasets involving clinical tasks (like suturing) are rare and usually tied to specific, hard-to-access research platforms (e.g., dVRK).
• Lack of Non-Invasive Alternatives: There is no affordable, open-source framework that can capture synchronized multimodal data across different robotic platforms without modifying the robot or requiring internal API access.

2. Methodology: The MiDAS System

The authors propose MiDAS, an open-source, platform-agnostic, non-invasive data acquisition system. It captures time-synchronized multimodal streams without interfering with the robotic platform or the surgeon's workflow.

System Architecture

• Client-Server Model: A central server coordinates heterogeneous data streams from multiple clients using Python multiprocessing.
• Synchronization: All data streams are stamped with Unix epoch timestamps at the server level. Video frames (from OBS) serve as the reference clock, with other modalities aligned causally to the nearest video frame.
• Cost: The estimated hardware cost is approximately $8,000 USD, making it accessible for research labs.

Sensing Modalities

MiDAS integrates four distinct sensing layers:

1. 3D Electromagnetic Hand Tracking (EmHT):
   • Hardware: NDI trakSTAR system with four miniature 6-DoF sensors.
   • Placement: Sensors are mounted on the surgeon's thumb and middle-finger contact pads of the Master Tool Manipulators (MTMs).
   • Function: Captures continuous 3D position and orientation at 270 Hz.
   • Processing: Raw sensor data is mapped to robot reference frames (MTM/PSM) via a calibrated rigid-body transformation, augmented by a Multi-Layer Perceptron (MLP) to learn residual errors and noise.
2. RGB-D Hand Tracking (HandKP):
   • Hardware: ZED Mini stereo camera mounted above the console.
   • Function: Captures 720p color video and dense depth maps at 30 Hz.
   • Processing: Uses MediaPipe to extract 3D hand keypoints (thumb/index), which are transformed into the MTM frame using an MLP to approximate end-effector motion.
3. Pedal Sensing System (PSS):
   • Hardware: Custom open-source module using Arduino microcontrollers and thin-film Force-Sensitive Resistors (FSRs).
   • Function: Sensors are affixed directly to the pedal surfaces to detect binary states (pressed/not pressed) and analog force traces at 30 Hz without modifying the console hardware.
4. HD Stereoscopic Video:
   • Hardware: Blackmagic SDI recorders (da Vinci) or ZED cameras (Raven-II).
   • Function: Captures synchronized stereoscopic surgical video at 30 Hz via Open Broadcaster Software (OBS).

3. Key Contributions

1. MiDAS System: A fully open-source, non-invasive framework for collecting synchronized multimodal data (kinematics, video, pedals) across different robotic platforms.
2. Alternative Modalities: Demonstrates that external sensing (EmHT and HandKP) can effectively approximate internal robot kinematics, serving as a viable substitute when proprietary data is unavailable.
3. New Datasets:
   • Raven-II Dataset: 15 trials of "Peg Transfer" with full ground-truth kinematics.
   • da Vinci Xi Dataset: 17 trials of hernia repair suturing (inguinal and ventral) performed on high-fidelity KindHeart porcine tissue models. This is the first multimodal dataset capturing realistic wet-lab suturing with annotated gestures.
4. Gesture Taxonomy: A structured taxonomy of 8 robotic suturing gestures (e.g., Orient Needle, Push Needle, Pull Suture, Square Knot) developed in collaboration with expert surgeons.
5. Public Release: The system code, annotated datasets, and baseline benchmarks are publicly available.

4. Results

Data Validation (Correlation Analysis)

The system was validated on the Raven-II platform by comparing external signals against ground-truth internal kinematics:

• EmHT vs. Robot Kinematics:
  • Position: Achieved high correlation (Cosine Similarity > 0.80 for MTM and PSM across X, Y, Z axes).
  • Orientation: Good alignment for Yaw (0.90), moderate for Pitch/Roll.
  • Grasper Angle: Moderate agreement (IoU ≈ 0.53), sufficient for open/close detection but not precise angle measurement.
• HandKP vs. Robot Kinematics:
  • Achieved robust positional similarity (CoS > 0.70) but suffered from occlusions and field-of-view limitations, leading to lower performance than EmHT.
• Pedal Sensing:
  • Achieved high recall (0.95 for Raven-II, 0.86 for da Vinci) and F1 scores of 0.85 and 0.78 respectively, with detection latencies under 170ms.

Downstream Task: Gesture Recognition

The authors benchmarked state-of-the-art temporal models (MTRSAP and MS-TCN++) on gesture recognition tasks:

• Peg Transfer (Raven-II):
  • EmHT alone achieved performance comparable to ground-truth kinematics (F1: 0.86 vs. 0.87 for MTM/PSM).
  • HandKP performed poorly (F1: 0.38) due to detection gaps.
  • Vision-only models underperformed significantly due to limited data (15 trials).
• Suturing (da Vinci Xi):
  • EmHT performed competitively with vision-only models (F1: 0.60 vs. 0.67).
  • Multimodal Fusion (EmHT + Image): Achieved the best results (F1: 0.70, Acc: 0.71), demonstrating that motion dynamics and visual context are complementary.
  • HandKP + Image: Showed limited improvement over vision alone.

5. Significance and Impact

• Democratization of Research: MiDAS removes the barrier of proprietary telemetry, enabling researchers to collect high-fidelity multimodal data on commercial robots (like da Vinci) and open-source platforms alike.
• Platform Agnosticism: The system works across different robots (Raven-II, da Vinci Xi, Hugo, etc.) without requiring hardware modifications or internal software access.
• Clinical Relevance: The inclusion of hernia repair suturing on realistic tissue models bridges the gap between abstract dry-lab tasks and complex clinical workflows, providing a unique testbed for training surgical AI.
• Non-Invasive Proxy: The study proves that external motion sensing (specifically EmHT) is a robust surrogate for internal robot kinematics, facilitating activity recognition and skill assessment even when internal data is inaccessible.
• Community Resource: By releasing the system and the first multimodal wet-lab suturing dataset, the authors provide a foundation for reproducible, cross-platform research in surgical robotics and multimodal learning.