SurgSync: Time-Synchronized Multi-Modal Data Collection Framework and Dataset for Surgical Robotics

This paper introduces SurgSync, a time-synchronized multi-modal data collection framework implemented on a da Vinci Research Kit that addresses the lack of training data for surgical AI by integrating advanced sensors and synchronization tools to generate a validated dataset of 214 surgical instances for skill assessment and autonomous control development.

The Gist

Imagine you are trying to teach a robot to perform surgery. To do this well, the robot needs to "watch" a human surgeon and learn from them, just like a medical student watches a master surgeon. But there's a big problem: the robot is blind to the details.

Most existing data is like a blurry, out-of-sync video where the audio (the robot's movements) doesn't match the picture (what the surgeon sees). If you try to teach a robot with bad data, it learns bad habits.

This paper introduces SurgSync, a new "super-camcorder" system designed to fix this. Think of it as upgrading from a shaky, old camcorder to a high-definition, perfectly synchronized 4K studio setup, specifically for the operating room.

Here is how SurgSync works, broken down into simple concepts:

1. The "Perfect Sync" Problem (The Orchestra Analogy)

Imagine an orchestra where the violinist plays a note, but the drummer hits the drum a split second later. It sounds terrible, right? That's what happens in current surgical robots. The camera sees the tissue, but the robot's computer records its own movement a tiny bit later.

SurgSync's Solution: They built two special "conductors" (recorders) to keep everything in perfect time.

• The Live Conductor (Online Mode): This works in real-time. It forces the camera and the robot to wait for each other, like a strict conductor making sure the violin and drums hit exactly together. It's great for live demonstrations.
• The Post-Production Editor (Offline Mode): This is like recording a concert first and editing it later. It records everything at maximum speed without stopping to check the time. Afterward, a computer program lines up the video and the robot movements perfectly. This is faster and captures more data, perfect for training AI.

2. The "Eyes" Upgrade (The Glasses Analogy)

The old surgical robots used cameras that were like wearing thick, foggy glasses. The images were grainy and hard to see details.

SurgSync's Solution: They swapped the old camera for a modern "chip-on-tip" endoscope. Think of this as upgrading from foggy glasses to high-definition contact lenses. The images are now so sharp that the AI can see tiny blood vessels and tissue textures clearly. They proved this by showing that the new camera captures 30 times more detail than the old one.

3. The "Touch" Sensor (The Magic Gloves)

Robots can see, but they can't "feel" if they are touching tissue. It's like trying to thread a needle while wearing thick winter gloves.

SurgSync's Solution: They added a capacitive contact sensor. Imagine the surgical tools are wearing "magic gloves" that can sense when they brush against something.

• When the tool touches the tissue, the sensor sends a "ding!" signal to the computer.
• This gives the AI ground truth (the absolute truth) about exactly when contact happens, which is crucial for learning delicate tasks like stitching.

4. The "Magic Toolbox" (The Post-Processing Lab)

Collecting the data is only half the battle. You also need to clean it up and label it. SurgSync comes with a digital toolbox that does the heavy lifting:

• Depth Estimation: It turns 2D flat images into 3D maps, so the robot understands how deep a cut is.
• Kinematic Reprojection: This is a fancy way of saying "drawing a target." The system takes the robot's 3D arm position and projects a glowing "heat map" onto the video image, showing exactly where the tool tip is pointing. It's like a laser pointer that helps the AI focus on the action.
• Annotation: A user-friendly interface lets humans label the video (e.g., "Now the surgeon is tying a knot"), creating a textbook for the AI.

5. The "Classroom" (The User Study)

To test if this system works, the researchers didn't just use fake plastic models. They invited 13 people to perform real surgical tasks on chicken hearts, beef, and pork (which feel very similar to human tissue).

• The Students: 4 beginners, 5 experienced users, and 4 professional surgeons.
• The Tasks: They did things like moving pegs, stitching wounds, and cutting tissue.
• The Result: They collected 214 high-quality recordings.

Why Does This Matter?

The researchers used this new data to train an AI to grade surgical skills. They fed the video and robot data into a computer model, and the model successfully predicted how good a surgeon was at stitching.

The Big Picture:
SurgSync is the foundation for the future of robotic surgery. By providing a clean, perfectly synced, and rich dataset, it allows engineers to build AI that can eventually:

1. Assist surgeons by warning them if they are about to make a mistake.
2. Perform semi-autonomous tasks (like holding a camera steady or tying a knot) under supervision.
3. Train new surgeons faster by analyzing their movements against the "gold standard" data collected here.

In short, SurgSync is building the perfect library of surgical knowledge so that robots can finally learn to be safe, precise, and helpful partners in the operating room. All the software and data are now open for anyone to use, accelerating the entire field of medical robotics.

Technical Summary

1. Problem Statement

Despite the success of Robot-Assisted Surgery (RAS) systems like the da Vinci Surgical System, the development of Artificial Intelligence (AI) for higher-level control (e.g., supervised autonomy) is hindered by a critical lack of high-quality, well-annotated real-world training data. Existing datasets in the surgical robotics domain suffer from three primary limitations:

• Temporal Misalignment: Weak or inconsistent time synchronization across sensing modalities (vision, kinematics, contact), which obscures cause-and-effect relationships and degrades sequence models.
• Legacy Imaging: Outdated imaging pipelines that limit visual fidelity, hindering downstream computer vision tasks.
• Narrow Scope & Tooling: Limited task coverage and a lack of post-collection tooling for calibration and data processing, constraining reproducibility.

Furthermore, while synthetic data exists, the "sim-to-real" gap limits its effectiveness for complex procedures involving fine motor actions and tissue dynamics.

2. Methodology

The authors propose SurgSync, an open-source framework implemented on the da Vinci Research Kit (dVRK) and dVRK-Si to address these gaps. The methodology is divided into four core components:

A. Dual-Mode Synchronized Recorders

The framework implements two distinct recording modes in modern C++ (ROS-based) to handle different data collection needs:

1. Online-Matching Recorder: Enforces strict time synchronization in real-time using multi-threading. It admits only samples within a user-defined time tolerance (e.g., 10 ms), ensuring tight alignment for real-time inference applications. This results in slightly irregular time intervals but preserves natural teleoperation continuity.
2. Offline-Matching Recorder: Decouples recording from alignment to maximize throughput. It logs raw video and kinematic streams independently. A post-processing pipeline then reconstructs a fixed-rate frame sequence, interpolating kinematic data using nearest-timestamp lookups. This yields uniform intervals and higher data volume, ideal for training large-scale datasets.

B. Advanced Hardware Integration

• Modern Imaging: Integration of a Chip-on-Tip endoscope (Cornerstone Robotics) with the dVRK-Si. This replaces legacy scopes, providing 1080p video at 60 Hz with significantly higher image quality (measured by Laplacian variance).
• Contact Sensing: A novel capacitive contact sensor (Arduino-based) is interfaced with the dVRK controller's digital input. It detects tool-tissue contact on ex-vivo tissues, providing ground-truth binary labels (contact/non-contact) synchronized with the data stream.
• Multi-View Setup: Includes a side-view camera (Intel RealSense) to capture external instrument movements alongside the endoscopic view.

C. Post-Collection Processing Toolbox

A configurable toolbox is provided to standardize data preparation:

• Kinematic Reprojection: Uses hand-eye calibration and a Gaussian heatmap to project 3D tool-tip kinematics onto 2D endoscopic images, creating "attention-weighted" images to highlight regions of interest.
• Derived Modalities: Generates depth maps (via stereo matching/FoundationStereo) and optical flow (via RAFT model).
• Annotation GUI: A PyQt-based interface for labeling contact events, surgical phases, and gestures.

D. Data Collection Protocol

User studies were conducted with 13 participants (Novice, Experienced, and Professional surgeons) performing canonical tasks (Peg Transfer, Suturing, Tissue Manipulation, Dissection) on phantoms and ex-vivo tissues (chicken, beef, pork).

3. Key Contributions

1. Time-Synchronization Design Pattern: A dual-mode recorder system (online/offline) that treats time synchronization as a first-class constraint, solving the issue of inconsistent alignment across modalities.
2. Upgraded Imaging Stack: The first integration of a modern chip-on-tip endoscope with the dVRK, achieving image quality comparable to clinical systems.
3. Ground-Truth Contact Sensing: A practical capacitive sensor solution providing binary contact data synchronized with robot states.
4. Comprehensive Toolbox: An extensible suite for rectification, depth estimation, optical flow, and kinematic reprojection.
5. Validated Dataset: A new dataset of 214 validated instances across multiple tasks and skill levels, available publicly.

4. Results

• Image Quality: The new endoscope system demonstrated a 30x improvement in Laplacian variance (529.48 vs. 16.93) compared to the legacy dVRK-Si scope, significantly aiding segmentation and depth estimation.
• Synchronization Performance:
  • Online-Matching: Achieved a mean time latency of 6.36 ms (std 4.72 ms).
  • Offline-Matching: Achieved a mean time latency of 1.35 ms (std 0.81 ms) after post-processing, with a stable 10 Hz recording frequency.
• Contact Sensor Accuracy: The sensor achieved 99.1% accuracy for tissue manipulation, 74.3% for dissection, and 45.2% for suturing (lower accuracy in suturing attributed to intermittent short-circuits with conductive needles).
• Skill Assessment Validation: A state-of-the-art skill assessment model (using kinematic, visual, and gesture features) was trained on the dataset. It achieved a Spearman's rank correlation (SROCC) of 0.803 for suturing and 0.765 for knot tying, demonstrating the dataset's utility for training AI models.

5. Significance

SurgSync addresses the "data bottleneck" in surgical robotics by providing a robust, reproducible framework for collecting high-fidelity, time-synchronized multi-modal data.

• Enabling AI Autonomy: By providing ground-truth contact data and precise temporal alignment, the framework enables the training of complex sequence models necessary for supervised autonomy.
• Reproducibility: The open-source nature of the software, hardware schematics, and the dataset allows the research community to build upon a standardized baseline, facilitating cross-institutional collaboration.
• Bridging the Gap: The framework bridges the gap between legacy research platforms and modern clinical imaging standards, making it easier to translate research algorithms to real-world clinical applications.

All software and data are publicly available at surgsync.github.io.