Point Tracking in Surgery--The 2025 Surgical Tattoos in Infrared Challenge (STIRC2025)

Technical Summary: Point Tracking in Surgery–The 2025 Surgical Tattoos in Infrared (STIR) Challenge

Problem Statement
Accurate point tracking in surgical environments is a foundational requirement for downstream medical computer vision tasks, including segmentation, 3D reconstruction, virtual tissue landmarking, autonomous probe-based scanning, and subtask autonomy. While knowledge of tissue motion and location is critical for automated dexterity and autonomous scanning, existing methods often struggle with the specific challenges of surgical video, such as tissue deformation, occlusion, and varying lighting conditions. The 2025 STIR Challenge (STIRC2025) was organized to address these gaps by evaluating and advancing algorithms for point tracking in both 2D and 3D spaces.

Methodology and Dataset
The challenge utilized the STIR Challenge 2025 (STIRC2025) dataset, which consists of 32 stereo video sequences collected using a da Vinci 5 system. The dataset includes both in vivo (porcine subjects) and ex vivo (mixed tissue) cases.

• Annotation Protocol: Ground truth labels were generated by tattooing tissue with indocyanine green (ICG) ink. The endoscope captured infrared (IR) images at the start and end of an action to define point locations, while the intervening video was recorded in white light.
• Data Format: Each sequence includes start and end IR images, binary segmentation masks of the IR ink, and the corresponding white-light stereo video (1280 × 1024 pixels).
• 3D Reconstruction: 3D ground truth positions were calculated via epipolar search using normalized cross-correlation on segmented points, followed by back-projection using calibrated camera parameters (baseline, focal length, and principal points).

Evaluation Metrics
Participants were evaluated on two primary components:

1. Accuracy: Measured using the outlier-resistant metric $\delta_{avg}$ (introduced in TAP-Vid), which calculates the percentage of tracked points falling within specific distance thresholds ($l_i$) at the final frame.
   • 2D Thresholds: 4, 8, 16, 32, and 64 pixels.
   • 3D Thresholds: 2, 4, 8, 16, and 32 millimeters.
   • Points were matched to their nearest ground truth label in the final frame.
2. Efficiency: Measured by inference latency. The evaluation focused on the 95th and 99th percentile latencies, as well as the mean, to capture worst-case performance critical for real-time surgical applications. A submission was eligible for efficiency ranking only if its accuracy exceeded 90% of the top-performing method.

Key Contributions and Submissions
Seven teams participated in the challenge, with all submitting 2D solutions and two teams (NCT TSO and MFTIQProb) submitting 3D solutions. The paper details several distinct methodological approaches:

• Baseline Methods: Included Control (zero motion), RAFT, CSRT, MFT (Multi-Frame Tracking), and a 3D baseline combining RAFT with RAFT-Stereo.
• Participant Innovations:
  • BlueJays: Proposed a self-supervised method using contrastive random walks and global matching (GMFlow) with a moving reference frame strategy to handle occlusion.
  • CCG DGIST: Developed a hybrid framework combining dense optical flow with keypoint-based tracking (SuperPoint, ALIKED, LightGlue, RoMA) to balance local accuracy with long-term robustness.
  • MFTIQProb: Introduced a probabilistic multi-point tracking pipeline using Gaussian distributions for candidate points and Bayesian updates for visibility handling.
  • MoriLabNU: Adapted MFT for laparoscopic tracking using Elastic Weight Consolidation (EWC) to prevent catastrophic forgetting while fine-tuning feature extractors on surgical textures.
  • NCT TSO: Built a pipeline on LiteTracker (derived from CoTracker3), incorporating Kalman filtering for motion modeling and keyframe feature buffers to reduce drift.
  • SRV: Replaced the optical flow component in MFT with WAFT (Warping-Along Field Transforms) to avoid large cost volumes.
  • UII VRI: Utilized a frozen, general-domain dense point tracker (AllTracker) with temporal attention and Kalman smoothing, avoiding surgical-domain fine-tuning.

Results

• 2D Accuracy: The overall best-performing method was the baseline MFT ($\delta_{avg} = 83.50$). Among participant submissions, CCG DGIST ranked first ($\delta_{avg} = 82.99$), followed by MFTIQProb ($\delta_{avg} = 82.91$) and MoriLabNU ($\delta_{avg} = 82.74$).
• 3D Accuracy: Team NCT TSO achieved the highest accuracy ($\delta_{avg} = 68.46$), outperforming all baselines. MFTIQProb ranked second ($\delta_{avg} = 57.09$). The authors note that top 3D methods relied on filtered combinations of 2D tracking and stereo triangulation rather than native 3D tracking architectures.
• Efficiency: Team NCT TSO won the efficiency category with a per-frame latency of 26.25 ms (95th percentile), significantly outperforming the next-fastest competitor (UII VRI at 140.52 ms).

Significance and Claims
The paper positions the STIRC2025 challenge and its associated dataset as a critical resource for the surgical robotics and computer vision communities. The authors claim that the challenge highlights the importance of:

1. Robust Cross-Validation: The fact that a baseline (MFT) outperformed many specialized submissions underscores the difficulty of generalization and the risk of overfitting to released test sets.
2. Efficiency vs. Accuracy Trade-offs: The efficiency category demonstrates that high accuracy does not preclude real-time feasibility, with NCT TSO achieving both top-tier 3D accuracy and the lowest latency.
3. Future Directions: The authors modestly suggest that future work should focus on pseudo-labeling, self-supervised training with surgical features, and the development of low-latency models capable of long-term tracking and relocalization without relying on fully 3D native architectures, which remain an open challenge.

The challenge serves as a benchmark for testing, comparing, and iterating on algorithms for tissue tracking, ultimately aiming to support applications in robotic-assisted surgery.