Technical Summary: Surgical Anatomy Recognition with Context Learning using Foundation Representations

Problem Statement

Accurate recognition of anatomical structures is critical for safe and effective minimally invasive surgery (MIS). However, progress in surgical computer vision for this specific task has been hindered by two primary factors:

1. Data Scarcity: A lack of large-scale, diverse, and richly annotated datasets that capture the variability of real surgical practice. Existing datasets often suffer from limited procedural diversity, few videos, and incomplete anatomical class coverage, which restricts model generalization.
2. Methodological Limitations: Conventional approaches often prioritize object tracking or are tailored to natural scenes, failing to leverage the specific contextual cues (e.g., procedure type, surgical phase) that surgeons rely on to identify anatomy under limited viewpoints and without tactile feedback.

Methodology

1. The ATLAS-120k Dataset

The authors introduce ATLAS-120k, a large-scale clip-level semantic segmentation dataset designed to address data limitations.

• Scale and Diversity: Comprising over 120,000 annotated frames from 100 surgical videos, the dataset spans 14 distinct procedures (including laparoscopic and robot-assisted MIS) and covers 42 anatomical classes.
• Annotation Pipeline: A scalable workflow was employed involving:
  • Expert manual labeling of initial frames by surgical research fellows under surgeon supervision.
  • Automated propagation using a lightweight object-tracking model.
  • Iterative refinement where the tracker was fine-tuned on the annotated data at intervals (10k, 25k, 50k frames).
  • Manual correction of propagation errors and independent verification of the first frame of every clip by experienced surgeons.
• Standardization: Videos were downsampled to 15 fps, with spatial resolutions ranging from 480×640 to 1080×1920.

2. The ATLAS Model

The authors propose ATLAS (Anatomy Recognition with Context Learning using Foundation Representations), a video semantic segmentation model specifically architected for surgical anatomy. Unlike standard video segmentation which focuses heavily on tracking, ATLAS integrates foundation-model embeddings with context learning.

• Architecture: Built upon EoMT (a state-of-the-art image segmentation model using ViT encoders and learnable queries), ATLAS extends this to video by introducing context queries alongside standard segmentation and tracking queries.
• Input Representation: For each frame $t$, the transformer input $X_t$ includes:
  • Visual patch tokens ($P_t$).
  • Learnable segmentation queries ($Q_{seg}$).
  • Propagated queries from the previous frame ($\hat{Q}_{t-1}$).
  • Context Queries: Learnable initial queries for procedure ($Q_{proc}$) and surgical phase ($Q_{phase}$), which are refined over time to capture global and temporal context.
• Context Mechanisms:
  • Procedure Queries: Encoded via a Multi-Layer Perceptron (MLP) to predict the procedure label, supervised by cross-entropy loss. This captures stable, high-level context (e.g., instrument presence, exposure patterns).
  • Phase Queries: Encoded to capture finer temporal variations. An InfoNCE contrastive loss is applied to ensure embeddings from temporally adjacent frames within the same clip are close, while repelling embeddings from different clips or distant timestamps.
• Temporal Propagation: Queries are propagated from frame $t-1$ to $t$ using a linear layer, ensuring short-term memory and temporal consistency similar to VidEoMT but augmented with the new context dimensions.

Key Contributions

1. ATLAS-120k Dataset: A new benchmark dataset offering unprecedented procedural diversity (14 procedures) and scale (120k+ frames), bridging the gap between single-procedure datasets and the complexity of real-world surgical practice.
2. ATLAS Model: A novel architecture that combines foundation-model representations with lightweight temporal reasoning and explicit context queries (procedure and phase). This design enables the model to utilize procedural knowledge to resolve ambiguities in anatomical recognition.
3. Open Resources: The models, dataset annotations, and annotation platform are publicly released to facilitate further research.

Experimental Results

The model was evaluated on a patient-level split (70 train, 10 validation, 20 test) with 30 consolidated classes.

• Performance: The ATLAS ViT-L variant achieved:
  • 0.64 AP (Average Precision)
  • 0.49 mDice (Mean Dice coefficient)
  • 0.79 mVC24 (Mean Video Consistency over 24 frames)
  • 64 FPS inference speed on an NVIDIA H100, demonstrating real-time feasibility.
• Comparisons: ATLAS consistently outperformed natural-image foundation models (e.g., DINOv2/3, SAM variants) and in-domain medical models (e.g., EndoViT, SurgeNet).
• Ablation Studies:
  • Pretraining: In-domain pretraining (using DINOv3 on the SurgeNet dataset) provided a substantial boost over default weights.
  • Components: Adding temporal query propagation improved AP and AP50. Incorporating context queries further improved detection, region-overlap, and temporal consistency metrics.
  • Efficiency: These improvements yielded a >20% increase in AP relative to the baseline while increasing the parameter count by less than 1%.

Significance and Claims

The paper claims that the combination of the ATLAS-120k dataset and the ATLAS model establishes a practical foundation for robust surgical scene understanding. The authors assert that:

• Context is Critical: Robust anatomical perception in MIS requires more than just visual tracking; it necessitates the integration of procedural and phase-dependent context, which the proposed model successfully captures.
• Bridging the Gap: The work aims to close the gap between research prototypes and clinically useful, anatomy-aware guidance systems.
• Real-World Applicability: By demonstrating high accuracy and temporal consistency across diverse procedures in real-time, the framework supports the development of safer, more intelligent assistance systems for the operating room.

The authors maintain a modest tone regarding the absolute Dice scores, acknowledging the challenging nature of the dataset (highly imbalanced classes, long-tail distribution) and positioning ATLAS-120k as a demanding benchmark for future models rather than a solved problem.