SurgMotion: A Video-Native Foundation Model for Universal Understanding of Surgical Videos

1. Problem Statement

Current foundation models for surgical video analysis suffer from three fundamental limitations that hinder robust, universal understanding:

• Inefficient Learning Objectives: Most existing models rely on pixel-level reconstruction (e.g., MAE, VideoMAE). In surgical environments, this forces models to waste capacity reconstructing high-frequency, non-semantic noise (smoke, specular reflections, fluid motion) rather than learning critical semantic structures like instrument-tissue interactions.
• Lack of Spatiotemporal Stability: Surgical videos exhibit strong temporal continuity and structured relationships. Current pretraining pipelines often lack explicit constraints to enforce relational stability across time, leading to feature drift during significant motion or viewpoint changes.
• Representation Collapse in Sparse Textures: Surgical scenes often contain texture-sparse regions (e.g., smooth organ surfaces, adipose tissue). Self-supervised learning in these environments risks representation collapse, where the model learns low-variance, redundant features instead of diverse, informative channels.
• Data Fragmentation: Existing datasets are typically small-scale and single-procedure specific (e.g., only cholecystectomy or only pituitary surgery), preventing models from learning universal representations capable of generalizing across diverse anatomical regions and surgical modalities.

2. Methodology

The authors propose SurgMotion, a video-native foundation model built on the Video Joint Embedding Predictive Architecture (V-JEPA). Instead of reconstructing pixels, SurgMotion predicts latent representations of masked spatiotemporal regions. The framework introduces three key technical innovations tailored to surgical video:

A. Motion-Guided Latent Masked Prediction

To address the mismatch between objectives and semantics, the model shifts from pixel space to latent space.

• Mechanism: It introduces a motion saliency signal based on spatiotemporal gradients.
• Function: The prediction loss is reweighted to prioritize masked tubes (spatiotemporal patches) that exhibit strong motion changes (e.g., moving tools, tissue deformation).
• Benefit: This reduces sensitivity to low-level visual disturbances (smoke, reflections) and forces the model to focus on critical dynamic regions.

B. Spatiotemporal Affinity Self-Distillation

To enforce relational stability without manual labels, the model employs a teacher-student architecture with an Exponential Moving Average (EMA) teacher.

• Mechanism: A Global Token Affinity Matrix models pairwise relationships between all latent tokens.
• Function: A self-distillation loss aligns the affinity structures of the student network with the momentum-updated teacher.
• Benefit: This explicitly imposes spatiotemporal coherence and robustness against augmentations, ensuring the model understands the structural relationships between tools and anatomy over time.

C. Spatiotemporal Feature Diversity Regularization (SFDR)

To prevent representation collapse in texture-sparse scenes:

• Mechanism: SFDR operates on two levels:
  1. Regularizing the variance of pairwise feature similarities to maintain diversity.
  2. Minimizing inter-channel covariance to reduce redundancy.
• Benefit: This encourages the network to learn complementary and informative feature channels even in visually homogeneous surgical environments.

3. Key Contributions

A. The SurgMotion-15M Dataset

The authors curated SurgMotion-15M, the largest surgical video dataset to date, designed to facilitate large-scale self-supervised learning.

• Scale: 3,658 hours of video comprising 45,757 videos.
• Diversity: Aggregated from 50 distinct sources (public and private) covering 13 anatomical regions and 100+ procedures (including neurosurgery, laparoscopy, ophthalmology, open surgery, etc.).
• Impact: This dataset breaks the "data silo" problem, exposing the model to unprecedented variability in surgical modalities and institutions.

B. Architectural Innovation

The integration of V-JEPA with surgical-specific constraints (Motion-guided masking, Affinity self-distillation, SFDR) creates a model that learns motion-centric semantics rather than static pixel statistics.

4. Experimental Results

SurgMotion was evaluated across 17 benchmarks covering workflow recognition, action recognition, triplet recognition, skill assessment, segmentation, and depth estimation. It consistently outperformed state-of-the-art (SOTA) general vision models (DINOv3, VideoMAE) and surgical-specific models (EndoFM, EndoMamba, GSViT).

• Surgical Workflow Recognition:
  • EgoSurgery: +14.6% F1-score improvement over the best prior surgical foundation model.
  • PitVis: +10.3% F1-score improvement.
  • Cholec80: Achieved 91.05% accuracy and 77.95% Jaccard index, outperforming DINOv3-L by +8.35% in Jaccard.
• Fine-Grained Action Understanding:
  • Action Triplet Recognition (CholecT50): Set a new SOTA with 39.54% mAP-IVT, surpassing DINOv3-L by +2.7%.
  • Skill Assessment (JIGSAWS): Achieved a Mean Absolute Error (MAE) of 2.649 and Spearman correlation of 0.770, significantly reducing error by 26% compared to the next best model.
• Dense Prediction:
  • Polyp Segmentation: Demonstrated superior generalization on Out-of-Distribution (OOD) datasets (e.g., 0.9112 Dice on CVC-300), capturing generalizable morphological features rather than dataset artifacts.
  • Depth Estimation: Achieved best-in-class RMSE (1.88) on C3VD, implicitly capturing 3D spatial structure despite not being explicitly trained for geometry.

5. Significance

SurgMotion establishes a new standard for universal surgical video understanding by fundamentally shifting the learning paradigm from pixel reconstruction to latent motion prediction.

• Universal Generalization: It proves that large-scale, diverse data combined with motion-centric objectives allows models to generalize across different surgical specialties, institutions, and imaging modalities (laparoscopic, open, endoscopic, microscopic).
• Clinical Applicability: The model's robustness in workflow recognition, skill assessment, and dense prediction makes it a viable foundation for real-world applications such as intraoperative decision support, automated documentation, and objective surgical training.
• Resource Availability: The release of SurgMotion-15M addresses the critical bottleneck of data scarcity in surgical AI, providing a comprehensive resource for future research.

In summary, SurgMotion demonstrates that by prioritizing motion semantics and enforcing spatiotemporal consistency on a massive, diverse dataset, AI models can achieve a level of surgical understanding that was previously unattainable with pixel-based reconstruction methods.