Technical Summary: SurgOnAir

Problem Statement

Current surgical AI systems struggle to provide real-time, temporally grounded understanding of surgical workflows. While Multimodal Large Language Models (MLLMs) have advanced general video comprehension, their reliance on heavy backbones and offline processing paradigms (requiring access to the entire video sequence) prevents immediate reaction to evolving surgical dynamics. In the operating room, critical decisions depend on subtle, moment-to-moment changes in instrument movement and tissue states; even slight perceptual delays can compromise safety or limit assistance. Furthermore, existing methods often treat surgical workflows as flat sequences, overlooking the inherent hierarchical structure (Phases $\rightarrow$ Steps $\rightarrow$ Actions) that is crucial for contextual grounding and reducing hallucinations. There is a lack of models capable of streaming video processing while simultaneously generating multi-level, hierarchy-aware narrations.

Methodology

1. Data Curation: SurgOnAir-11K

The authors introduce SurgOnAir-11K, a curated hierarchical dataset designed to bridge the gap between raw surgical video and structured narration. The data curation pipeline involves:

• Temporally Grounded ASR: Using WhisperX-large to extract word-level transcripts with precise timestamps, creating triplets of $\langle word, t_{start}, t_{end} \rangle$.
• Domain-Aware Correction: Low-confidence sentences (log probability < -0.15) are corrected using GPT-4o with strict constraints to normalize terminology without altering content.
• Action-Aware Filtering: Sentences are classified into ACTION, EXPLANATION, and INTERACTION. Only ACTION sentences, which are strictly grounded in visual content, are retained for supervision.
• Hierarchical Construction: The dataset is structured into a three-level hierarchy: Phase $\rightarrow$ Step $\rightarrow$ Word. High-level procedural steps are generated via LLMs to ensure non-overlapping, temporally monotonic partitions, which are then aligned with metadata-defined phases.

2. Model Architecture: SurgOnAir

SurgOnAir is a streaming vision-language model built upon the LiveCC-7B-Base backbone, designed for real-time frame-to-token generation without future access.

• Dense Temporal Interleaving: The model adopts a streaming input strategy where visual frames ($\langle F \rangle$) and text tokens ($\langle W \rangle$) are densely interleaved: $[Con] \langle F_{t:t+k} \rangle \langle W_{t:t+k} \rangle \dots$. The stride $k$ is set to 1 with a frame rate of 2 FPS to ensure fine-grained alignment.
• Hierarchical State Tokens: To enforce hierarchy-aware narration, the model utilizes special transition tokens (e.g., <State>Phase=P1, Step=S1</State>). These tokens are conditionally inserted into the text stream only when a state transition occurs, acting as discrete anchors for the workflow. This allows the model to function simultaneously as a continuous narrator and a strict state tracker.
• Training Strategy: The model employs a staged conditioning strategy. In the first two epochs, it is conditioned on previous ASR text to stabilize training. In the final epoch, ASR context is removed (retaining only the video title), forcing the model to rely directly on visual evidence.

Key Contributions

1. Hierarchical Temporal Grounding Pipeline: A novel method to convert raw surgical videos and ASR transcripts into a multi-level, temporally aligned video-language corpus (SurgOnAir-11K) featuring explicit Phase, Step, and Word supervision.
2. SurgOnAir Framework: A hierarchical streaming framework that interleaves multi-modal tokens for real-time processing. It uniquely leverages specialized transition tokens to explicitly predict state changes, enabling the generation of narrations that are aware of the procedural hierarchy.
3. Real-Time Performance: The demonstration of a single vision-language model capable of unifying streaming across multiple hierarchies of the surgical workflow, outperforming both offline and existing streaming baselines in real-time narration quality.

Experimental Results

The model was evaluated on an 80/20 split of the SurgOnAir-11K dataset using a pairwise choice task judged by GPT-4o, prioritizing semantic correctness and grounding over syntactic perfection.

• Comparison with Baselines:
  • Offline Models: Offline Video-LLMs (LLaVA-Video-7B, Qwen2.5-VL-7B) performed poorly (win rates of 11.3% and 6.2% respectively), highlighting their unsuitability for real-time surgical scenarios.
  • Streaming Baselines: The generic streaming model LiveCC-7B achieved a 16.7% win rate, outperforming offline models but failing to capture complex surgical nuances.
  • SurgOnAir Variants: A flat-narration version trained on in-domain data (SurgOnAir-base) achieved a 60.4% win rate. The full SurgOnAir model, incorporating hierarchical modeling, achieved the highest win rate of 66.1%.
• Ablation Studies:
  • Hierarchy: Removing hierarchical structure (SurgOnAir-base) dropped the win rate to 39.4% against the full model, confirming the necessity of structural guidance.
  • State Formulation: A variant forcing state prediction at every word token (SurgOnAir-v1) achieved 42.5%, compared to 57.5% for the proposed transition-aware formulation, indicating that redundant state generation harms narration fluency.
  • Phase Correctness: When the model correctly predicted the surgical phase, the win rate against the base model jumped to 65.8%, demonstrating that accurate phase grounding directly supports contextually aligned narration.
• Qualitative Results: The model successfully transitions coherently across procedural boundaries (e.g., from "Mesh Introduction" to "Mesh Fixation") and maintains micro-level temporal alignment, accurately describing instruments and anatomy at the moment of interaction.

Significance and Claims

The paper claims that SurgOnAir represents the first real-time surgical narration framework built upon hierarchical modeling. Its primary significance lies in advancing real-time surgical understanding beyond conventional video captioning by producing descriptions that are both temporally aligned and hierarchy-aware. The authors state that this work lays a foundation for future surgical automation and intelligent surgical teaching systems.

The paper remains modest regarding its limitations, acknowledging that the current dataset scale is limited and that the model does not yet perform future action prediction, noting these as promising directions for further exploration.