SurgAtt-Tracker: Online Surgical Attention Tracking via Temporal Proposal Reranking and Motion-Aware Refinement

Imagine you are watching a complex surgery on a tiny screen. The surgeon is working with incredibly small tools inside a patient's body, and the camera (the endoscope) is being held by a human assistant.

The Problem:
Human assistants get tired. Their hands shake, they get distracted, or they just can't keep up with the surgeon's fast thinking. Sometimes the camera drifts away from the action, or it gets stuck on a tool instead of the tissue being cut. This is dangerous. The surgeon needs the camera to be a "sixth sense," always looking exactly where the surgeon's eyes are focused, without the surgeon having to ask for it.

The Old Way:
Previous attempts to fix this were like trying to guess where a person is looking by only watching their hands. If the surgeon is holding a scalpel, the camera just follows the scalpel. But what if the surgeon is looking at a bleeding spot near the scalpel, or looking at a piece of tissue behind it? The old cameras got confused, jittery, or looked at the wrong thing when there were too many tools on screen.

The New Solution: SurgAtt-Tracker
The authors of this paper built a smart AI system called SurgAtt-Tracker. Think of it as a "Super-Focus Assistant" that doesn't just follow tools, but actually understands what the surgeon is thinking about.

Here is how it works, broken down into simple metaphors:

1. The "Heatmap" (The Map of Focus)

Instead of just drawing a box around one object (like a tool), the AI draws a heat map.

Analogy: Imagine a thermal camera. The hottest spot (bright red) is exactly where the surgeon is looking. As you move away from that spot, it gets cooler (yellow, then blue).
Why it matters: This tells the camera, "The surgeon is focused here, but also slightly there." It captures the nuance of human attention, which is rarely just a single dot.

2. The "Candidate Pool" (The Shortlist)

The AI doesn't try to guess the answer from scratch every single second. That's too hard and prone to mistakes.

Analogy: Imagine a hiring manager looking for a new employee. Instead of interviewing one person and hoping they are perfect, they interview 100 people (the "Top-K proposals"). They know the perfect candidate is somewhere in that group, but they aren't sure who it is yet.
The Tech: A fast, frozen detector quickly scans the video and says, "Here are 100 possible places the surgeon might be looking."

3. The "Time-Traveler" (Temporal Reranking)

This is the magic sauce. The AI looks at the video from the previous second to help decide who to pick from the shortlist.

Analogy: Imagine you are watching a movie and trying to guess which character the director is focusing on. If you only look at one frozen frame, it's hard to tell. But if you remember what happened one second ago, it becomes obvious.
The Tech: The system asks, "Who was the focus last second? Which of these 100 candidates makes the most sense continuing from where we were?" It re-ranks the list based on consistency. It filters out the "noise" (like a tool that just happened to move) and picks the candidate that fits the story of the surgery.

4. The "Smooth Operator" (Motion-Aware Refinement)

Even after picking the best candidate from the list, the box might be slightly off-center or the wrong size.

Analogy: Imagine you are throwing a dart. You aim at the bullseye (the candidate), but your hand shakes a little. The "Refinement" module is like a gentle hand that nudges the dart the final few millimeters to hit the exact center, taking into account how fast you were moving.
The Tech: It looks at how the camera and tools moved between frames and makes tiny, smooth adjustments to the final focus point.

5. The "Giant Library" (SurgAtt-1.16M)

To teach this AI, the researchers didn't just use a few videos. They built a massive library of 1.16 million frames of surgery videos.

Analogy: It's like teaching a student to drive. You don't just let them drive on one empty street. You give them a library of driving videos from rainy days, snowy days, busy cities, and empty highways.
The Result: Because the AI learned from so many different surgeries (stomach, rectum, uterus, kidneys) and different hospitals, it works really well even on surgeries it has never seen before.

Why This is a Big Deal

Safety: The camera will never lose focus, even if the surgeon's hands are shaking or the view is blurry.
Fatigue: The human assistant can relax because the AI is doing the heavy lifting of keeping the camera steady.
Intelligence: It doesn't just follow tools; it follows intent. If the surgeon stops cutting and looks at a specific tissue to inspect it, the camera follows that inspection, not the tool.

In a nutshell: SurgAtt-Tracker is like a highly intelligent, tireless co-pilot that watches the surgeon's eyes (via their actions), predicts exactly where they want to look next, and steers the camera smoothly to that spot, ensuring the surgeon always has the perfect view.

1. Problem Definition

The paper addresses the critical challenge of Field-of-View (FoV) guidance in Minimally Invasive Surgery (MIS). Current automated camera control systems often rely on:

Explicit external inputs (e.g., gaze, voice), which increase surgeon cognitive load.
Heuristic instrument tracking, which assumes surgeon attention is always centered on a specific tool. This fails in complex scenarios involving multiple instruments, rapid focus shifts, or when attention is directed at tissue rather than tools.
Direct regression models that attempt to predict camera motion or attention points directly from single frames, leading to instability due to occlusion, motion blur, and low inter-class separability.

Core Problem: The authors argue that surgeon attention is a latent cognitive state that is context-dependent, temporally coherent, and spatially distributed. Existing methods fail to model this as a continuous, dense spatial signal, leading to unstable FoV control. The goal is to track this "surgical attention" as a dense heatmap in real-time to provide a robust signal for downstream robotic camera control.

2. Methodology: SurgAtt-Tracker

The authors propose SurgAtt-Tracker, a holistic framework that treats attention tracking as a spatio-temporal learning problem rather than isolated frame-wise detection. The architecture consists of three main stages:

A. Frozen Detector & Proposal Generation

Instead of training a detector from scratch, the system uses a pre-trained, frozen detector (e.g., YOLOv12) as a high-recall hypothesis generator.
It generates a Top-K proposal set ( $B_t$ ) and extracts multi-scale feature pyramids ( $F_t$ ) from the image.
Rationale: While the detector's Top-1 prediction is often unstable due to visual ambiguity, the true attention target is consistently present within the Top-K set (high recall).

B. Multi-Scale ROI Decoder (MSR)

To handle varying scales of surgical interactions (from tiny instrument tips to large anatomical regions), the MSR maps discrete bounding box candidates back into the continuous semantic space of the feature pyramid.
It uses ROIAlign to extract fixed-size features from multiple pyramid levels, fusing them into a unified, box-aligned embedding ( $f_t$ ) that is scale-invariant.

C. Attention Score Rerank (AS-Rerank) Module

Mechanism: This module shifts the selection criterion from static confidence scores to dynamic temporal consistency.
Process: It employs a cross-attention mechanism to compare the embeddings of current proposals ( $f_t^k$ ) against a trusted reference embedding ( $f_r$ ) from a previous frame ( $t-n$ ).
Output: It re-ranks the proposals to select the one with the highest temporal affinity, outputting a refined Top-1 hypothesis ( $\hat{B}_t$ ). This effectively filters out noisy detections caused by occlusion or blur.

D. Motion-Aware Adaptive Refine (MAA-Refine) Module

Problem: The selected proposal is still quantized by the detector's anchor grid, leading to residual misalignment.
Solution: A lightweight MLP fuses the visual semantic embedding ( $\hat{f}_t$ ) with a geometric motion descriptor ( $g_t$ ) derived from the spatial evolution between the current and reference boxes.
Formulation: Instead of standard Cartesian offset regression, it uses a polar-based formulation to predict:
1. Displacement direction ( $\theta_t$ ) and magnitude ( $d_t$ ).
2. Channel-wise scale factors ( $s_t$ ).
This allows for continuous, precise geometric correction based on both local texture and the momentum of the surgeon's attention.

3. Key Contributions

A. SurgAtt-1.16M Benchmark

The authors introduce SurgAtt-1.16M, the largest and most diverse benchmark for surgical attention tracking to date.

Scale: Contains 1.16 million frames across multiple organs (Rectum, Stomach, Uterus, Kidney) and procedures.
Annotation Protocol: A novel hierarchical strategy that bridges discrete expert intent with continuous heatmaps:
1. Latent Attention Center ( $B_t$ ): Experts annotate a discrete bounding box representing the primary interaction (e.g., tissue-tool contact).
2. Attention Density ( $H_t$ ): These discrete boxes are converted into dense, anisotropic Gaussian heatmaps using temporal exponential decay. This simulates human visual persistence and provides a smooth, probabilistic target for training.
Composition: Aggregates the new SurgAtt-SZPH dataset (141 hours of clinical footage) with re-annotated versions of AutoLaparo and Hamlyn datasets.

B. Novel Framework Design

Decoupling: Separates attention modeling from downstream camera control.
Two-Stage Optimization: Combines Proposal Reranking (to select the best candidate from a high-recall set using temporal coherence) and Motion-Aware Refinement (to achieve sub-pixel precision).
Loss Functions: Introduces a composite loss including Top-M Listwise Ranking (to smooth decision boundaries) and Log-Space Scale Loss (to handle scale variance).

4. Experimental Results

The method was evaluated on SurgAtt-SZPH, SurgAtt-AutoLaparo, and SurgAtt-Hamlyn against four categories of baselines: U-Net-based, Regression-based, Object Tracking-based, and Detection-based.

Performance: SurgAtt-Tracker achieved State-of-the-Art (SOTA) results across all metrics.
- On SurgAtt-SZPH: NSS = 2.580, CC = 0.871, SIM = 0.829, with significantly lower errors (MSE=0.015, MAE=0.051) compared to the next best method (RT-DETRv2).
- Improvement: Outperformed RT-DETRv2 by 8.9% in NSS and 12.2% in SIM.
Robustness: The framework demonstrated strong robustness under:
- Occlusion: Maintained tracking when instruments or tissues were blocked.
- Multi-instrument Interference: Correctly identified the active tool/tissue among clutter.
- Cross-Domain Generalization: Showed strong zero-shot transfer to AutoLaparo and Hamlyn datasets, with further gains after fine-tuning.
Efficiency: Runs at 12.5 FPS, approaching real-time capability for closed-loop endoscopic control.

5. Significance and Impact

Clinical Relevance: By providing a dense, interpretable attention heatmap, the system offers a continuous signal for FoV guidance that is more nuanced than discrete directional cues. This can directly support robotic FoV planning and automatic camera control, reducing surgeon fatigue and operative risks.
Paradigm Shift: Moves the field from "instrument tracking" to "intent tracking," acknowledging that surgeon attention is a dynamic, context-dependent cognitive state.
Open Science: The release of SurgAtt-1.16M democratizes access to high-quality, clinically grounded data, enabling future research in surgical AI and human-robot collaboration.
Future Work: The authors plan to integrate SurgAtt-Tracker into robotic systems for safe, intent-aware endoscope control, emphasizing the need for "human-in-the-loop" validation before clinical deployment.