A 4D Representation for Training-Free Agentic Reasoning from Monocular Laparoscopic Video

1. Problem Statement

Spatiotemporal reasoning is critical for AI in soft-tissue surgery to enable intelligent assistive systems and autonomous robotics. While current surgical AI relies heavily on 2D vision-language models (VLMs) and 2D image-space reasoning, surgical scenes possess inherent 3D spatial complexity and temporal dynamics that 2D models struggle to capture.

• The Gap: Existing methods often lack explicit 4D (3D space + time) representations, making it difficult for AI agents to understand anatomical dissection planes, tool-tissue interactions, and manipulation directions.
• The Challenge: Reconstructing 4D scenes from monocular laparoscopic video is difficult due to adverse visual features (specular highlights, lack of camera motion) and the need for training-free solutions that do not require fine-tuning large models on specific surgical data.

2. Methodology

The authors propose a framework that constructs an explicit, tracked semantic 4D representation from monocular video and uses it as a "toolkit" for a Multimodal Large Language Model (MLLM) agent. The system operates in a training-free manner.

A. Construction of the 4D Representation

The pipeline converts a 2D monocular video into a coherent 4D point cloud with consistent semantics ($P_{semantic}$):

1. Depth & Segmentation: Uses Depth Anything 3 (DA3) for metric depth estimation and camera parameters, and SASVi for semantic segmentation masks.
2. Point Tracking: Utilizes Cotracker 3 to track 2D points across frames.
3. Lifting to 4D:
   • Control Points: 2D tracked points are lifted to 3D using DA3 depth and camera parameters to create a sparse control point cloud ($P_{control}$).
   • Densification: Pixels are associated with their $K$-nearest control points and interpolated to create a dense point cloud ($P_{dense}$).
   • Semantic Consistency: Frame-wise semantic masks are converted to instance segmentations. Instance labels are lifted to 3D and tracked to a reference timestep. A union-find operation merges instances from different frames that share the same semantic class and high 3D containment scores, creating time-persistent 3D instances.
4. Robustness Mechanisms:
   • 3D Control Point Assignment: Unprojects pixels before assignment to prevent background/foreground mixing.
   • Jump Filtering: Removes points with abrupt $z$-axis depth changes to filter 2D tracking errors.
   • Depth Maintenance: "Forward fills" depth values for occluded points to prevent them from jumping into the foreground when occluded.
   • Single-View Tracking: Initializes tracking from the middle frame of a clip (tracking both forward and backward) to maximize the likelihood of capturing relevant points, avoiding noise from multi-view initialization.
   • Depth Gradient Filtering: Sharpens object boundaries to counteract DA3's oversmoothing.

B. Agentic Reasoning with Spatiotemporal Tools

The 4D representation is packaged as an agentic toolkit for an MLLM (specifically Qwen3-VL).

• Tool Calling: The agent does not process raw video pixels for reasoning but instead calls tools to query the 4D representation.
• Available Tools:
  • Compute minimum distances between semantic instances.
  • Calculate 3D overlap scores and positions.
  • Determine relative motion between instances over time.
  • Retrieve 4D trajectories.
  • Frame Fetching: Request specific video frames at timesteps to resolve ambiguities (e.g., distinguishing between poking and coagulating).
• Workflow: Upon receiving a query, the agent iteratively requests spatiotemporal data from the 4D model to ground its natural language response in 3D space and time.

3. Key Contributions

1. Training-Free 4D Framework: A novel method to equip surgical agents with spatiotemporal tools derived from explicit 4D representations without fine-tuning the MLLM.
2. Robust 4D Reconstruction: A pipeline that successfully generates tracked, semantic 4D point clouds from monocular laparoscopic video despite challenging visual conditions (specular highlights, low camera motion).
3. Agentic 4D Grounding: Demonstrates that "spatiotemporal intelligence" can be assembled by combining 2D MLLMs with 3D computer vision models, allowing agents to answer complex questions about where, when, and in what direction surgical interactions occur.
4. New Benchmark: Introduction of a dataset with 134 clinically relevant queries covering spatial, temporal, and directional reasoning on the Cholec-80 dataset.

4. Experimental Results

The method was evaluated against 2D MLLM baselines (with and without Set-of-Mark prompting) on 134 queries across 25 clips.

• Spatial Reasoning (3D Localization):
  • The proposed agent significantly outperformed the best 2D baseline, reducing the L2 error by nearly 50% (46.54 vs. 90.76 pixels).
  • Notably, providing semantic prompts to the 2D baseline inhibited its performance, whereas the 4D agent thrived on explicit geometric tools.
• Temporal Reasoning (Point-in-Time & Intervals):
  • The agent achieved the lowest error for Point-in-Time (PIT) queries (3.62 timesteps vs. 5.12 for baseline).
  • While the agent was slightly less accurate on temporal intervals (IoU), it demonstrated superior capability in localizing specific events in time.
• Directional Reasoning:
  • The agent improved over the best baseline by 79% in predicting the dominant 3D direction of tissue manipulation (L1 error 0.17 vs. 0.83).
  • Access to 4D trajectories was identified as the key factor for this improvement.
• Ablation Studies:
  • Removing jump filtering or depth maintenance significantly degraded performance, particularly in temporal tasks.
  • Multi-view tracking (initializing from multiple frames) performed worse than single-view tracking, likely due to accumulated noise.
  • Smaller model sizes (8B vs. 32B) resulted in significantly worse performance due to difficulties in parsing tool outputs.

5. Significance

• Paradigm Shift: Moves surgical AI from 2D visual question answering to 4D agentic reasoning, enabling systems to understand the physical dynamics of surgery (e.g., grasping points, dissection planes).
• Clinical Relevance: The ability to determine direction and timing of tool-tissue interaction is crucial for skill assessment, real-time guidance, and autonomous robotics.
• Scalability: By leveraging foundation models (Depth Anything, Cotracker, Qwen) in a training-free manner, the approach is adaptable to various surgical disciplines without the need for massive, domain-specific annotated datasets.
• Future Potential: While currently an offline process, the components are moving toward real-time capabilities, paving the way for intraoperative AI assistants that can "see" and "reason" in 3D space.

Code and Data Availability: The authors have made their code, data, and examples publicly available at https://tum-ai.github.io/surg4d/.