Towards Comprehensive Real-Time Scene Understanding in Ophthalmic Surgery through Multimodal Image Fusion

1. Problem Statement

Vitreoretinal surgeries are highly intricate procedures requiring precise manipulation of delicate ocular tissues. Current surgical visualization relies primarily on the Operating Microscope (OPMI), which offers a wide-angle view but lacks the micrometer-level axial resolution needed to visualize subsurface structures or accurately estimate the distance between surgical instruments and the retina. While intraoperative Optical Coherence Tomography (iOCT) provides this depth information, it is often processed separately or lacks temporal integration.

The core challenge addressed in this work is the lack of unified, real-time architectures that can effectively fuse complementary OPMI (surface/appearance) and iOCT (depth/structural) data streams. Existing methods typically rely on single-modality inputs, failing to leverage the holistic scene understanding required for safe, depth-aware robotic or augmented surgery. Furthermore, the scarcity of annotated clinical multimodal datasets hinders the development of such systems.

2. Methodology

The authors propose a Multimodal (MM) and Recurrent Multimodal (RMM) framework designed for real-time, joint instrument detection, keypoint localization, and tool-tissue distance estimation.

Architecture Overview

The system extends a single-modality Yolo-NAS detector with a secondary branch for iOCT data, featuring three main components:

1. Dual-Stream Backbone:
   • OPMI Stream: Uses a Yolo-NAS-m backbone to process standard microscope frames, generating multi-scale feature pyramids.
   • iOCT Stream: Uses a modified ResNet-18 encoder to extract features from cross-sectional B-scans. It employs adaptive pooling to preserve lateral resolution, outputting column descriptors representing depth profiles.
2. Cross-Attention Fusion Module:
   • To address the geometric misalignment between the en-face OPMI view and orthogonal iOCT slices, the authors introduce a cross-attention mechanism.
   • OPMI pixel features act as queries, while iOCT column descriptors act as keys and values.
   • This allows the network to selectively inject spatially aware depth cues into the OPMI feature maps based on semantic relevance rather than fixed pixel alignment.
   • Sinusoidal positional encodings are added to model spatial correspondence.
3. Recurrent Module (RMM):
   • To enhance temporal coherence and robustness against transient data corruption (e.g., signal loss), a lightweight recurrent unit (GRU or LSTM) processes regional descriptors across consecutive frames.
   • This module is supervised by a cosine contrastive loss to encourage feature similarity between intact and corrupted inputs.

Prediction Heads

The framework utilizes task-specific heads on the fused, temporally refined features:

• Detection & Keypoints: Standard Yolo-NAS heads for bounding boxes and keypoints.
• Distance Estimation: A novel distributional regression head. Instead of predicting a single scalar, it predicts a probability distribution over distance bins. This allows the system to output a certainty score (based on distribution sharpness), which is crucial for safety-critical applications.

3. Key Contributions

• First Temporal Fusion Framework: This is the first work to demonstrate temporal feature fusion between OPMI and iOCT for ophthalmic surgery, enabling joint multi-task prediction.
• Cross-Attention Fusion Mechanism: A novel module that effectively aligns and fuses heterogeneous data (2D surface vs. 1D depth profiles) without requiring rigid pixel-level registration.
• Certainty-Aware Distance Estimation: The introduction of a distributional regression head that provides calibrated uncertainty estimates, allowing the system to flag low-confidence predictions.
• Synthetic Dataset Creation: Due to the lack of clinical multimodal datasets, the authors utilized a photorealistic synthetic dataset (SynthesEyes) with full ground truth (keypoints, segmentation, distances) to train and evaluate the models.

4. Results

Experiments were conducted on a synthetic dataset comprising 69,134 frames from simulated retinal membrane peeling and subretinal injection surgeries.

• Performance Metrics:

  • Instrument Detection: The Multimodal (MM) model achieved 95.79% mAP50, a statistically significant improvement over the Single-Modality (SM) baseline (94.27%).
  • Keypoint Localization: No significant difference was found between SM and MM, indicating that depth data does not hinder 2D localization.
  • Distance Estimation (The Major Breakthrough):
    • Overall Accuracy: The MM model reduced the Mean Absolute Error (dMAE) from 480.93 µm (SM) to 128.32 µm (MM) (a 73% reduction).
    • Critical Range (<1 mm): For close-range interactions (safety-critical), the error dropped dramatically from 284.01 µm (SM) to 33.05 µm (MM) (an 88% reduction).
  • Real-Time Performance: The system processes frames in 22.5 ms, maintaining real-time capability.

• Robustness (RMM):

  • The Recurrent Multimodal (RMM) model showed resilience to short-term noise. However, when iOCT data was persistently corrupted, the model suffered from "modality collapse," becoming over-reliant on the missing stream and failing to revert to the SM baseline performance.

• Certainty Calibration:

  • The distributional approach successfully correlated prediction confidence with accuracy. For high-certainty predictions (>90%), the MM model achieved a dMAE of 53.74 µm, compared to 142.99 µm for the SM model.

5. Significance and Conclusion

This work establishes a proof-of-concept for multimodal, depth-aware surgical scene understanding.

• Clinical Impact: By drastically improving distance estimation accuracy (especially within 1 mm of the retina), the framework addresses a primary safety concern in vitreoretinal surgery, potentially reducing the risk of retinal damage.
• Technical Advancement: It demonstrates that cross-attention fusion can effectively merge disparate imaging modalities (microscope and OCT) in real-time, overcoming geometric misalignment issues.
• Future Directions: The study highlights the challenge of maintaining robustness when one modality fails (modality collapse). Future research must focus on adaptive fusion mechanisms that can dynamically balance modality contributions based on data quality.

In summary, the paper proves that fusing OPMI and iOCT data via a tailored deep learning architecture significantly enhances surgical perception, offering a pathway toward safer, more precise, and potentially autonomous image-guided ophthalmic procedures.