Technical Summary: Modality-Invariant Coarse-to-Fine Retinal Image Registration

1. Problem Statement

Retinal image registration is critical for ophthalmic diagnosis, longitudinal disease monitoring, and multimodal analysis. However, existing registration methods are predominantly modality-dependent. They are typically optimized for a single imaging modality (mono-modal) or a fixed pair of modalities (cross-modal). This reliance limits their flexibility in practical scenarios where diverse combinations of retinal imaging modalities (e.g., Color Fundus, Infrared, Fluorescein Angiography, Multicolor SLO, Fundus Autofluorescence) must be aligned without retraining or fine-tuning for each specific pair.

Furthermore, accurate registration requires a coarse-to-fine approach: global alignment via sparse features is insufficient for precise local alignment, which requires dense optical flow. While two-stage pipelines exist, a unified framework that achieves modality invariance in both the coarse (global) and fine (local) stages remains unexplored. Additionally, obtaining dense ground-truth flow for real cross-modal retinal pairs is difficult, hindering the training of supervised dense registration models.

2. Methodology

The authors propose a generalizable, two-stage framework: Modality-Invariant Coarse-to-Fine Retinal Image Registration.

Stage 1: Modality-Invariant Coarse (Global) Alignment

To achieve robust global alignment across arbitrary modalities, the authors introduce a Universal Retinal Vessel Segmentation Model (URVSM) as a domain adapter.

• Domain Adaptation: Instead of matching raw images from different modalities, both the source and target images are processed by URVSM to generate unified vessel segmentation maps. This transforms diverse modalities into a shared, modality-independent representation.
• Sparse Feature Matching: A standard sparse-feature-based registration pipeline (keypoint detection, description, and outlier rejection) is applied to these vessel maps.
• Generalization: A single network, trained on a specific modality pair (Color Fundus and Infrared), is used to detect and match keypoints on vessel maps derived from any modality. The outlier rejection network is kept frozen (pretrained) as it operates on coordinates rather than image appearance.
• Output: An affine transformation matrix is estimated via weighted least squares to globally align the source image to the target.

Stage 2: Modality-Invariant Fine (Local) Alignment

To refine the alignment, the authors propose MI-RAFT (Modality-Invariant Recurrent All-Pairs Field Transforms), an optical flow estimation network.

• Architecture: MI-RAFT extends the RAFT architecture by incorporating features from pretrained, modality-invariant vision models (specifically DINOv3 and Diffusion-based features/DIFT) alongside the standard flow-optimized features.
• Feature Fusion: The model learns to refine and fuse flow-specific features with modality-invariant features. These fused features are used to construct the all-pairs correlation volume for iterative flow estimation.
• Training Strategy (Synthetic Pairs): Since real cross-modal pairs with dense ground-truth flow are unavailable, the model is trained on synthetically generated pairs.
  • Geometric Synthesis (GS): A single retinal image is transformed using random affine transformations and non-linear displacement fields to create a source-target pair with known ground-truth flow.
  • Intensity Synthesis (IS): To simulate modality differences, the synthetic pairs undergo independent random intensity augmentations, low-frequency illumination perturbations, and focus/resolution perturbations. This encourages the model to rely on structural geometry rather than appearance.
• Correlation-Lookup Feature Alignment (CLFA): A novel training strategy where two synthetic pairs are generated from the same image with identical geometry but different intensity transformations. The model is trained to ensure that the correlation-lookup features (local matching evidence retrieved from the correlation pyramid) are consistent across these pairs, further enhancing modality invariance without directly constraining the fused descriptors.
• Loss Function: The total loss combines flow estimation loss, smoothing loss, and the CLFA loss.

3. Key Contributions

1. First Unified Framework: The paper proposes the first framework for retinal image registration that achieves modality invariance in both coarse (global) and fine (local) alignment stages.
2. Universal Domain Adapter: It demonstrates that a universal retinal vessel segmentation model (URVSM) can effectively serve as a modality-invariant domain adapter, enabling a single sparse-feature model to generalize across diverse modality combinations.
3. MI-RAFT Model: A novel optical flow architecture (MI-RAFT) is introduced for fine registration, featuring a new design for fusing pretrained vision features and a synthetic dual-pair training strategy with CLFA.
4. New Dataset: The authors collected and released a new dataset of paired Fundus Autofluorescence (FAF) and Multicolor (MC) images to address the lack of registration data for these modalities.
5. Superior Performance: The method achieves state-of-the-art performance across multiple mono-modal and cross-modal datasets without requiring dataset-specific fine-tuning.

4. Experimental Results

The framework was evaluated on four datasets: FIRE (CF-CF), CF-IR, CF-FA, and the newly collected FAF-MC.

• Coarse Alignment: The proposed URVSM-driven method outperformed state-of-the-art sparse correspondence methods (e.g., SuperPoint, SuperGlue, SuperRetina) and modality-invariant baselines (MIF-Net, MINIMA) across all datasets. It achieved the highest mean Dice scores, particularly in challenging cross-modal scenarios like CF-FA and FAF-MC.
• Fine Alignment: MI-RAFT (using DINOv3 features) significantly outperformed fine-tuned baselines, including generic RAFT, CrossRAFT, MCM-RAFT, and retina-specific methods like GAMorph. It achieved the highest Dice and soft-Dice scores on all datasets.
• Ablation Studies:
  • The CLFA strategy and intensity perturbations (illumination/focus) consistently improved performance.
  • Fusing features before correlation computation was found to be more effective and computationally efficient than fusing correlation volumes or correlation-lookup features.
  • The pretrained outlier-rejection network proved sufficient without further fine-tuning.
• Robustness: The method demonstrated strong generalization to unseen modality combinations (e.g., FAF-MC) where other methods failed or performed poorly.

5. Significance and Claims

The authors claim that this work provides a generalizable foundation for multimodal retinal image analysis. By eliminating the need for separate models or fine-tuning for every modality pair, the proposed framework addresses the real-world need for a flexible solution in clinical and research settings.

• Practicality: The ability to handle diverse mono-modal and cross-modal combinations with a single model reflects the complexity of real-world ophthalmic imaging.
• Data Efficiency: The synthetic training strategy allows for supervised learning of dense flow without the prohibitive cost of collecting large paired datasets with ground-truth flow.
• Scope: The authors explicitly note that the current framework is designed for retinal images with comparable fields of view (30–45°). It does not currently address extreme differences in field of view (e.g., 135° ultra-widefield vs. 30° fundus), which remains a challenge for future work.

The paper concludes that by reducing the dependency on paired training data and modality-specific models, this approach facilitates more efficient co-localization of multimodal information, longitudinal disease monitoring, and other retinal image analysis applications.