Technical Summary: Geometry-Consistent Endoscopic Representations for Image-Guided Navigation via Structured Foundation Model Adaptation

1. Problem Statement

Accurate vision-based navigation in monocular endoscopy (e.g., bronchoscopy, sinus surgery, colonoscopy) is hindered by the lack of depth cues, weak tissue textures, non-rigid deformations, and significant appearance variations across different clinical domains. These factors complicate pose estimation, depth prediction, and image-to-anatomy alignment. While recent vision foundation models (FMs) offer powerful pre-trained representations, they often lack geometry-consistent features necessary for stable spatial reasoning. Furthermore, adapting these large models to endoscopic scenes is difficult due to the scarcity of large-scale, labeled clinical datasets containing precise geometric supervision (e.g., camera poses, 3D structure, dense depth). Existing methods either focus on semantic perception without enforcing geometric consistency or rely on domain-specific pretraining that remains sensitive to cross-domain shifts in hardware and illumination.

2. Methodology

The authors propose a unified framework to learn geometry-consistent and domain-robust image representations through three core components:

A. Synthetic Data Pipeline with Multi-Level Domain Randomization

To overcome the lack of clinical geometric labels, the framework utilizes a purpose-built synthetic dataset generated from CT-derived anatomical models.

• Generation: Virtual endoscopic views are rendered along sampled camera trajectories to produce paired images with ground-truth depth and relative poses.
• Augmentation: To bridge the sim-to-real gap, the pipeline employs CycleGAN for realistic appearance translation and includes four image modalities (RGB, synthetic RGB, rendered depth, and off-the-shelf disparity predictions).
• Randomization: Training involves extensive randomization, including photometric transformations, blur, noise, thin-plate spline deformations, and pair-level reshuffling to ensure domain invariance.

B. Hierarchy-Aware Geometry–Semantic Adaptation (HGSA)

Instead of uniformly fine-tuning the entire foundation model, the authors introduce HGSA, a structured adaptation strategy that inserts low-rank adapters (LoRA) selectively into specific layers of a pretrained Vision Transformer.

• Hierarchical Search: The framework employs a coarse-to-fine search to determine the optimal adapter configuration, including:
  1. Layer Placement: Identifying which transformer layers ($L_a$) benefit most from adaptation (e.g., layers 5–12 were found optimal).
  2. Target Modules: Selecting which attention projections ($V, QV, QVK, QVKO$) to modify.
  3. Capacity: Tuning the adapter rank ($r$) and scaling factor ($\alpha$).
• Rationale: This approach aligns adaptation with the transformer's inherent hierarchy, encouraging intermediate layers to learn geometric correspondences while preserving semantic abstraction in deeper layers.

C. Hierarchy-Aware Supervision Objectives

The training objective combines three loss terms to enforce both geometric consistency and semantic robustness:

1. Geometry-Aware Feature Correspondence ($L_{ms}$): Applied to intermediate features, this loss uses a multi-scale dense token loss combining PatchNCE and cosine reprojection loss. It warps features from one view to another using ground-truth flow fields derived from synthetic depth, enforcing spatial consistency.
2. Semantic-Aware Representation Regularization ($L_{InfoNCE}$): Applied to global tokens in deeper layers, this instance-level contrastive loss aligns scene-level embeddings from paired views, ensuring semantic consistency across domains.
3. Teacher Distillation ($L_{Gram}$): A Gram-based regularization using a frozen DINO teacher model to preserve the correlation structure of patch features, preventing the model from over-focusing on global semantics at the expense of spatial fidelity.

3. Key Contributions

The paper claims the following contributions:

• Unified Pipeline: An end-to-end framework integrating a synthetic geometric dataset, the HGSA mechanism, and hierarchy-aware, layer-wise objectives to learn endoscopic representations that are both geometrically consistent and domain-robust.
• HGSA Efficacy: Controlled experiments demonstrating that HGSA is an effective adaptation strategy that improves representation quality. The resulting representations support downstream navigation tasks (pose and depth estimation) and remain useful across different endoscopic domains.
• Generalization and Scalability:
  • Sim-to-Real Transfer: Demonstrated strong transfer from synthetic bronchoscopy data to real patient videos without additional encoder adaptation.
  • Cross-Domain Transfer: Showed the ability to adapt bronchoscopy-trained encoders to sinus and colonoscopy datasets under limited supervision.
  • Scalability: Validated that the framework scales favorably with larger backbone models (ViT-S to ViT-L) and increased training data sizes.

4. Experimental Results

The framework was evaluated on bronchoscopy, sinus, and colonoscopy datasets using linear probing and downstream navigation tasks.

• Representation Quality (Probing):

  • HGSA configurations (specifically Cfg 7.4.8: layers 5–12, QVKO modules, rank 32, scale 64) achieved the highest combined probing scores ($S_{probe}$), balancing semantic accuracy (scene classification) and geometric accuracy (depth estimation).
  • Ablation studies confirmed that removing any component (HGSA, warp supervision, contrastive losses) degraded performance, with $L_{InfoNCE}$ being critical for semantic separability and $L_{ms}$ for geometric consistency.
  • Token-level geometric correspondence metrics (PCK@1, Cosine Similarity) showed the proposed method outperformed both vanilla DINO and full LoRA baselines.

• Downstream Navigation Tasks:

  • Pose Estimation: On real bronchoscopy videos, the proposed encoder achieved superior rendering-image alignment (NC, DS, SI metrics) compared to baselines, indicating more accurate pose optimization.
  • Depth Estimation: The method produced more structurally consistent depth maps.
  • Cross-Domain Transfer: When adapted to sinus and colonoscopy with limited target data, the framework consistently outperformed vanilla and full LoRA baselines in depth estimation. Pose estimation improvements were significant for colonoscopy but more modest for sinus, attributed to the limited size of the sinus dataset for fine-tuning.

• Scalability: Performance improved monotonically with larger backbones (ViT-L > ViT-B > ViT-S) and larger training data subsets (100% > 50% > 25%), suggesting no early saturation.

5. Significance and Claims

The authors position this work as a practical approach for endoscopic representation learning that addresses the critical gap between general-purpose foundation models and the specific geometric requirements of surgical navigation.

• Practical Utility: The framework provides a scalable pathway to robust navigation tools without requiring dense clinical annotations, leveraging synthetic data and structured adaptation.
• Modest Claims on Generalization: The authors explicitly note that while the framework shows promising transferability to sinus and colonoscopy, the results are based on limited target-domain adaptation rather than zero-shot generalization. They acknowledge that performance variations (e.g., in sinus endoscopy) highlight the need for additional data or targeted adaptation in specific domains.
• Core Insight: The paper argues that "hierarchy-aware, geometry-guided adaptation" is essential for transforming generic visual features into representations capable of supporting stable, geometry-consistent reasoning in the challenging environment of endoscopy.