🤖 AI

CoGE: Sim-to-Real Online Geometric Estimation for Monocular Colonoscopy

This paper presents CoGE, a novel framework for online monocular geometric estimation in colonoscopy that leverages an illumination-aware supervision module and a structure-aware perception module to achieve state-of-the-art performance on real-world data while being trained solely on simulated data.

Original authors: Liangjing Shao, Beilei Cui, Hongliang Ren

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Liangjing Shao, Beilei Cui, Hongliang Ren

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to navigate a dark, narrow, and twisting tunnel using only a single flashlight. That is essentially what a surgeon faces during a colonoscopy. The camera is a tiny, one-eyed eye looking into a long, closed tube. To move safely and find the right path, the surgeon needs to understand the 3D shape of the tunnel—how deep it is, where the walls curve, and where obstacles are.

This paper introduces CoGE, a new "smart assistant" for this camera. Its job is to instantly turn the flat, 2D video from the camera into a 3D map of the inside of the colon.

Here is how CoGE works, explained through simple analogies:

1. The Big Problem: The "Training Gap"

Usually, to teach a computer to see in 3D, you need to show it thousands of real videos where you already know exactly what the 3D shape looks like (the "ground truth"). But in a real colon, you can't easily measure the exact 3D shape because the space is too tight and closed.

So, the researchers taught CoGE using simulated data (a perfect, computer-generated video game version of a colon).

The Challenge: A video game colon looks very different from a real one. Real ones have weird lighting, shiny spots (glare), and messy textures. It's like teaching a driver to drive on a sunny, perfect track, but then asking them to drive in a rainy, foggy city immediately after. Usually, the driver gets confused.

2. The Solution: Three Special "Glasses"

CoGE is special because it can take that training from the "perfect video game" and apply it to the "messy real world" without needing extra training on real data. It does this using three clever tricks:

The "Wavelet" Glasses (Seeing the Skeleton):
Imagine looking at a photo of a colon. Some parts are smooth curves (low frequency), and some are tiny wrinkles or edges (high frequency).
CoGE uses a technique called Wavelet Decomposition. Think of this as putting on special glasses that separate the image into its "skeleton" (the big shapes) and its "details" (the tiny wrinkles). It learns that the skeleton of a colon looks the same whether it's in a video game or real life, even if the lighting is different. This helps it recognize the structure regardless of the messiness.
The "Light Detective" Glasses (Ignoring Glare):
Real colonoscopy videos are tricky because the light bounces off wet surfaces, creating bright white spots (specular reflections) that confuse the camera.
CoGE uses a module based on Retinex theory (a way of understanding how our eyes separate color from light). It acts like a detective that says, "That bright spot is just a glare, not a real wall." It tells the system, "Don't trust the depth calculation in this bright spot; it's just a reflection." This stops the system from getting fooled by the lighting.
The "Memory Filter" (Forgetting the Past):
As the camera moves through the long colon, it builds up a memory of what it has seen. But sometimes, the memory gets cluttered with old, irrelevant information that doesn't match the current view.
CoGE has a Memory Forget Mechanism. Imagine you are walking through a hallway and looking back at a door you passed 10 seconds ago. If that door is now blocked by a new wall, your brain should "forget" the old door to focus on the new wall. CoGE does this mathematically: it checks its memory, sees what is no longer relevant, and deletes it so it doesn't get confused.

3. The Results: "Game-Perfect" Training, "Real-World" Performance

The researchers tested CoGE and found something amazing:

They trained it only on the simulated (video game) data.
They tested it on real colonoscopy videos (which it had never seen before).
The Outcome: CoGE performed better than other top-tier methods that were trained on huge amounts of real data. It could create accurate 3D maps and depth estimates in real-time (over 15 frames per second), which is fast enough for a surgeon to use while moving.

Summary

In short, CoGE is a system that learns to see 3D depth in a colon by studying a perfect simulation. It uses special "glasses" to ignore bad lighting, separate important shapes from noise, and clean up its memory. This allows it to work perfectly in the messy, real world without needing expensive, difficult-to-get real-world training data.

Technical Summary: CoGE – Sim-to-Real Online Geometric Estimation for Monocular Colonoscopy

Problem Statement

Geometric estimation, encompassing depth estimation and scene reconstruction, is vital for providing surgeons with 3D spatial perception and navigation during colonoscopy. However, acquiring geometric ground truth (depth maps, point clouds, camera parameters) in real-world colonoscopy is prohibitively difficult and costly due to the narrow, enclosed nature of the colon. While self-supervised methods exist, they often underperform compared to supervised models. Conversely, supervised models typically require large datasets with ground truth, which are scarce in realistic settings. Although simulation software (e.g., VR-Caps) can generate geometric ground truth, a significant "feature gap" exists between simulated data and realistic data due to artifacts and illumination diversity. Furthermore, existing sim-to-real approaches often rely on style translation networks to preprocess realistic data, which reduces inference efficiency and may degrade cross-scene performance.

Methodology

The authors propose CoGE, a foundation model-based framework designed for online monocular geometric estimation that is trained solely on simulated data but performs effectively on realistic scenes. The framework operates on a pair of observations (cached previous frame $I_{t-1}$ and current frame $I_t$ ) to output point maps, camera poses, illumination maps, and confidence maps.

1. Structure-Aware Perception (SAP)

To address the feature gap and extract common structural features across simulated and realistic scenes, the authors embed a Structure-Aware Perception block based on wavelet transformation.

Decomposition: Input images are decomposed via Discrete Wavelet Transformation (DWT) into four sub-bands: Low-Low (approximation), Low-High (horizontal details), High-Low (vertical details), and High-High (diagonal details).
Feature Extraction: The model utilizes Window-based Multi-Head Self-Attention (W-MHSA) to extract features from the low-frequency (Low-Low) component, while employing specific convolutional kernels ( $7\times1$ , $1\times7$ , and $3\times3$ ) to extract high-frequency local details from the other bands.
Fusion: Features from different wavelet compositions are fused and reconstructed via Inverse DWT to form the final structural representation.

2. Illumination-Aware Supervision (IAS)

To handle the diverse and unstable illumination conditions in the colon, a self-supervised Illumination-Aware Supervision module is introduced.

Theory: Based on Retinex theory, the module assumes the intrinsic component of an image is invariant to illumination in the gradient domain.
Mechanism: The IAS module generates a light influence map ( $L_t$ ) and an intrinsic graph ( $A_t$ ). During training, $L_t$ is used to adjust the confidence map ( $C_t$ ) of the estimated results, effectively controlling the model's confidence based on light distribution to mitigate illumination interference.

3. Memory Forget Mechanism

Given the long sequences typical in colonoscopy, the framework employs a memory system with a forget mechanism to manage redundant information.

Reading: Previous features retrieve relevant information from key ( $M_k$ ) and value ( $M_v$ ) memory caches.
Forgetting: A threshold-based mechanism filters memory tokens. If a memory token has attention weights below a specific threshold (5e-4) for less than 5% of feature tokens, it is deemed irrelevant and deleted. This prevents the accumulation of redundant memory that could interfere with current observations.
Updating: New memory is encoded from memory-related features and concatenated with the filtered cache.

4. Architecture Overview

The backbone utilizes six ViT-L blocks followed by the SAP block. The decoder consists of six blocks utilizing self-attention and cross-attention to process current and memory-related features. Downstream heads (DPT) output the final point maps, camera poses, and confidence maps.

Key Contributions

Structure-Aware Perception: A novel block utilizing wavelet transformation to extract common structural and local features (edges) shared between simulated and realistic colonoscopy scenes.
Illumination-Aware Supervision: A module that adjusts the confidence of geometric estimates based on light distribution, addressing the challenge of diverse illumination on the colon surface.
Memory Forget Mechanism: A strategy to filter redundant and irrelevant memory tokens, reducing interference from past observations in long sequences.
Sim-to-Real Performance: A model trained exclusively on simulated data that achieves state-of-the-art performance on both unseen simulated scenes and realistic colonoscopy datasets without requiring style translation preprocessing.

Experimental Results

The model was trained on 24 simulated sequences from VR-Caps (SimCol3D) and evaluated on realistic datasets (C3VD, C3VDv2) and in-house images.

Monocular Depth Estimation: On the SimCol dataset, CoGE achieved an Abs Rel of 0.027 and a $\delta < 1.25$ accuracy of 0.995, outperforming other foundation models like STream3R, MonST3R, and CUT3R. On realistic datasets (C3VD and C3VDv2), it consistently achieved the lowest error rates (e.g., Abs Rel 0.083 on C3VD) and highest accuracy metrics compared to SOTA methods.
3D Reconstruction: Using mean Euclidean Distance (mED) and the ratio of points within a specific error threshold ( $\delta < N$ ), CoGE demonstrated superior reconstruction accuracy. For instance, on C3VD, it achieved an mED of 3.957 and a $\delta < 4.0$ ratio of 0.645, significantly outperforming competitors like STream3R and CUT3R.
Ablation Studies: Removing the SAP block, IAS module, or Memory Forget mechanism resulted in measurable performance degradation across all datasets, confirming the necessity of each component.

Significance and Claims

The paper claims that CoGE represents a significant advancement in sim-to-real geometric estimation for endoscopy. By training solely on simulated data, the method bypasses the need for costly and difficult-to-obtain real-world geometric ground truth. The framework successfully bridges the domain gap through specific architectural innovations (wavelet-based structure extraction and illumination-aware confidence control) rather than relying on inefficient style transfer preprocessing.

The authors note that while the model achieves real-time inference (>15 FPS), the inclusion of the structure-aware perception blocks slightly decreases efficiency compared to lighter models. Additionally, they acknowledge that performance may degrade in the presence of motion blur, acute dynamic changes, or specific lesion tissues, which remain challenges for future work. The primary contribution is demonstrating that high-quality, online 3D perception in colonoscopy is achievable using only simulated training data when the model is explicitly designed to handle structural commonalities and illumination variance.