Structure-to-Image: Zero-Shot Depth Estimation in Colonoscopy via High-Fidelity Sim-to-Real Adaptation

This paper proposes a Structure-to-Image paradigm for zero-shot colonoscopic depth estimation that leverages phase congruency and cross-level structure constraints to bridge the sim-to-real domain gap, achieving a 44.18% reduction in RMSE compared to existing methods.

The Gist

The Big Picture: Fixing the "Uncanny Valley" of Colonoscopies

Imagine you are trying to teach a robot how to navigate a dark, winding cave (the human colon) using a map.

• The Problem: The only maps you have are drawn by a child using crayons (simulated data). They look okay from a distance, but up close, the colors are wrong, the textures are flat, and the lighting is weird. If you send the robot into the real cave using just these crayon maps, it gets confused, bumps into walls, and misses important things like hidden rocks (polyps).
• The Goal: You need to turn those crayon maps into a hyper-realistic, 3D movie of the real cave so the robot can learn properly.

The Old Way: "Image-to-Image" Translation (The Broken Translator)

Previously, researchers tried to use AI to translate the "crayon map" directly into a "real photo." They would say to the AI: "Take this fake depth map and make it look real, but don't change the shape."

Think of this like asking a translator to translate a book into another language while strictly forbidding them from changing the sentence structure, even if the grammar doesn't make sense in the new language.

• The Result: The AI gets confused. It tries to keep the shape but also add realistic details (like blood vessels or shiny wet spots). It ends up creating a "Frankenstein" image: the shape is slightly warped, and the shiny spots look like plastic rather than wet tissue.
• The Consequence: When the robot (the depth estimation model) tries to learn from these broken images, it learns the wrong lessons. It thinks a shiny reflection is a bump in the wall, leading to errors.

The New Way: "Structure-to-Image" (The Architect)

The authors of this paper flipped the script. Instead of asking the AI to "guess the shape and the look at the same time," they said: "Here is the perfect blueprint (the structure). Now, just paint the realistic details on top of it."

They call this "Structure-to-Image."

How it works (The Analogy):

Imagine you are a master painter.

1. The Blueprint (Depth Map): You have a perfect, 3D architectural drawing of a house. You know exactly where the walls, windows, and stairs are.
2. The Painting (Real Image): Your job is to paint the house. You don't need to guess where the walls go; you just need to decide what color the bricks are, how the light hits the glass, and where the moss grows.

By treating the depth map as the foundation rather than a rule, the AI stops making mistakes about the shape. It focuses entirely on making the texture look real.

The Secret Sauce: Two Special Tools

To make sure the painting looks truly real (not just a cartoon), the authors added two special tools to their AI:

1. The "Phase Congruency" Lens (The Micro-Texture Detective)

• The Problem: Standard AI is good at seeing big shapes (like a wall) but bad at seeing tiny details (like the veins in a leaf or the texture of skin).
• The Solution: They used a mathematical trick called Phase Congruency. Imagine looking at a photo through a special pair of glasses that highlights edges and patterns regardless of how bright or dark the light is.
• Why it helps: This ensures the AI doesn't just paint a smooth, fake-looking wall. It forces the AI to paint the tiny, complex network of blood vessels and the rough texture of the colon lining, making it look exactly like a real human organ.

2. The "Normal" Compass (The 3D Alignment)

• The Problem: Sometimes the AI paints a bump that looks right from the front but is actually flat from the side.
• The Solution: They added a "Normal Consistency" check. Think of this as a compass that checks the angle of every tiny surface.
• Why it helps: It ensures that if the blueprint says a fold in the colon is steep, the realistic image shows a steep fold, not a gentle slope. It keeps the 3D geometry honest.

The Results: A Massive Leap Forward

The researchers tested this new method on a "phantom" dataset (a fake colon made of plastic used for testing).

• The Old Way: The robot made big mistakes, missing details or misjudging distances.
• The New Way: The robot became incredibly accurate.
• The Stat: They reduced the error rate by 44% compared to the best existing methods.

Why This Matters

In colonoscopies, missing a polyp (a small growth that can become cancer) is a huge problem. Current AI tools often miss them because they are trained on fake data that doesn't look real enough.

By using this "Structure-to-Image" method, doctors can train AI systems on realistic, high-quality data without needing to take thousands of real patient photos (which is hard to do because of privacy and the lack of "ground truth" depth in real scans).

In short: They stopped trying to guess the shape and started using the shape as a solid foundation to build a perfect, realistic picture. This makes the AI smarter, safer, and better at saving lives.

Technical Summary

1. Problem Statement

Monocular Depth Estimation (MDE) is critical for creating 3D maps during colonoscopy to reduce polyp miss-rates. However, training MDE models on real-world data is impossible due to the lack of ground-truth depth labels. Consequently, models are trained on synthetic data, creating a significant domain gap between simulated and real images.

• Current Limitations: Existing Sim-to-Real adaptation methods (primarily CycleGAN-based) treat depth maps as posterior constraints (i.e., trying to preserve depth after or during image translation). This approach often fails to balance realism with structural consistency, leading to:
  • Structural distortions (e.g., warped folds).
  • Specular artifacts (fake highlights).
  • Inability to capture fine-grained micro-structures (e.g., vascular patterns) while maintaining macro-geometry.

2. Methodology: Structure-to-Image (S2I)

The authors propose a paradigm shift from "Image-to-Image" to "Structure-to-Image." Instead of treating depth as a constraint to be preserved, they treat the depth map as an active generative foundation.

Core Framework

The system utilizes a unified CycleGAN framework trained on unpaired real images and synthetic depth maps. It operates via two branches:

1. Image-to-Depth: Generates accurate depth maps from real images (used to create paired data).
2. Depth-to-Image (The Core): Generates realistic colonoscopy images directly from depth maps.
   • Input Transformation: Synthetic 16-bit depth maps are converted to inverse depth maps to handle stair-step artifacts common in simulation.
   • Goal: The generator learns to synthesize realistic textures and lighting based strictly on the provided structural geometry, reducing learning uncertainty.

Key Technical Innovations

To ensure the generated images are both geometrically accurate and rich in micro-details, the authors introduce a Cross-Level Structure Constraint composed of two specific loss functions:

1. Phase Congruency (PC) Loss:

   • Motivation: Traditional edge detectors fail to distinguish between shadows and tissue or miss fine vascular textures. Phase Congruency identifies structural features where Fourier components reach maximum consistency, making it robust to lighting changes.
   • Function: It enforces similarity between the generated image and real images in the frequency domain, specifically targeting micro-structures (vascular patterns) while preserving macro-geometric contours.
   • Implementation: Uses multi-scale Log-Gabor filters to compute local energy and amplitude, calculating a loss based on the similarity of PC maps and gradient magnitudes.

2. Normal Consistent Loss:

   • Function: Aligns the surface normals of the simulated depth map with the reconstructed depth map derived from the generated image.
   • Goal: Ensures fine-grained geometric consistency, preventing the generator from "flattening" or distorting the 3D shape of folds and polyps.

3. Key Contributions

1. New Paradigm: Proposes the "Structure-to-Image" generation strategy, elevating depth from a passive constraint to the primary generative prior.
2. Novel Constraint: Introduces a Cross-Level Structure Constraint combining Phase Congruency (for micro-textures) and Normal Consistency (for macro-geometry) to co-optimize realism and structural fidelity.
3. First Application of PC: Marks the first introduction of Phase Congruency to the domain of colonoscopic Sim-to-Real adaptation.
4. Zero-Shot Performance: Demonstrates that MDE models fine-tuned on their generated data achieve superior zero-shot performance on real-world datasets without requiring real depth labels.

4. Experimental Results

The method was evaluated on a publicly available phantom dataset (C3VD) and real-world datasets (Colon10K, Colon-Ours).

• Image Generation Quality:

  • Outperformed state-of-the-art methods (XDCycleGAN, Struct-Preserve, Sim2Real) in PSNR (20.65), SSIM (0.74), and Inception Score (3.47).
  • Qualitative results showed the elimination of structural distortions and specular artifacts while successfully synthesizing realistic vascular textures.

• Image-to-Depth Accuracy:

  • The unified framework achieved depth estimation performance comparable to specialized models like NormDepth, with an RMSE of 7.53 mm on the C3VD phantom dataset.

• Downstream Zero-Shot Depth Estimation (Main Result):

  • A pre-trained MDE model (DepthAnythingV2-small) was fine-tuned on the generated data and tested zero-shot on real/phantom data.
  • Performance Gain: The proposed method reduced the RMSE by 44.18% compared to the Sim2Real baseline and 25.95% compared to the supervised baseline.
  • Comparison: It significantly outperformed all competing methods, proving that high-fidelity structural generation directly translates to better depth estimation accuracy.

5. Significance and Impact

• Clinical Relevance: By bridging the Sim-to-Real gap without requiring real depth ground truth, this method enables the training of robust 3D reconstruction tools for colonoscopy. This can directly improve polyp detection rates and reduce miss-rates.
• Paradigm Shift: The work challenges the standard "constraint-based" approach in domain adaptation, suggesting that treating structural data as a generative prior yields superior results in medical imaging where geometry is paramount.
• Open Source: The code is publicly available, facilitating further research in medical domain adaptation.

Limitation & Future Work: The current method relies on predicted depth, which can introduce bias. Future work aims to build smoother synthetic datasets and develop controllable vascular texture generation to further refine the realism.