🤖 AI

Boosting Generalizable Depth Estimation in Endoscopy by Mixture of Lightweight Experts and Intrinsic Image Alignment

This paper proposes EndoMINI, a novel self-supervised framework for generalizable endoscopic depth estimation that combines a mixture of low-rank experts (MiLoRE) for parameter-efficient adaptation and intrinsic image alignment (IIA) to mitigate illumination interference, demonstrating superior performance on both supervised and zero-shot benchmarks.

Original authors: Liangjing Shao, Beilei Cui, Yiming Huang, Changjing Liu, Hongliang Ren

Published 2026-08-04

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Liangjing Shao, Beilei Cui, Yiming Huang, Changjing Liu, Hongliang Ren

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to navigate a dark, wet cave using only a flashlight. If the light bounces off the wet walls in weird ways, or if the cave looks different every time you turn a corner, it becomes incredibly hard to tell how far away the walls are. This is exactly the challenge doctors face during minimally invasive surgeries, where tiny cameras (endoscopes) are used inside the human body. The inside of the body is a tricky environment: tissues are shiny, lights can reflect wildly, and every patient's anatomy looks a bit different. To help robots and surgeons "see" in 3D, scientists use a technique called depth estimation, which is basically a computer guessing how far away everything is just by looking at a flat picture. While computers are getting good at this in the outside world (like for self-driving cars), they often get confused inside the body because the lighting is so inconsistent and the textures are so unique.

This paper introduces a new, clever system called EndoMINI designed to help these cameras understand depth better, even when the lighting is messy or the scene changes. The researchers built this system using two main tricks. First, they created a "team of specialists" instead of relying on one general brain. Think of it like a school where, instead of one teacher trying to teach every subject to every student, you have a smart switch that picks the best teacher for the specific lesson—whether it's math, art, or history. In the computer's brain, this means it can instantly switch to the right "expert" mode depending on whether it's looking at a shiny liver or a dull intestine. Second, they taught the computer to ignore the glare. Just like how you might squint to see past a reflection on a window, this system learns to separate the "color" of the tissue from the "shine" of the light, allowing it to measure distance accurately even when the light is bouncing around.

The team tested their new EndoMINI system on several datasets, including one called SCARED, and found that it was better at guessing distances than previous methods, both when it had been trained on similar images and when it had to guess on completely new, unseen videos (a test called "zero-shot"). They also showed that it could figure out how the camera was moving and even predict the camera's own settings without being told beforehand. The results suggest that by using this mix of specialized experts and by ignoring the confusing glare, the system can provide a much clearer 3D view for surgeons, potentially making robotic surgery safer and more precise.

The Problem: Why Endoscopic Depth is Tricky

Depth estimation is the art of figuring out how far away things are from a camera. In the outside world, computers have gotten pretty good at this, but inside the human body, it's a nightmare. The paper points out two main villains:

The "Shiny" Problem: Tissues inside the body are wet and reflective. Light bounces off them in unpredictable ways, making a flat surface look like a bump or a hole.
The "Different Every Time" Problem: Every patient is different, and even different parts of the same patient look unique. A model trained to recognize one type of tissue might get completely confused by another.

Existing methods tried to fix this, but they often struggled to adapt to these new scenes or to ignore the blinding glare of the lights.

The Solution: A Team of Experts and a Glare Filter

The authors propose EndoMINI, a new framework that tackles these issues with two innovative approaches.

1. The Mixture of Low-Rank Experts (MiLoRE)
Imagine a library where you usually have to ask one librarian for help. If you ask about a rare book, they might not know the answer. Now, imagine a library with a smart robot that knows exactly which specialist to call. If you ask about history, it calls the history expert; if you ask about science, it calls the science expert.

In EndoMINI, the computer uses a "Mixture of Low-Rank Experts" (MiLoRE). Instead of having one giant, rigid brain that tries to learn everything, the system has a "router" that looks at the image and decides which small, specialized "expert" modules to activate.

How it works: The system uses a technique called LoRA (Low-Rank Adaptation), which is like adding a small, adjustable layer of "training wheels" to a pre-existing smart brain. The MiLoRE system has several of these small experts. When the camera sees a specific type of tissue, the router picks the best expert (or a mix of them) to handle that specific scene.
The Result: This allows the model to adapt quickly to different endoscopic scenes without needing to be completely retrained from scratch. It's efficient and flexible.

2. Intrinsic Image Alignment (IIA)
Now, let's talk about the glare. When you take a photo of a shiny object, the bright spot makes it hard to see the object's true shape. The paper introduces a way to separate the "true color" of the object from the "shine" of the light.

The Analogy: Think of a painting. The paint itself is the "reflectance" (the true color of the tissue), and the light hitting it is the "shading." The paper's system learns to peel these two apart. It uses a special network to decompose the image into a "reflectance map" (what the tissue actually looks like) and a "shading map" (where the light is hitting).
The Magic: By aligning the "reflectance" maps between different video frames, the system can calculate depth without being confused by the changing lights. It essentially says, "Ignore the bright spot; look at the actual color underneath."

What They Found

The researchers tested EndoMINI on three different datasets: SCARED, Hamlyn, and SERV-CT. These datasets contain real videos from robotic surgeries.

Supervised Performance: On the SCARED dataset, where the model was trained with known answers, EndoMINI beat all other state-of-the-art methods. For example, it achieved an Absolute Relative Error (RelAbs) of 0.047, which is lower (better) than the next best method, EndoDAC, which had 0.052.
Zero-Shot Performance: This is the real test. The model was trained on SCARED and then asked to guess depths on the Hamlyn and SERV-CT datasets without any extra training. EndoMINI still came out on top, showing it generalizes well to new environments. On the Hamlyn dataset, it achieved a RelAbs of 0.140, beating the previous best of 0.143 (DVSMono).
Ego-Motion and Camera Settings: The system didn't just guess depth; it also figured out how the camera was moving (ego-motion) and even predicted the camera's internal settings (intrinsics) with high accuracy, outperforming methods that relied on pre-provided settings.

Why It Matters

The paper concludes that by combining a flexible "team of experts" (MiLoRE) with a smart way to ignore glare (Intrinsic Image Alignment), EndoMINI creates a much more reliable 3D view for endoscopic surgery. This isn't just a small tweak; it's a significant step toward making robotic surgery safer and more precise, as the robot can finally "see" the world inside the body as clearly as a human surgeon would hope. The authors suggest that this high-quality 3D perception could be a game-changer for minimally invasive procedures, helping surgeons navigate complex anatomy with confidence.

Technical Summary: EndoMINI – Boosting Generalizable Depth Estimation in Endoscopy

Problem Statement
Depth estimation is critical for 3D perception and navigation in minimally invasive and robotic endoscopic surgeries. While self-supervised learning frameworks have become mainstream for natural scenes, they face significant challenges when applied to endoscopic environments. The primary obstacles include:

Feature Diversity: Endoscopic scenes exhibit vast variations in tissue characteristics, even within the same dataset, causing standard feature encoders to struggle with representation extraction across different scenes.
Illumination Inconsistency: Variations in lighting and light reflectance on tissue surfaces severely degrade the accuracy of depth and ego-motion estimation. Existing methods that rely on appearance and optical flow often fail to fully mitigate the impact of reflectance under varying illumination conditions.

Methodology: EndoMINI
The authors propose EndoMINI, a novel self-supervised framework designed to enhance model generalization and robustness in endoscopic scenes. The architecture builds upon a pretrained Vision Transformer (ViT) encoder with convolutional blocks and a DPT-like decoder, integrating two core innovations:

Mixture of Low-Rank Experts (MiLoRE):
To address feature diversity, the framework employs a parameter-efficient fine-tuning (PEFT) strategy. Instead of a single adaptation layer, MiLoRE utilizes a Mixture of Experts (MoE) framework where Low-Rank Adaptation (LoRA) layers serve as individual experts.
- Mechanism: A router network generates a probability distribution over experts based on input features. The top- $k$ experts are selected to perform weighted LoRA adaptation on the frozen weights of the pretrained ViT encoder.
- Implementation: MiLoRE modules are inserted into each Transformer block, specifically targeting the "QKV Linear" (query, key, value) projections and the two Multi-Layer Perceptrons (MLPs) within the Feed-Forward Network (FFN).
- Robustness: A noise vector is integrated into the router's prediction to enhance robustness.
Intrinsic Image Alignment (IIA):
To tackle illumination inconsistency, the framework introduces an intrinsic image decomposition network and corresponding alignment losses.
- Decomposition: A self-supervised network decomposes images into reflectance (albedo) and shading components. To handle style gaps, the network uses Adaptive Instance Normalization (AdaIN) controlled by style vectors generated via MLPs.
- Alignment Losses:
  - Illumination-Free Alignment: Since reflectance is independent of lighting, the framework warps the source reflectance image to the target view and minimizes the photometric loss between it and the target's decomposed reflectance.
  - Intrinsic Reconstruction Alignment: The framework reconstructs the target image by combining the warped reflectance and shading images, aligning this with the actual target image (corrected for appearance flow) to optimize depth and ego-motion.

The total loss function combines intrinsic image decomposition losses, illumination-free alignment, reconstruction alignment, and smoothness constraints.

Key Contributions

MiLoRE Framework: A novel PEFT module combining MoE and LoRA to boost model adaptation to endoscopic scenes with diverse tissue characteristics.
Intrinsic Image Alignment: A self-supervised approach integrating intrinsic image decomposition into the training loop to alleviate the influence of varying light conditions and surface reflectance.
Performance: The proposed method achieves state-of-the-art results in both supervised and zero-shot depth estimation scenarios, while also providing accurate ego-motion estimation and camera intrinsics prediction without relying on provided camera parameters.

Experimental Results
The framework was evaluated on three public datasets: SCARED (for supervised training and evaluation), Hamlyn, and SERV-CT (for zero-shot evaluation).

Supervised Depth Estimation (SCARED): EndoMINI outperformed state-of-the-art methods (including Depth Anything v2, IID-SfMLearner, and EndoDAC) across all metrics (RelAbs, RelSq, RMSE, RMSELog, and $\delta$ ). Notably, it achieved a RelAbs of 0.047 and RMSE of 4.246, with only 3.6M trainable parameters.
Zero-Shot Depth Estimation: The model demonstrated superior generalization on the Hamlyn and SERV-CT datasets, achieving the lowest error rates compared to models trained on SCARED.
Ego-Motion and Intrinsics: The framework provided the most accurate ego-motion estimation and camera intrinsics prediction among compared methods, even without ground-truth intrinsics.
Ablation Studies: Experiments confirmed that the specific configuration of MiLoRE (3 experts for QKV, 5 for FFN) and the inclusion of Intrinsic Image Alignment were critical for performance gains.

Significance
The paper claims that EndoMINI significantly advances the state of the art in endoscopic depth estimation by effectively addressing the dual challenges of scene diversity and illumination inconsistency. By leveraging a mixture of lightweight experts and intrinsic image decomposition, the method offers high generalizability and robustness. The authors posit that this work promotes the development of minimally invasive surgery, particularly in robot-assisted procedures, by enabling high-quality 3D perception in complex and variable surgical environments.

The Problem: Why Endoscopic Depth is Tricky

The Solution: A Team of Experts and a Glare Filter

What They Found

Why It Matters

Technical Summary: EndoMINI – Boosting Generalizable Depth Estimation in Endoscopy

More like this