💻 computer science

Iris: Bringing Real-World Priors into Diffusion Model for Monocular Depth Estimation

The paper proposes Iris, a deterministic framework for monocular depth estimation that integrates real-world priors into diffusion models via a two-stage Priors-to-Geometry Deterministic (PGD) schedule, achieving superior detail preservation, strong synthetic-to-real generalization, and efficiency with limited training data.

Original authors: Xinhao Cai, Gensheng Pei, Zeren Sun, Yazhou Yao, Fumin Shen, Wenguan Wang

Published 2026-03-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xinhao Cai, Gensheng Pei, Zeren Sun, Yazhou Yao, Fumin Shen, Wenguan Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand the 3D world just by looking at a single 2D photograph. This is called Monocular Depth Estimation. It's like trying to guess how far away a mountain is just by looking at a flat painting of it.

For a long time, the best way to do this was to feed the robot millions of photos and their correct 3D answers (like a student memorizing a textbook). But this is expensive, and the robot often misses small details like the texture of a brick wall or the sharp edge of a leaf.

Recently, scientists tried using Diffusion Models (the same technology behind AI art generators like Midjourney). These models are great at imagining details, but they struggle to be accurate in the real world and often get confused when moving from "fake" computer graphics to "real" photos.

Enter Iris, a new method that acts like the perfect student-teacher team. Here is how it works, using simple analogies:

The Problem: The "Blurry Teacher" vs. The "Perfect Student"

The researchers realized there was a mismatch:

Real-world data (The Teacher): Imagine a teacher who has seen millions of real photos. They are great at knowing the big picture (where the horizon is, how big a car is), but their vision is a bit blurry when it comes to tiny details like individual leaves or sharp edges.
Synthetic data (The Student's Practice): Imagine a video game world where the rules are perfect. The computer knows the exact distance of every pixel, but the world looks a bit "plastic" and lacks the messy, natural feel of the real world.

If you try to teach the robot using both at the same time, it gets confused. The "blurry teacher" tells it to be smooth, while the "perfect student" tells it to be sharp. The robot ends up with a confused, mediocre result.

The Solution: The "Two-Stage Dance" (PGD)

Iris solves this by splitting the learning process into two distinct steps, like a dance with two different partners.

Stage 1: The "Big Picture" Sketch (High Timestep)

The Analogy: Imagine an artist starting a painting with a charcoal sketch. They aren't worried about the fine details yet; they just want to get the shape of the nose, the curve of the smile, and the position of the eyes right.
What Iris does: The model looks at real-world photos and asks the "Blurry Teacher" for help. But, it uses a special filter called Spectral-Gated Distillation (SGD).
The Magic Filter: This filter acts like a sieve. It lets the "Big Picture" information (low-frequency data) pass through from the teacher, but it blocks the "blurry noise" and bad details. The robot learns the layout of the room and the size of objects without getting confused by the teacher's fuzzy edges.

Stage 2: The "Fine Detail" Polish (Low Timestep)

The Analogy: Now that the sketch is perfect, the artist switches to fine brushes and high-quality paint to add the eyelashes, the texture of the skin, and the sharp highlights.
What Iris does: The model switches to "perfect" computer-generated data. It uses a second filter called Spectral-Gated Consistency (SGC).
The Magic Filter: This time, it focuses only on the sharp edges and tiny textures. It looks back at the sketch from Stage 1 and says, "Okay, the shape is right, now let's make the edges crisp and the details pop."

Why is this special?

Most other methods try to do everything at once, which leads to a messy result. Iris separates the tasks:

First, get the structure right using real-world intuition (but ignoring the bad details).
Second, add the sharp details using perfect synthetic data (but keeping the real-world structure).

The Result

Because Iris separates these tasks, it gets the best of both worlds:

It understands the real world: It knows how big things are and where they are located, just like a human.
It sees the details: It can see the sharp edge of a window or the texture of a tree bark, which other AI models often miss.
It's efficient: It doesn't need millions of perfectly labeled photos. It can learn with a much smaller dataset, saving time and money.

In summary: Iris is like a master architect who first consults a rough, experienced builder to get the blueprints right, and then hires a precision engineer to add the perfect finishing touches. The result is a depth map that is both structurally sound and incredibly detailed.

1. Problem Statement

Monocular Depth Estimation (MDE) faces a fundamental trade-off between generalization and detail fidelity, exacerbated by data limitations:

Feed-forward methods (e.g., Depth Anything V2): Rely on massive real-world datasets but suffer from imperfect supervision (noisy depth maps, poor detail preservation). They often produce smooth outputs that lack fine-grained boundaries and textures.
Diffusion-based methods: Leverage rich generative priors from text-to-image models (like Stable Diffusion) to preserve fine details. However, they typically struggle with synthetic-to-real domain transfer, often failing to generalize well to real-world scenes without massive real-data supervision.
The Core Conflict: Training a model to learn both low-frequency global layout (from noisy real-world pseudo-labels) and high-frequency fine details (from precise synthetic ground truth) simultaneously causes gradient interference, leading to degraded performance or artifacts.

2. Methodology: The Iris Framework

The authors propose Iris, a deterministic diffusion-based framework that integrates real-world priors through a novel Priors-to-Geometry Deterministic (PGD) schedule. Instead of iterative sampling, Iris operates as a feed-forward network using a two-stage process with shared weights.

A. Priors-to-Geometry Deterministic (PGD) Schedule

The framework decouples the learning process into two distinct stages based on diffusion timesteps ( $t$ ):

Prior Alignment Stage (High Timestep, Low SNR):
- Operates at a high diffusion timestep ( $t_{high}$ ).
- Focuses on low-frequency components (global layout, metric scale).
- Supervised by a frozen teacher model trained on real images (providing real-world priors).
- Leaves high-frequency details unconstrained to avoid gradient interference from noisy pseudo-labels.
Geometry Refinement Stage (Low Timestep, High SNR):
- Operates at a low diffusion timestep ( $t_{low}$ ).
- Focuses on high-frequency components (fine textures, sharp boundaries).
- Supervised by precise synthetic ground truth to refine geometry and metric accuracy.
- Inherits high-frequency cues from Stage 1.

B. Key Technical Innovations

To address the frequency-reliability mismatch between real-world pseudo-labels (reliable low-freq, unreliable high-freq) and synthetic data (reliable high-freq), Iris introduces two spectral gating mechanisms:

Spectral-Gated Distillation (SGD):
- Goal: Transfer real-world priors without imprinting teacher artifacts.
- Mechanism: A lightweight, learnable low-pass gate in the Fourier domain. It softly attenuates the high-frequency content of the teacher's pseudo-labels while passing the reliable low-frequency structure.
- Result: The student learns global layout and scale from real data but is free to learn high-frequency details from synthetic data in the next stage.
Spectral-Gated Consistency (SGC):
- Goal: Transfer sharp boundaries and fine details from Stage 1 to Stage 2.
- Observation: Surprisingly, Stage 1 (trained with low-pass alignment) produces crisper boundaries than expected because low-pass supervision forces steeper transitions at semantic edges.
- Mechanism: A lightweight high-pass gate aligns the Stage 2 output with the Stage 1 output in the high-frequency band. An auxiliary constraint prevents Stage 1 from over-activating high-frequency noise.
- Result: Ensures the final output retains the sharp details generated in the prior stage while refining geometry with synthetic data.

C. Training Objective

The total loss combines depth estimation, distillation, consistency, and an auxiliary image reconstruction term (to prevent catastrophic forgetting of the diffusion backbone's generative capabilities):
$\mathcal{L} = \mathcal{L}_{\text{depth}} + \alpha \mathcal{L}_{\text{sgd}} + \beta \mathcal{L}_{\text{sgc}} + \gamma \mathcal{L}_{\text{recon}}$

3. Key Contributions

Iris Framework: A deterministic, feed-forward diffusion model that achieves state-of-the-art MDE performance with significantly less training data than massive feed-forward models (e.g., Depth Anything V2).
PGD Schedule: A novel two-stage training strategy that decouples prior alignment (real-world layout) from geometry refinement (synthetic details), effectively mitigating gradient interference.
Spectral Gating Mechanisms:
- SGD: Enables robust transfer of low-frequency real-world priors while filtering out high-frequency noise.
- SGC: Leverages the "crispness" of the prior stage to enforce high-frequency fidelity in the final output.
Efficiency: Converts the stochastic diffusion paradigm into a single-step deterministic inference, drastically reducing inference time compared to multi-step diffusion methods.

4. Experimental Results

The paper evaluates Iris on zero-shot affine-invariant depth estimation across five diverse real-world benchmarks (KITTI, NYUv2, ETH3D, ScanNet, DIODE).

Performance: Iris achieves the best overall performance among all methods tested.
- It outperforms previous diffusion-based SOTA methods (e.g., Marigold, Lotus, GenPercept) in cross-domain generalization.
- It rivals or exceeds massive feed-forward models (Depth Anything V2, trained on 62.6M images) despite being trained on only 59K synthetic images + 100K real pseudo-labeled images.
Detail Fidelity: Qualitative results show Iris preserves fine textures and sharp boundaries significantly better than DAv2, which tends to produce smoother, less detailed outputs.
Efficiency:
- Inference Time: Iris runs in 1.3 seconds (at 1536² resolution), which is faster than DAv2 (2.2s) and orders of magnitude faster than multi-step diffusion methods like Marigold (377.7s).
- Training Cost: Achieves high performance with a fraction of the data and compute required by large-scale discriminative models.

5. Significance

Bridging the Gap: Iris successfully bridges the gap between the generalization of data-hungry feed-forward models and the detail fidelity of diffusion models.
Data Efficiency: It demonstrates that high-quality MDE does not require massive real-world datasets if real-world priors are integrated intelligently via spectral gating and staged training.
Practical Deployment: By eliminating iterative sampling and reducing training data requirements, Iris makes high-fidelity depth estimation more feasible for real-world applications (e.g., autonomous driving, 3D reconstruction) where compute and labeled data are limited.
Theoretical Insight: The paper provides a new perspective on diffusion-based perception, showing that separating frequency bands (low vs. high) and training stages can resolve the conflict between noisy real supervision and perfect synthetic supervision.