🤖 AI

From Editor to Dense Geometry Estimator

This paper introduces FE2E, a framework that adapts a Diffusion Transformer-based image editing model for dense geometry estimation by leveraging its superior structural priors and reformulating its training objectives, thereby achieving state-of-the-art zero-shot performance in depth and normal estimation without requiring massive training datasets.

Original authors: JiYuan Wang, Chunyu Lin, Lei Sun, Rongying Liu, Lang Nie, Mingxing Li, Kang Liao, Xiangxiang Chu

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: JiYuan Wang, Chunyu Lin, Lei Sun, Rongying Liu, Lang Nie, Mingxing Li, Kang Liao, Xiangxiang Chu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart artist who is famous for two things:

Painting new worlds from scratch based on a description (like "a cat on a moon").
Editing existing photos (like changing a sunny day to a rainy one, or swapping a red car for a blue one).

For a long time, computer scientists thought the "Painter" (the generative model) was the best tool to guess the 3D shape of a room just by looking at a flat photo. But this paper, FE2E, argues that we've been using the wrong tool for the job. It says: "Stop asking the Painter to guess the shape; ask the Editor instead."

Here is the breakdown of how they did it, using some simple analogies.

1. The Big Idea: The Editor vs. The Painter

The Painter (Generative Model): This model is used to create new images. It's great at imagination, but when you ask it to guess the depth of a specific object in a photo, it has to "unlearn" its habit of making things up. It's like asking a creative writer to solve a math problem; they have to stop writing stories and start crunching numbers. It's a messy, unstable process.
The Editor (Editing Model): This model is used to change existing images. It already understands that if you change the color of a wall, the wall's shape stays the same. It has a natural "feel" for the structure of an image.
The Discovery: The researchers found that the Editor is naturally better at understanding geometry. It doesn't need to be taught from scratch; it just needs to be told, "Hey, instead of changing the color, tell me how deep the objects are." It's like taking a master carpenter (the Editor) and asking them to measure a room, rather than asking a painter (the Painter) to measure it while trying to paint a masterpiece.

2. The Three Tricks (How they fixed the Editor)

Even though the Editor was the right choice, it wasn't perfect for this specific job. The team had to make three adjustments to turn the Editor into a "Geometry Estimator."

Trick A: The "Straight Line" Rule (Consistent Velocity)

The Problem: Usually, these models take a "winding path" to get from a blurry guess to a clear answer. Imagine trying to drive from New York to London by taking a zig-zag path through the ocean. It's slow and prone to getting lost.
The Fix: They forced the model to take a straight line. They told the model, "Don't wander. Just go directly from 'blurry' to 'clear' in one straight shot." This makes the answer much more stable and accurate, like a high-speed train on a straight track.

Trick B: The "Microscope" Upgrade (Logarithmic Quantization)

The Problem: The Editor was trained to handle colors (Red, Green, Blue). Colors are easy to measure; a little bit of error doesn't matter much. But Depth (how far away something is) is tricky. If you are measuring a mountain 80 miles away, a tiny error in your math makes you think the mountain is in the next town over. The Editor's "ruler" (its internal math precision) wasn't fine enough for deep distances.
The Fix: They gave the Editor a specialized microscope. Instead of measuring distance in a straight line (which is bad for huge distances), they changed the math to measure it logarithmically. Think of it like a volume knob: turning it from 1 to 2 is a big jump, but turning it from 100 to 101 is a tiny, precise adjustment. This allowed the model to see both close-up details and far-away mountains with perfect clarity.

Trick C: The "Two-for-One" Deal (Cost-Free Joint Estimation)

The Problem: When the Editor does its job, it usually produces two outputs: one it uses, and one it throws away (like a printer that prints a test page and then scrapes it off). That's a waste of energy.
The Fix: They realized the "trash" output actually contained useful information about the surface angles (normals) of the objects. So, they told the model: "Don't throw that away! Use it to guess the angles at the same time you guess the depth."
The Result: They got two answers (Depth + Angles) for the price of one calculation. It's like ordering a burger and getting a free side of fries because the kitchen was already cooking them.

3. The Result: A Super-Efficient Wizard

The result is FE2E.

Data Efficiency: Most modern AI models need to "eat" millions of photos to learn how to guess depth. FE2E learned the same skills by looking at a tiny fraction of that data (about 0.2%). It's like a student who reads one textbook and gets an A+, while others read 500 books and still get a B.
Performance: It beat the current "champions" (like DepthAnything) on almost every test, especially on tricky datasets like ETH3D.
Speed: Because it uses the "straight line" trick and the "two-for-one" deal, it's faster and more efficient.

Summary

The paper says: "Stop trying to force a creative artist to do math. Use the editor who already understands the structure of the image, give them a better ruler, and let them do two jobs at once."

By doing this, they created a tool that sees the 3D world in 2D photos with incredible accuracy, using very little training data and very little computing power.

1. Problem Statement

Dense geometry prediction (specifically monocular depth and surface normal estimation) is an ill-posed problem requiring strong priors to infer 3D structure from a single 2D image.

Current Limitations: Recent state-of-the-art (SoTA) approaches have leveraged pre-trained Text-to-Image (T2I) generative models (e.g., Stable Diffusion) as foundations. However, T2I models are inherently designed for generating new content from noise, which leads to unstable feature reshaping when fine-tuned for deterministic image-to-image tasks like geometry estimation.
The Gap: There is a lack of systematic analysis comparing Image Editing models (Image-to-Image, I2I) against Generative models for dense prediction. Furthermore, adapting existing editing models (often based on Diffusion Transformers, or DiTs) to geometry tasks introduces specific challenges regarding training objectives, numerical precision, and computational efficiency.

2. Methodology: FE2E Framework

The authors propose FE2E (From Editor to Estimator), a framework that adapts a pre-trained DiT-based image editing model (specifically Step1X-Edit) for dense geometry prediction. The methodology consists of three core technical innovations:

A. Theoretical Foundation: Editor vs. Generator Analysis

The authors conducted a systematic analysis comparing fine-tuning an editing model (Step1X-Edit) versus a generative model (FLUX).

Findings: Editing models possess inherent structural priors aligned with input image geometry. Fine-tuning an editor acts as a "refinement" and "focusing" of existing features, leading to stable convergence. In contrast, generative models require "characteristics reshaping" from chaotic states, resulting in oscillating loss and a performance bottleneck.
Conclusion: Editing models provide a superior foundation for dense estimation tasks.

B. Key Technical Contributions

To bridge the gap between image editing and deterministic geometry estimation, FE2E introduces three specific adaptations:

Consistent Velocity Flow Matching with Deterministic Departure:
- Problem: Standard Flow Matching learns an instantaneous velocity field over stochastic paths, introducing approximation errors during numerical integration (ODE solving) for high-precision tasks.
- Solution: The authors reformulate the training objective.
  - Consistent Velocity: The model is trained to predict a velocity vector that is independent of time $t$ and the starting noise, effectively learning a straight-line trajectory ( $v = z_1 - z_0$ ).
  - Fixed Origin: Instead of sampling from a Gaussian distribution ( $z_0 \sim \mathcal{N}(0,1)$ ), the starting point is fixed to zero ( $z_0 = 0$ ).
- Result: This eliminates integration errors, simplifies inference to a single forward pass ( $\hat{y} = f_\theta(x)$ ), and significantly improves stability.
Logarithmic Quantization for Precision:
- Problem: Modern editing models are trained in BFloat16 (BF16) precision. While sufficient for RGB images, BF16 lacks the dynamic range for depth maps (e.g., 0.1m to 80m). Uniform quantization causes massive errors at close range, while inverse quantization (disparity) fails at long range.
- Solution: The authors propose Logarithmic Quantization. They transform the ground truth depth $D$ to $D_{log} = \ln(D + \epsilon)$ , normalize it, and quantize.
- Result: This maintains a nearly constant relative error (AbsRel) across both near and far distances, resolving the precision conflict without requiring FP32 training.
Cost-Free Joint Estimation:
- Problem: DiT-based editors typically concatenate input latents and noise, producing an output where half the spatial dimensions correspond to the original noise (which is discarded).
- Solution: FE2E repurposes this "discarded" region. The model is trained to predict both depth and surface normals simultaneously in a single forward pass.
- Result: This achieves joint estimation with zero additional computational cost (no extra parameters or inference time) while leveraging the DiT's global attention to improve performance on both tasks.

3. Key Results

FE2E was trained on a mixture of synthetic datasets (Hypersim and Virtual KITTI) totaling only ~71k images (0.071M), which is orders of magnitude smaller than data-driven models like DepthAnything (trained on 62.6M images).

Zero-Shot Depth Estimation:
- Outperforms the DepthAnything V2 series (trained on 100x more data) across multiple benchmarks.
- Achieves a 35% improvement in AbsRel on the ETH3D dataset compared to the second-best method.
- Achieves the best average ranking (1.4) across all tested metrics and datasets.
Zero-Shot Normal Estimation:
- Sets a new SoTA on four benchmarks (NYUv2, ScanNet, iBims-1, Sintel), outperforming specialized normal estimation models like StableNormal and Lotus.
Efficiency:
- The "Cost-Free Joint Estimation" and "Fixed Origin" strategies allow for faster inference compared to iterative diffusion sampling methods.

4. Significance and Impact

Paradigm Shift: The paper challenges the prevailing trend of using T2I generative models for dense prediction, demonstrating that Image Editing models are a more natural and effective foundation due to their inherent structural priors.
Data Efficiency: FE2E proves that high-performance geometry estimation does not require massive datasets; leveraging the right foundation model and training protocol (editing priors) yields superior results with minimal data.
Technical Innovation: The proposed Consistent Velocity objective and Logarithmic Quantization offer generalizable solutions for adapting low-precision DiT editors to high-precision deterministic tasks, potentially applicable to other vision domains beyond depth and normals.
Practicality: By repurposing discarded computation for joint estimation, the method offers a highly efficient pipeline for 3D reconstruction tasks in AR/VR and robotics.

In summary, FE2E successfully transitions a powerful image editing foundation model into a dense geometry estimator, achieving state-of-the-art performance with significantly less training data and computational overhead than its generative counterparts.