💻 computer science

Towards Robust Driving Perception: A Flexible Scale-Driven Family for Self-Supervised Monocular Depth Estimation

The paper introduces FlexDepth, a flexible, scale-driven family of self-supervised monocular depth estimation models that utilizes a static-dynamic decoupled training strategy and a Scale-Driven Decoder to achieve state-of-the-art performance and real-time efficiency on resource-constrained automotive edge devices without requiring ground truth or auxiliary information.

Original authors: Zhaowen Zhu, Li Zhang, Yujie Chen, Tian Zhang, Yingjie Wang, Mingxia Zhan

Published 2026-07-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhaowen Zhu, Li Zhang, Yujie Chen, Tian Zhang, Yingjie Wang, Mingxia Zhan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess how far away everything is in a room, but you only have one eye (a single camera) and no ruler. This is the challenge of Monocular Depth Estimation. For self-driving cars, this is like trying to navigate a busy highway using only a single video feed, without knowing exactly how far the cars or pedestrians are.

For a long time, computers struggled with this, especially when things were moving. If a car drove past, the computer got confused because it assumed the whole world was still. Also, making these computers smart enough to see clearly usually meant making them huge and slow, like trying to run a supercomputer on a tiny smartphone battery.

The paper introduces a new family of AI models called FlexDepth. Think of FlexDepth as a set of "smart, adaptable glasses" for self-driving cars that come in five different sizes, from a tiny pair of reading glasses (Flex-Nano) to a heavy-duty pair of binoculars (Flex-X-Large). No matter the size, they are designed to be fast, accurate, and tough enough to handle the chaos of real-world driving.

Here is how they made it work, explained simply:

1. The "Scale-Driven" Decoder (The Smart Upscaler)

Imagine you are trying to draw a map of a city. If you just stretch a small, blurry sketch to make it big, the edges of the buildings will look fuzzy and the roads will look wobbly. Traditional AI does this; it tries to make the image bigger using simple math, which blurs the details.

FlexDepth uses a special tool called a Scale-Driven Decoder (SDD). Instead of just stretching the image, this tool acts like a dynamic construction crew.

For small models (like Flex-Nano): It uses a lightweight, efficient crew that focuses on speed, making sure the car doesn't lag.
For big models (like Flex-X-Large): It brings in a more detailed crew that pays extra attention to the edges of objects and textures.
The Magic: It doesn't just guess where the edges are; it actively "re-samples" the image, looking at the content to decide exactly where to place the pixels. This keeps the boundaries of cars and trees sharp, even when the image is being enlarged.

2. The "Two-Stage" Training (The Static vs. Dynamic Drill)

One of the biggest headaches for self-driving AI is moving objects. If a car drives by, the AI might think the whole world is moving, or it might get confused about how far away that car is.

The authors solved this with a two-stage training strategy, which is like a two-part drill for the AI student:

Stage 1 (The Basics): The AI learns to understand the world and how the camera moves, just like a student learning the rules of the road.
Stage 2 (The "Spot the Difference" Test): The AI is shown the same scene at the beginning of its training and at the end.
- Static things (like buildings and trees) look the same in both versions. The AI learns to trust these.
- Dynamic things (like other cars or people) look different because they moved. The AI learns to say, "Wait, this part changed! I shouldn't trust my depth guess for this moving object yet."

By separating the "still" world from the "moving" world, the AI learns to ignore the confusing motion and focus on getting the depth right for the stable background, which actually helps it understand the moving objects better too.

3. The Results: Fast, Light, and Sharp

The paper claims that FlexDepth is a "family" of models that can fit into any car, from a cheap sensor to a high-end autonomous vehicle.

The Tiny Model (Flex-Nano): It is incredibly light, requiring only 0.7 GFLOPs (a measure of computing power). It can run at 37.6 frames per second on mobile chips. This is like having a super-fast runner who can keep up with traffic without getting tired.
The Big Model (Flex-X-Large): It is the most accurate, beating almost all other models on standard driving tests (KITTI and Cityscapes) while still being faster than many "heavy" competitors.
No Cheating: Unlike some other methods that need extra sensors (like flow sensors) or pre-made labels to learn, FlexDepth learns entirely on its own from video footage, just by watching how the world changes.

Summary

In short, the paper presents FlexDepth, a flexible system that allows self-driving cars to "see" depth clearly and quickly. It uses a smart, adaptable decoder to keep edges sharp and a two-step training method to stop moving cars from confusing the system. Whether the car needs a tiny, fast processor or a powerful one, FlexDepth scales up or down to provide a clear, safe view of the road.

Technical Summary: FlexDepth – A Flexible Scale-Driven Family for Self-Supervised Monocular Depth Estimation

1. Problem Statement

Self-Supervised Monocular Depth Estimation (MDE) is critical for autonomous driving and robotics due to its independence from dense ground-truth depth labels. However, existing approaches face significant limitations in complex driving environments:

Scale and Robustness Trade-off: Most lightweight models rely on single-scale designs or simplified decoders, leading to performance degradation in dynamic scenes (e.g., moving vehicles, pedestrians) and poor structural fidelity at object boundaries.
Complexity of Dynamic Handling: Methods designed to handle dynamic objects often introduce excessive complexity, auxiliary tasks (e.g., optical flow, semantic segmentation), or heavy backbones (e.g., Vision Transformers), making them unsuitable for resource-constrained edge devices.
Limitations of Foundation Models: While supervised foundation models (e.g., Depth Anything V2) offer strong zero-shot generalization, their reliance on massive datasets and heavy computational requirements hinders deployment in real-time, edge-based automotive scenarios.

The core challenge addressed is how to narrow the robustness gap between lightweight self-supervised models and large supervised models while maintaining data efficiency and deployability on edge hardware.

2. Methodology

The authors propose FlexDepth, a family of scale-driven self-supervised MDE models tailored for dynamic driving scenarios. The architecture consists of three primary innovations:

A. Scale-Driven Decoder (SDD)

Unlike traditional decoders that use fixed pipelines (pre-convolution $\to$ upsampling $\to$ concatenation), FlexDepth introduces a decoder that adaptively selects components based on the model's scale (Nano, Small, Medium, Large, X-Large).

Post-Convolution Design: The authors remove the pre-convolution stage to prevent the irreversible loss of global semantic information. Instead, they focus on post-convolution ( $C$ $C$ ), upsampling ( $U$ $U$ ), and prediction heads ( $P$ $P$ ).
- Small Scales (Nano/Small): Utilize High-Efficiency Bottlenecks (HEB) to ensure gradient flow stability and a traditional prediction head order.
- Large Scales (Medium/X-Large): Utilize High-Performance Bottlenecks (HPB) with Cross Stage Partial (CSP) structures for finer local feature capture and an inverted prediction head (upsampling before prediction) to leverage high-channel feature consistency.
Dynamic Upsampling ( $U$ ): Replaces standard bilinear interpolation with a two-stage process: offset generation and dynamic resampling. A convolutional layer predicts offsets ( $\Delta P$ ) based on semantic content, allowing the model to adaptively sample optimal locations from feature maps, resulting in sharper object boundaries.

B. Two-Stage Static-Dynamic Decoupled Training

To address the instability caused by moving objects without relying on external priors, FlexDepth employs a two-stage training strategy:

Stage 1 (Joint Optimization): Simultaneously trains the Depth Prediction Network and the PoseNet (ResNet-18 based) using standard photometric reprojection loss, minimum reprojection loss, and edge-aware smoothness loss.
Stage 2 (Dynamic Refinement): Leverages the observation that dynamic regions exhibit high predictive instability (variance between early and late training epochs) compared to static backgrounds.
- A Static-Dynamic Decoupled Mask ( $M$ ) is generated by comparing disparity maps from the initial and final epochs of Stage 1.
- An adaptive thresholding function $g(\phi(f_{si}, f_{sl}))$ determines the mask, distinguishing dynamic objects from static backgrounds based on feature evolution rather than fixed thresholds.
- The depth network is retrained using this mask with a masked normal distribution loss and geometric depth smoothing, effectively decoupling dynamic noise from static scene learning.

C. Self-Supervised Training Pipeline

The model is trained entirely on unlabeled video sequences using photometric consistency.

Loss Functions: Combines photometric error ( $L_p$ ), minimum reprojection loss ( $L_r$ ), edge-aware smoothness ( $L_{smooth}$ ), and in Stage 2, a masked normal distribution loss ( $L_m$ ) and geometric depth smoothing ( $L_{GDS}$ ).
No Auxiliary Data: The method operates without semantic labels, optical flow, or pseudo-depth maps.

3. Key Contributions

FlexDepth Family: A scalable family of models (Nano to X-Large) that systematically studies the accuracy-efficiency trade-off, achieving state-of-the-art (SOTA) performance across arbitrary scales with minimal computational overhead.
Scale-Driven Decoder (SDD): A novel decoder architecture that dynamically selects bottleneck types (HEB vs. HPB) and upsampling strategies based on model size, balancing real-time inference with high-precision depth mapping.
Static-Dynamic Decoupled Training: A two-stage strategy that isolates dynamic objects from static backgrounds using an adaptive mask derived from training instability, enhancing robustness in dynamic environments without additional inference costs or auxiliary inputs.

4. Experimental Results

Experiments were conducted on KITTI, Cityscapes, and Make3D (for zero-shot generalization).

Performance vs. Efficiency:
- Flex-Nano: Requires only 0.7 GFLOPs and achieves 37.6 FPS on mobile platforms (Snapdragon 8 Elite), outperforming Lite-Mono's Tiny and Small variants in accuracy while being significantly lighter.
- Flex-X-Large: Sets new SOTA on KITTI and Cityscapes with 24.6 GFLOPs, outperforming heavier models like MonoVit (59.7 GFLOPs) and DSI-MonoViT.
Dynamic Scene Robustness: On Cityscapes, Flex-X-Large achieves superior accuracy in dynamic regions (Abs Rel 0.091) compared to methods relying on auxiliary cues (e.g., ProDepth, DSI-MonoViT), despite using no extra information.
Zero-Shot Generalization: The model trained on KITTI demonstrates strong transferability to the Make3D dataset, outperforming other self-supervised methods and approaching the performance of larger foundation models in specific metrics.
Ablation Studies: Confirm that the SDD components (HPB, dynamic upsampling, inverted head) and the two-stage training strategy are critical for performance gains. Removing the second stage leads to severe performance drops in dynamic scenes.

5. Significance and Claims

The paper claims that FlexDepth successfully bridges the gap between lightweight efficiency and robust perception in dynamic driving environments.

Deployment Viability: By achieving SOTA performance with minimal computational cost (e.g., 0.7 GFLOPs for Nano), FlexDepth enables reliable real-time perception on resource-constrained automotive edge devices.
Intrinsic Robustness: The method demonstrates that robust handling of dynamic scenes can be achieved through architectural design and training strategy (static-dynamic decoupling) rather than relying on complex auxiliary pipelines or heavy backbones.
Complement to Foundation Models: While acknowledging the superior zero-shot capabilities of large foundation models (e.g., Depth Anything V2), the authors argue that in-domain self-supervised training (like FlexDepth) offers practical advantages for edge deployment due to its lower resource footprint and comparable or superior accuracy in specific driving contexts.

Limitations: The authors note that FlexDepth is specifically designed for driving perception and its applicability to indoor environments with irregular textures and close-range occlusions is limited.