💻 computer science

JEPADepth: Masked Predictive Representation Learning for Self-Supervised Monocular Depth Estimation

JEPADepth introduces a self-supervised monocular depth estimation framework that enhances standard photometric pipelines by incorporating a masked predictive representation learning objective inspired by I-JEPA, achieving state-of-the-art performance on benchmarks like KITTI and superior zero-shot transfer capabilities without adding inference-time costs.

Original authors: Ionuţ Grigore, Călin-Adrian Popa

Published 2026-07-30

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Ionuţ Grigore, Călin-Adrian Popa

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to see the world in 3D, just like a human does, but you only have a single camera and no ruler to measure distances. This is the challenge of "monocular depth estimation." For a long time, the best way to teach a robot this trick was to show it two pictures taken a split-second apart and ask it to figure out how much the world shifted between them. It's like holding your thumb up and closing one eye, then the other; your brain calculates the distance based on how much your thumb "jumped." In the world of AI, this is called "photometric reconstruction." The computer tries to warp one image to look like the other, and if the math works out, it assumes it has guessed the depth correctly.

However, this method is a bit fragile. It relies heavily on the assumption that the lighting stays the same and that objects don't move or change shape. If a car drives by, or the sun suddenly glares off a wet road, the computer gets confused because its "thumb-jump" math breaks down. It's like trying to solve a puzzle where the pieces keep changing color. To make these robots smarter and more robust, scientists are looking for ways to teach them not just to match pixels, but to understand the structure of the scene—like knowing that a tree is a solid object with a trunk and branches, regardless of whether the sun is shining or it's cloudy. This is where the new paper, JEPADepth, steps in with a clever new training trick.

The Paper's Big Idea: Teaching by Filling in the Blanks

The authors, Ionut Grigore and Călin-Adrian Popa from the Politehnica University of Timișoara, propose a method called JEPADepth. Think of their approach as giving the robot a "fill-in-the-blanks" test, but instead of words, the blanks are parts of an image, and instead of guessing the missing letters, the robot has to guess the meaning of the missing parts.

In the old way (photometric reconstruction), the robot tries to copy the exact colors and brightness of a neighboring image. It's like a photocopier trying to reproduce a picture perfectly. If the paper is crumpled or the ink is smudged, the copy fails.

In the new JEPADepth way, the robot uses a "masked predictive" strategy inspired by a technique called I-JEPA. Here's how it works:

The Mask: The computer takes an image and covers up several random chunks (like putting sticky notes over parts of a photo).
The Guess: The robot looks at the visible parts (the context) and tries to predict what the hidden parts should look like, not in terms of exact pixels (colors), but in terms of "concepts" or "features" (like "this is a tree trunk" or "this is a road").
The Teacher: A special "teacher" network (which is just a copy of the main brain that updates slowly) provides the correct answer for what those hidden chunks represent. The robot compares its guess to the teacher's answer and learns from the difference.

The magic is that this happens in "representation space." Imagine the robot isn't trying to guess the exact shade of green of a leaf; it's guessing the idea of a leaf. This makes it much harder to fool the robot with changes in lighting or texture. If the sun hits the leaf differently, the "idea" of the leaf remains the same, so the robot still gets the depth right.

What They Found: Smarter, Not Just Bigger

The researchers tested this new method on the KITTI dataset, which is a standard collection of driving videos used to train self-driving cars. They compared their new JEPADepth model against the best existing methods.

Here is the good news: JEPADepth works.

Better Accuracy: When tested on the KITTI dataset, the new method beat the previous "gold standard" models. For example, on a standard test, it achieved an error rate (called AbsRel) of 0.101, which was better than the previous best transformer-based model (MonoViT) and significantly better than older CNN-based models.
The "Zero-Shot" Superpower: The most exciting finding is how well the robot generalized to new places. The model was trained only on KITTI (which looks like highways in California), but then they tested it on Cityscapes (a dataset of busy European cities) and Make3D (a dataset of outdoor scenes) without giving it any extra training or "fine-tuning."
- On Cityscapes, JEPADepth tied for the best accuracy and was the top performer in 5 out of 7 metrics.
- On Make3D, it achieved the best accuracy (AbsRel of 0.275) and the best log-error (RMSElog of 0.143), beating even the strongest competitors.

This suggests that by learning to predict the "structure" of the scene rather than just copying pixels, the robot learned a more universal understanding of 3D space that works even in environments it has never seen before.

The Trade-Off: A Little Extra Homework, No Extra Homework Later

You might wonder, "Does this make the robot slower or require a bigger computer?" The authors were careful to design this so that the answer is no.

The "fill-in-the-blanks" training (the JEPA part) does require a bit more work while the robot is learning. The computer has to run an extra step to make predictions and compare them to the teacher. This added about 16% to the total training time (taking about 8.6 hours instead of 7.4 hours on a single high-end GPU).

However, once the training is done, the "teacher" and the "prediction" parts of the brain are thrown away. The final robot that goes out into the world is exactly the same size and speed as the standard version. It's like a student who does extra practice problems at home to get smarter, but on test day, they only need to use the knowledge they gained, not the extra practice sheets. The paper confirms that this extra training cost results in a model that is just as efficient to deploy but much smarter.

What the Paper Says It's Not

The authors were also clear about what their method is not.

It is not a magic bullet that solves all depth estimation problems. They note that their current "decoder" (the part that turns the robot's brain into a final depth map) is still relatively simple. They tried swapping it for a more complex design (called DPT), but it didn't help in their self-supervised setup, suggesting that the bottleneck is in how the robot uses its learned features, not just the features themselves.
It is not a method that works without a strong starting point. The experiments showed that if they started with a random, untrained brain instead of using a pre-trained "DINOv3" brain, the method failed miserably. The "fill-in-the-blanks" trick only works if the robot already has a good vocabulary of visual concepts to start with.
It does not rely on pixel-perfect reconstruction. The paper explicitly argues against the idea that predicting exact pixel colors is the best way to learn depth. Their tests showed that predicting "features" (the abstract meaning of the image) works better than trying to guess the exact color of every pixel.

The Bottom Line

JEPADepth suggests that the future of teaching robots to see depth isn't just about showing them more pictures or making the pictures brighter. It's about teaching them to understand the story of the scene. By adding a "guess the hidden part" game to the standard training routine, the authors created a model that is more accurate, more robust to changes in lighting and scenery, and better at handling new environments it has never seen before—all without making the final robot any heavier or slower. It's a promising step toward making self-driving cars and robots that can truly "see" the world, not just copy it.

Technical Summary of JEPADepth: Masked Predictive Representation Learning for Self-Supervised Monocular Depth Estimation

Problem Statement
Self-supervised monocular depth estimation typically relies on photometric reconstruction losses, which enforce geometric consistency by warping source frames to match a target frame. While effective, this paradigm is inherently sensitive to assumptions such as brightness constancy and sufficient texture. It often struggles with reflective surfaces, illumination changes, and non-rigid motion. Furthermore, because the loss is computed in pixel space, optimization may encourage the model to memorize appearance statistics rather than learning higher-level scene structure, limiting robustness and cross-dataset transfer. While recent self-supervised representation learning (e.g., DINOv3) produces powerful encoders that capture semantic and structural regularities, standard depth pipelines do not fully leverage these features to enforce spatial reasoning beyond pixel reconstruction.

Methodology
The authors propose JEPADepth, a framework that integrates an Image Joint-Embedding Predictive Architecture (I-JEPA) objective as a complementary training signal to standard photometric supervision. The method operates as follows:

Backbone and Architecture: The system utilizes a pretrained DINOv3 Vision Transformer (ViT) encoder to extract patch-level tokens from input images. These tokens, which capture higher-level structure like object boundaries and layout, serve as the feature space for prediction. A lightweight convolutional Feature Pyramid Network (FPN) decoder is attached to the encoder to upsample patch tokens and predict disparity maps at multiple scales.
I-JEPA Objective: Instead of reconstructing pixels, the model is trained to predict latent representations of masked image regions.
- Masking: For each image, a context mask selects visible patches, while target masks select held-out regions.
- Prediction: A predictor network (a transformer) takes the visible context embeddings and infers the embeddings for the masked target regions.
- Target Generation: An Exponential Moving Average (EMA) updated target encoder processes the image to provide stop-gradient target embeddings for the masked regions.
- Loss: The model minimizes an L2 regression loss between the predicted target embeddings and the stop-gradient target embeddings in the representation space.
Training and Inference: The depth network is trained end-to-end using a combined loss function comprising the standard photometric view-synthesis loss, an edge-aware smoothness regularizer, and the JEPA representation prediction loss. Crucially, the predictor and the target encoder are discarded after training. Consequently, the inference-time architecture remains identical to the baseline, incurring no additional computational cost or deployment overhead.

Key Contributions

JEPA-Regularized Depth Estimation: The paper introduces the first method to integrate an I-JEPA-inspired masked representation prediction objective into a standard photometric self-supervised depth pipeline. This shifts the auxiliary learning signal from pixel-space reconstruction to representation-space prediction.
Improved Performance without Inference Overhead: The authors demonstrate that adding the I-JEPA loss to a DINOv3-based photometric framework yields superior results compared to training with photometric supervision alone. These gains are achieved without altering the inference-time model, as the auxiliary components are removed post-training.
Enhanced Generalization: The method shows improved robustness under domain shift. Models trained solely on the KITTI dataset achieve state-of-the-art or near-state-of-the-art zero-shot performance on Cityscapes and Make3D without fine-tuning.

Experimental Results
The authors evaluated JEPADepth on the KITTI, Cityscapes, and Make3D benchmarks:

KITTI (In-Domain): On the standard Eigen split, JEPADepth achieved an AbsRel of 0.101 and $\delta_1$ of 0.898, outperforming strong CNN-based baselines (e.g., Monodepth2, HR-Depth) and remaining competitive with transformer-based approaches like MonoViT, despite using fewer parameters (24M vs. 27M). On the improved ground truth benchmark, it tied MonoViT on AbsRel (0.075).
Zero-Shot Transfer:
- Cityscapes: Trained only on KITTI, JEPADepth tied for the best AbsRel (0.143) and achieved the best performance on five out of seven metrics (including RMSE and $\delta_1$ ) compared to other methods.
- Make3D: The model achieved the best AbsRel (0.275) and RMSElog (0.143) among all compared methods, representing a relative gain of 3.8% over MonoViT and 7.7% over the strongest CNN baseline (DIFFNet).
Ablation Studies: Experiments confirmed that end-to-end fine-tuning of the DINOv3 encoder is essential (freezing it degrades performance to basic CNN levels). The benefits of the JEPA loss were shown to be tightly coupled with the quality of pretrained representations; random initialization yielded poor results. Furthermore, performing prediction in representation space was proven superior to pixel-space reconstruction.

Significance and Claims
The paper claims that JEPADepth offers a practical and effective enhancement to existing self-supervised depth pipelines. By leveraging masked predictive learning in representation space, the method regularizes the depth network to understand spatial relationships and scene structure more robustly than pixel-reconstruction alone. The authors emphasize that these improvements in accuracy and cross-dataset generalization are achieved with a modest training overhead (~16% increase in wall-clock time) and zero additional cost at inference. The work suggests that integrating predictive representation learning objectives can effectively bridge the gap between static feature extraction and dynamic geometric reasoning in self-supervised vision tasks.

The Paper's Big Idea: Teaching by Filling in the Blanks

What They Found: Smarter, Not Just Bigger

The Trade-Off: A Little Extra Homework, No Extra Homework Later

What the Paper Says It's Not

The Bottom Line

More like this