💻 computer science

The JEPA Predictor: A Transferable Operator for Occluded Feature Completion

This paper demonstrates that the predictor component of Joint-Embedding Predictive Architectures (JEPAs) can be frozen and transferred via a simple linear projection to non-JEPA encoders, significantly boosting their performance on occluded inputs by effectively completing missing features without requiring retraining of the underlying models.

Original authors: William Nguyen, Christopher Nguyen

Published 2026-07-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: William Nguyen, Christopher Nguyen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to recognize a friend in a crowded room, but someone has placed a giant, opaque box over half their face. Your brain is amazing at filling in the blanks; it uses the visible ear, the hair, and the context of the room to guess what the hidden face looks like. In the world of artificial intelligence, computers have become incredibly good at recognizing things when they can see the whole picture. However, when parts of an image are missing—like a car partially hidden behind a tree or a medical scan cut off by a metal implant—these smart computers often get confused and make mistakes.

For years, scientists have built two main types of "smart eyes" (AI models) to solve this. One type learns by trying to guess missing pieces of a puzzle (predicting the future based on the present), while another type learns by comparing different pictures to find similarities. Usually, when these models are finished training, the part that does the "guessing" is thrown away because it's only needed during the learning phase. The final product is just the "eye" that sees the picture. But what if that discarded "guessing machine" was actually a superpower we forgot to use? This paper asks a simple, playful question: Can we take a "guessing machine" from one type of AI and bolt it onto a different type of AI to help it see through the fog of missing information?

The researchers, William and Christopher Nguyen from Aitomatic, Inc., discovered that the answer is a resounding yes. They found that the "predictor" part of a specific AI architecture called JEPA (Joint-Embedding Predictive Architecture) acts like a universal translator for missing information. Even though this predictor was trained on one specific type of AI, they showed that it can be "plugged in" to four completely different, popular AI models (like CLIP, DINOv3, DINOv2, and MAE) using a simple mathematical bridge.

Here is how their experiment worked: Imagine you have a photo of a dog, but 77% of it is covered by a black rectangle. A standard AI model, looking only at the tiny sliver of the dog that is visible, might guess it's a cat or a rock. The researchers took a "frozen" (unchanged) predictor from a JEPA model and connected it to the other AI models. When the AI saw the visible part of the dog, it passed that information through a simple linear bridge to the JEPA predictor. The predictor then used its training to "hallucinate" or predict what the features of the hidden 77% of the dog should look like. Finally, the system combined the real visible parts with the predicted hidden parts to make a final guess.

The results were striking, especially when the images were heavily blocked. On a dataset of dog breeds, a standard AI model (CLIP) dropped to a terrible accuracy of just 15.9% when 77% of the image was hidden. But once they attached the JEPA predictor, its accuracy skyrocketed to 52.1%. That is a massive jump of 36 percentage points. On another dataset with fewer categories, the model recovered almost all the accuracy it had lost due to the missing pieces.

The paper explains that this works because of a "cost-benefit" balance. The simple bridge connecting the two models isn't perfect; it slightly distorts the information of the parts of the image that are visible (the cost). However, the predictor is incredibly good at guessing the parts that are missing (the benefit). When an image is mostly visible, the distortion cost is too high, and the system doesn't help much. But when the image is heavily blocked (like 77% hidden), the benefit of having a good guess for the missing parts completely overwhelms the small cost of the distortion. The predictor essentially becomes a portable "feature completion" tool that works across different AI families without needing to retrain the main models.

The researchers also tested what happens if the missing parts are scattered randomly like static on a TV screen, rather than a solid block. In that case, the predictor didn't help much, suggesting it is specifically designed to handle large, contiguous chunks of missing information, like a tree blocking a car or a crop in a photo. They also proved that this wasn't just the AI "smoothing out" blurry edges or filling in pixels; it was actually reconstructing high-level features that allowed the computer to understand the object's identity.

In short, this paper shows that we don't always need to build a new, giant AI from scratch to handle missing data. We can take a specialized "guessing engine" that was built for one job and, with a simple mathematical adapter, use it to supercharge other AI models, helping them see clearly even when the picture is broken.

Technical Summary: The JEPA Predictor as a Transferable Operator for Occluded Feature Completion

1. Problem Statement

Frozen image encoders are frequently deployed on partial inputs (e.g., occluded autonomous-driving frames or truncated medical scans). As the fraction of masked patches ( $K$ ) increases, even the strongest discriminative encoders (such as CLIP, DINOv2, and MAE) suffer steep accuracy degradation. While Masked Image Modeling (MIM) and contrastive baselines exist, they typically degrade further under heavy occlusion.

Joint-Embedding Predictive Architectures (JEPAs) train a predictor head to map visible-context features to target-position features. However, standard downstream protocols discard this predictor, using only the encoder. This paper investigates a critical gap: Can the frozen JEPA predictor, traditionally discarded, serve as a portable "feature completion" operator for non-JEPA encoders? Specifically, can it recover accuracy lost to occlusion without retraining the host encoder or the predictor?

2. Methodology

The proposed protocol connects a frozen "host" encoder (non-JEPA) with a frozen "donor" JEPA predictor via a lightweight bridge.

Cross-Encoder Bridge ( $W$ ): To pair a non-JEPA host with a JEPA donor, a single linear projection $W$ is fitted between the host's feature space and the donor JEPA encoder's feature space. This is done via closed-form ridge least squares on 500 unmasked ImageNet-1k images. $W$ is fitted once per host-donor pair and reused across all mask fractions and datasets.
Test-Time Protocol:
1. Encoding: The host encoder processes the visible (unmasked) patches of an input image.
2. Projection: The visible features are projected via $W$ into the donor JEPA's feature space.
3. Prediction: The frozen JEPA predictor consumes the projected visible features and the positions of the masked patches to generate predicted features for the masked regions.
4. Stitching: The sequence is reconstructed by combining the projected visible features and the predicted masked features.
5. Classification: A linear probe, trained specifically on the same mask fraction $K$ used at test time (a "matched probe"), classifies the completed feature sequence.
Constraints: No backbone parameters (host or donor) are updated. No gradients flow through the encoder or predictor during inference. The only trainable component is the linear probe fitted per mask fraction.

3. Key Contributions

Protocol Introduction: A method to wire a frozen JEPA predictor to any frozen image encoder via a single closed-form linear projection, requiring no backbone retraining.
Empirical Benchmarking: Evaluation across four non-JEPA hosts (CLIP, DINOv3, DINOv2, MAE), two JEPA donors (I-JEPA, V-JEPA 2), three mask fractions ( $K \in \{0.25, 0.56, 0.77\}$ ), and two datasets (ImageNet-9, Stanford Dogs).
Mechanism Identification: Characterization of a cost-benefit dynamic where the linear projection incurs a fixed cost on visible patches, while the predictor provides a growing benefit on masked patches. This explains why the lift is monotonic with $K$ and where the linear bridge fails (low $K$ on fine-grained tasks).

4. Results

Monotonic Lift: Across all host-donor pairs, the accuracy lift ( $\Delta$ ) over the host's masked-encoder baseline grows monotonically with the mask fraction $K$ .
Heavy Occlusion Performance:
- On ImageNet-9 at $K=0.77$ , CLIP paired with the I-JEPA predictor recovered most of the accuracy lost to masking, achieving a +23.46 percentage point (pp) lift (reaching 97.08% vs. 73.62% baseline).
- On Stanford Dogs (fine-grained 120-way) at $K=0.77$ , the same pairing lifted accuracy from 15.87% to 52.05% (+36.18 pp).
- Seven of eight host-donor pairs showed statistically significant positive lifts (95% CI > 0) on both datasets at heavy occlusion.
Host Dependency: The lift is inversely correlated with the host's native occlusion robustness. CLIP (contrastive, least robust to occlusion) showed the largest gains, while DINOv3/DINOv2 (self-distillation, more robust) showed smaller but still positive lifts.
Within-JEPA Specialization: When a JEPA encoder is paired with its own native predictor (no bridge), it maintains near-full-image accuracy even at $K=0.77$ , outperforming the strongest encoder-only baselines by significant margins.
Mechanism Validation:
- Cost-Benefit: The linear bridge $W$ is "lossy" (low $R^2$ on feature variance), paying a cost on visible patches. However, the predictor's benefit on masked patches dominates at high $K$ .
- Failure Mode: At low $K$ (e.g., $K=0.25$ ) on fine-grained tasks, the projection cost exceeds the predictor's marginal benefit, resulting in negative lifts. This defines the boundary where the linear bridge breaks down.
- Contiguity: The lift is specific to contiguous occlusion (center-rect masks). It collapses on random Bernoulli masks where local neighbors are available, ruling out simple local smoothing as the cause.
- Latent vs. Pixel: The JEPA latent completion significantly outperforms pixel-space reconstruction (MAE decoder) followed by re-encoding, confirming the value of feature-space extrapolation.

5. Significance and Claims

The paper claims that the JEPA predictor functions as a portable operator for occluded feature completion that generalizes across encoder families.

Transferability: The predictor learns a function of feature-position-and-context that is respected by diverse encoder geometries, not just its native JEPA family.
Diagnostic Shift: The work argues that for reused predictors where pretraining loss does not constrain outputs to the target encoder's metric (e.g., V-JEPA 2), matched-probe accuracy is the appropriate diagnostic, rather than L2 reconstruction error.
Practicality: The approach requires no retraining of large models, only a one-time closed-form fit of a small linear projection and per-mask-fraction linear probes.
Limitations: The authors note the lift vanishes at low mask fractions ( $K < 0.25$ ) and the method is currently limited to contiguous occlusion and natural image distributions. The linear bridge fails on fine-grained tasks at low occlusion, suggesting a need for non-linear bridges in future work.

The paper concludes that the conventional ranking of self-supervised encoders based on full-image accuracy does not hold uniformly when a transferable completion module is introduced; the "best" encoder for occluded inputs depends on the mask fraction and the availability of a compatible predictor.