🤖 AI

LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives

LeVLJEPA introduces the first fully non-contrastive end-to-end vision-language pretraining method that utilizes cross-modal prediction without negatives, demonstrating superior performance in generating dense semantic features for downstream tasks like visual question answering and semantic segmentation while maintaining competitiveness in global readouts.

Original authors: Lukas Kuhn, Giuseppe Serra, Randall Balestriero, Florian Buettner

Published 2026-07-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lukas Kuhn, Giuseppe Serra, Randall Balestriero, Florian Buettner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand the world by showing it pictures and reading it stories about those pictures. For a long time, the best way to do this was like playing a high-stakes game of "spot the difference" with a huge crowd.

The Old Way: The "Spot the Difference" Game (Contrastive Learning)

Think of traditional methods (like CLIP) as a teacher holding up a photo of a cat and a story saying "This is a cat." To make sure the robot learns, the teacher also shows it 10,000 other photos and stories that don't match. The robot has to learn: "Okay, this specific photo matches this story, but it definitely does not match those 10,000 other things."

This works well for simple tasks, like looking at a picture and guessing, "Is this a cat or a dog?" (Zero-shot classification). But it has a flaw: the robot learns to focus on the big picture to win the game of "match vs. no-match," often ignoring the tiny, detailed parts of the image. It's like studying for a test by only memorizing the cover of the book, ignoring the chapters inside.

The New Way: The "Blindfolded Translator" (LeVLJEPA)

The authors of this paper, LeVLJEPA, asked a bold question: Can we teach the robot without ever showing it the "wrong" answers (the negatives)?

They built a new system that works like a Blindfolded Translator game:

The Setup: You have two experts. One sees the picture, the other reads the story.
The Game: The "Picture Expert" tries to guess what the "Story Expert" is thinking, and vice versa.
The Twist: The person being guessed at is "frozen" (they can't change their mind or give feedback). The guesser has to figure it out entirely on their own.
The Safety Net: To make sure the guesser doesn't just give up and say "I don't know" for everything (which is called "collapse"), they have to follow a rule: their guesses must be spread out evenly, like a fair distribution of cards, rather than clumping together.

This method uses no negative examples, no complex temperature settings, and no massive crowds of "wrong" answers. It just relies on prediction and a simple rule to keep things balanced.

The Big Surprise: The "Detail" vs. The "Summary"

The researchers tested this new method against the old "Spot the Difference" methods. Here is what they found:

The Summary Test (Global Features): If you ask the robot, "What is the main thing in this picture?" (like a simple classification), the old methods and the new method perform almost exactly the same. They are both good at the "big picture."
The Detail Test (Dense Features): This is where the magic happens. Modern AI systems don't just look at the "main thing"; they look at every single patch of the image to understand complex scenes, like finding a specific tool in a messy garage or following a robot's arm.
- The old "Spot the Difference" methods are like a tourist who only remembers the skyline of a city.
- The new LeVLJEPA method is like a local guide who knows every street, alley, and shop.

When the researchers used their new method as the "brain" for complex vision-language models (like robots that follow instructions or answer detailed questions), it crushed the competition. It was significantly better at:

Semantic Segmentation: Identifying exactly which pixels belong to which object (like drawing a perfect outline around a car).
Visual Question Answering: Answering tricky questions like "What is the red object sitting on the blue table?"

The Takeaway

The paper concludes that the old way of training (using negatives) is great for simple "match or no-match" games, but it's actually holding back the development of AI that needs to see the fine details.

By switching to this new "prediction without negatives" approach, they created a visual encoder that is much better at seeing the world in high definition, pixel-by-pixel, which is exactly what modern AI systems need to function in the real world.

In short: The old method taught the AI to recognize the cover of the book. The new method taught the AI to read the entire story, page by page.

Technical Summary: LeVLJEPA

Problem Statement

Vision-language pretraining (VLP) has historically been dominated by contrastive objectives (e.g., CLIP, SigLIP), which optimize the alignment of a single pooled image embedding with a text embedding. While vision-only self-supervised learning has largely shifted toward non-contrastive, predictive methods (e.g., DINO, I-JEPA, LeJEPA), VLP has not followed this trajectory.

This discrepancy creates a misalignment between training objectives and modern deployment needs. Vision-language encoders are increasingly used not as zero-shot classifiers but as frozen visual backbones for dense prediction systems (e.g., semantic segmentation) and vision-language models (e.g., LLaVA). These downstream systems consume the full grid of patch tokens rather than a single pooled vector. Contrastive objectives, however, primarily supervise the global pooled embedding, treating per-token features only as a byproduct. The paper asks: Can end-to-end vision-language pretraining work without contrastive negatives to better serve these dense, token-level applications?

Methodology: LeVLJEPA

The authors introduce LeVLJEPA, the first fully non-contrastive end-to-end vision-language pretraining method. It adapts the LeJEPA framework to the cross-modal setting using a predictive architecture with specific constraints to prevent representation collapse without negative pairs.

Architecture and Objective

The method employs two encoders: a Vision Transformer (ViT) for images and a GPT-2 based encoder for text. Both produce embeddings ( $Z_V$ and $Z_T$ ) which are passed through modality-specific predictors (MLPs) to generate cross-modal predictions.

The training objective, $L_{LeVLJEPA}$ , combines two components:

Cross-Modal Prediction with Stop-Gradients:
- The image encoder predicts the text embedding, and the text encoder predicts the image embedding.
- Crucially, the targets are detached via a stop-gradient operation. Gradients flow only through the predictor and the encoder producing the prediction, not back into the encoder producing the target.
- Loss: $L_{cross} = \frac{1}{B} \sum (\|h_v(z_v) - \text{sg}(z_t)\|^2 + \|h_t(z_t) - \text{sg}(z_v)\|^2)$ .
- This asymmetry prevents the encoders from collapsing into a shared low-dimensional subspace, a failure mode observed when using direct symmetric regression.
Per-Modality Distributional Regularization (SIGReg):
- To prevent collapse without negative pairs, each modality's embedding distribution is regularized to be an isotropic Gaussian.
- This is achieved using SIGReg (Sketched Isotropic Gaussian Regularization), which projects embeddings onto random 1D directions and applies a characteristic-function-based normality test (Epps–Pulley).
- This avoids the curse of dimensionality associated with direct density matching and requires no temperature parameters, momentum encoders, or teacher-student schedules.

Training Protocol

Datasets: Evaluated at two scales: CC12M (12M samples) and Datacomp-L (92M samples).
Stability: The method trains stably without the batch-size dependence characteristic of contrastive methods. The objective is computed per-sample, making it largely invariant to batch size.
Ablations: Experiments confirm that removing any component (e.g., using direct MSE without stop-gradients, or using predictors without SIGReg) leads to collapse or poor performance.

Key Results

1. Global Representation Quality (Pooled Embeddings)

When evaluated on protocols that reduce the image to a single vector:

Zero-Shot Transfer: At the Datacomp-L scale, contrastive baselines (InfoNCE, SigLIP) outperform LeVLJEPA. This is expected, as zero-shot classification directly measures the image-text alignment that contrastive methods optimize. At the smaller CC12M scale, the gap narrows significantly.
Linear Probing: On frozen CLS tokens, LeVLJEPA is on par with contrastive baselines across ImageNet, Places365, Aircraft, and Pets. This indicates that the global visual features learned by LeVLJEPA are equally discriminative, and the zero-shot gap is due to alignment mechanisms rather than feature quality.
Background Robustness: LeVLJEPA demonstrates superior robustness to background shifts (ImageNet-9), showing smaller accuracy drops when foregrounds are composited onto different backgrounds. This suggests a more object-centric representation.

2. Dense Semantic Features (Patch Tokens)

When evaluated on tasks consuming the full sequence of patch tokens, LeVLJEPA shows marked superiority:

Semantic Segmentation: On ADE20K and COCO-Stuff, LeVLJEPA outperforms both contrastive baselines by a significant margin (~2.0–2.2 mIoU) when used as a frozen backbone with a linear head. This indicates stronger spatial and semantic structure in the per-token features.
Vision-Language Models (VLMs): When used as a frozen visual backbone for VLMs (with a lightweight bridge trained to a frozen LLM), LeVLJEPA achieves the highest accuracy on GQA, VQAv2, and POPE across two distinct language model families (Llama-1B and Qwen-1.5B).
- Notably, SigLIP is the strongest zero-shot model but the weakest VLM backbone, while LeVLJEPA is the weakest zero-shot model but the strongest backbone.
- LeVLJEPA also yields the most balanced responses on POPE (object hallucination), reducing the "yes" bias seen in other models.

Significance and Claims

The paper establishes that non-contrastive pretraining is an effective means of producing dense semantic vision features, challenging the assumption that contrastive negatives are necessary for high-quality vision-language representations.

Decoupling Alignment and Feature Quality: The results demonstrate that the representational property yielding a strong zero-shot classifier (global alignment) is distinct from the property yielding a strong visual feature encoder (dense token structure).
Relevance to Modern Deployment: As vision-language encoders are increasingly deployed as frozen backbones for dense prediction and instruction-tuned models, the standard evaluation protocols (zero-shot, linear probing on pooled vectors) fail to capture the relevant performance metrics. LeVLJEPA excels in these deployment scenarios.
Simplicity and Stability: The method achieves these results with a simpler training recipe, eliminating the need for negative sampling, temperature tuning, momentum encoders, or teacher-student schedules, while remaining stable at large scales.

The authors conclude that while contrastive objectives retain an advantage in direct image-text alignment, non-contrastive predictive objectives are superior for learning the dense, per-token features required by the next generation of vision-language systems.