💻 computer science

AV-JEPA: Extending LeJEPA to Audio-Visual Self-Supervised Learning

This paper introduces AV-JEPA, a streamlined audio-visual self-supervised learning framework that extends LeJEPA using an early-fusion Vision Transformer and modality dropout to achieve competitive performance and zero-shot retrieval capabilities without relying on decoders, EMA teachers, or contrastive negatives.

Original authors: Benjamin Robson, Santeri Mentu, Wenshuai Zhao, Arno Solin

Published 2026-07-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Benjamin Robson, Santeri Mentu, Wenshuai Zhao, Arno Solin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers are like eager, blank-slate students who want to learn about everything, but they are stuck in a classroom with no teacher and no textbooks. This is the realm of self-supervised learning, a branch of artificial intelligence where machines try to teach themselves by finding patterns in mountains of unlabeled data. Usually, to learn how to recognize a cat or a car, a computer needs a human to point at a picture and say, "That's a cat." But what if the computer could learn just by looking at the picture and guessing what's missing? That's the magic of masked learning: you cover up part of an image or sound, and the computer has to fill in the blanks. For a long time, the best way to do this was to force the computer to rebuild the missing pieces pixel-by-pixel, like a puzzle solver trying to reconstruct a shattered vase. But there's a newer, more abstract idea called JEPA (Joint-Embedding Predictive Architecture). Instead of trying to rebuild the broken vase, JEPA asks the computer to predict the essence or the "vibe" of the missing piece. It's like guessing the flavor of a soup by smelling the steam, rather than trying to taste every single ingredient. This paper asks a big question: Can this "guessing the vibe" method work when the computer is trying to understand two different senses at once—sight and sound?

The authors of this paper, a team from Finland, say "Yes!" and introduce a new model called AV-JEPA. Think of AV-JEPA as a super-smart detective who learns to understand a crime scene by looking at the video and listening to the audio, but with a twist. Instead of trying to reconstruct the missing video frames or audio waves (which is like trying to redraw the whole crime scene from memory), this detective learns to predict the feeling of the scene when one sense is missing.

Here is how they did it. Imagine you are watching a video of a guitar player. Usually, a computer sees the video and hears the audio together. AV-JEPA, however, plays a game of "hide and seek" with the senses. It takes a clip and creates two versions: one where the video is completely blacked out (so the computer only hears the guitar), and another where the audio is silenced (so the computer only sees the guitar). The model's job is to look at the "audio-only" version and predict what the "video-only" version would feel like, and vice versa. It does this without any human telling it what is happening, and without trying to rebuild the missing parts. It just learns to align the "soul" of the sound with the "soul" of the image in a shared mental space.

The results are quite impressive for a method that skips the heavy lifting of rebuilding images. When they tested this model on a dataset called VGGSound (which contains videos of sounds), the model achieved a 57.1% accuracy rate in identifying what was happening in the video. On another massive dataset called AudioSet, it scored a 32.7 mAP (a measure of how well it identifies sounds). While these numbers are slightly lower than the current top models that use the old "rebuild the puzzle" method (which can reach up to 67% on VGGSound), the authors point out that AV-JEPA is much simpler. It doesn't need complex extra parts to rebuild images, nor does it need a "teacher" model to guide it. It just learns by itself.

One of the coolest things the paper found is that the model learned to pay attention to the right things all on its own. When they looked at where the model was "looking" when it heard a sound, it naturally focused on the source of the noise. If it heard a bird chirping, it looked at the bird. If it heard a flute, it looked at the musician's mouth. It did this without anyone ever teaching it to find the sound source; it just figured out that the sound and the visual source were connected because it had to predict one from the other.

The paper also showed that this model can do "zero-shot" retrieval, which is a fancy way of saying it can find a video just by searching for a sound, or find a sound just by searching for a video, without ever being explicitly trained to do that specific search task. It's like having a library where you can find a book just by humming a tune from it.

However, the authors are honest about the limitations. They found that the model is heavily biased toward the audio. When they tested it, the model's decisions were mostly driven by what it heard, not what it saw. It's as if the detective is a genius at hearing clues but is a bit distracted by the visual evidence. The paper suggests that while this is a huge step forward for a "clean" and simple architecture, there is still work to be done to make it as good at seeing as it is at hearing, especially compared to the more complex models that spend years rebuilding images. But for a method that throws away the reconstruction puzzle and just focuses on the connection between senses, AV-JEPA is a very promising new way for computers to learn how the world sounds and looks.

Technical Summary: AV-JEPA

Problem Statement
Self-supervised learning (SSL) in the audio-visual domain has been dominated by masked autoencoding (MAE) approaches (e.g., AV-MAE, CAV-MAE, MAViL). While effective, these methods rely on reconstructing masked audio spectrograms and video patches using dedicated decoder networks. This paradigm introduces architectural overhead, requires careful tuning of masking ratios, and often necessitates contrastive negative-pair losses or complex multi-term objectives. Furthermore, these reconstruction-based methods lack formal theoretical guarantees regarding the quality of the learned embeddings. The Joint-Embedding Predictive Architecture (JEPA), which operates in the latent space by predicting one view's embedding from another, offers a promising alternative. However, prior JEPA implementations, specifically LeJEPA, have been validated only on single-modality vision tasks and have not been extended to cross-modal audio-visual alignment.

Methodology
The authors propose AV-JEPA, an extension of LeJEPA to audio-visual self-supervised learning. The architecture and training pipeline are designed to achieve cross-modal alignment without decoders, EMA teachers, or contrastive negatives.

Architecture: The model utilizes an early-fusion Vision Transformer (ViT-Base). Raw audio (resampled to 16 kHz) is converted to a mel-spectrogram, and video frames are processed as tubelets. Both modalities are tokenized and concatenated into a single sequence alongside a learnable [CLS] token. Modality-type embeddings distinguish between audio and video tokens. A shared 12-layer ViT encoder processes this joint sequence.
View Generation (Modality Dropout): Instead of spatial masking, AV-JEPA employs modality dropout as the partial-view mechanism. For each training clip, the model generates:
- Global Views: Both audio and video modalities are present (with light augmentation).
- Local Views: Two views are created where one modality is zeroed out (audio-only and video-only).
  This setup creates an implicit cross-modal prediction task where the model must predict the joint representation from a single modality, entirely within the latent space.
Objective Function: The training minimizes the LeJEPA loss, which consists of two terms:
1. Invariance: Minimizes the distance between individual view embeddings ( $z_i$ ) and the joint-view center ( $\bar{z}$ ), forcing single-modality embeddings to align with the joint representation.
2. SIGReg (Sketched Isotropic Gaussian Regularization): Enforces that the embedding distribution matches an isotropic Gaussian ( $N(0, I)$ ). This is theoretically proven to minimize downstream prediction risk. The regularization uses a sliced characteristic-function test (Epps–Pulley) to match univariate projections of the embeddings to the Gaussian target.
Training: The model is pre-trained on AudioSet-2M and fine-tuned on VGGSound and AudioSet. During fine-tuning, classification heads (linear and attentive probes) are attached to the [CLS] token.

Key Contributions

Extension of JEPA to Audio-Visual Learning: The first application of JEPA-based SSL to cross-modal audio-visual representation learning.
Cross-Modal View Generation: Introduction of modality dropout to create alternating audio-only and video-only local views, enabling implicit cross-modal prediction in the latent space without reconstruction targets.
Clean Architecture: A unified early-fusion ViT backbone that eliminates the need for decoders, stop-gradients, EMA teachers, or contrastive negative pairs.
Empirical Validation: Demonstration that the resulting backbone achieves competitive performance on standard benchmarks and supports zero-shot cross-modal retrieval.

Results

VGGSound Classification: After pre-training on AudioSet-2M and fine-tuning, AV-JEPA achieves 57.1% top-1 accuracy using an attentive probe and 56.6% with a linear probe. This represents the first JEPA-based result at this level of accuracy for audio-visual classification.
AudioSet Classification: The model achieves 32.7 mAP on the AudioSet evaluation split. Per-modality analysis reveals a strong audio-driven representation (26.0 mAP audio-only vs. 12.8 mAP video-only), consistent with the dominance of audio in the dataset's classes.
Cross-Modal Retrieval: The model supports zero-shot audio-to-video and video-to-audio retrieval out of the box. On VGGSound and AudioSet, Recall@10 reaches approximately 36%, significantly above chance levels, with symmetric performance in both directions.
Qualitative Analysis: Attention maps from the final transformer layer show that the model naturally attends to sound-producing regions in video (e.g., instruments, mouths) and harmonic/temporal structures in audio, despite receiving no localization supervision.

Significance and Claims
The paper positions AV-JEPA as a theoretically grounded, architecturally clean alternative to the prevailing masked autoencoding pipelines for multimodal learning. By leveraging the LeJEPA framework, the authors demonstrate that high-quality cross-modal alignment can be achieved without the complexity of reconstruction decoders or contrastive losses.

The authors acknowledge limitations, noting that the current performance gap with state-of-the-art MAE-based methods (which reach 63–67% on VGGSound and 42–53 mAP on AudioSet) likely stems from the absence of reconstruction/contrastive objectives, the lack of a video-specific pre-training stage, and the absence of ImageNet initialization for the visual stream. They suggest that future work involving longer pre-training, larger models, and hybrid initialization strategies could further close this gap. Nevertheless, the work establishes that JEPA is a viable and promising paradigm for audio-visual self-supervised learning.

Technical Summary: AV-JEPA

More like this