💻 computer science

RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture

RadJEPA is a self-supervised framework that learns robust radiology encoders for chest X-rays by predicting latent representations of masked image regions without language supervision, achieving state-of-the-art performance across multiple downstream medical tasks.

Original authors: Anas Anwarul Haq Khan, Mariam Husain, Kshitij Jadhav

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anas Anwarul Haq Khan, Mariam Husain, Kshitij Jadhav

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to understand chest X-rays. Usually, the best way to teach a computer about images is to show it a picture and then read the doctor's report out loud, saying, "This is a pneumonia," or "This heart is big." This is like teaching a child by showing them a picture of a dog and saying, "Dog."

However, the authors of this paper, RadJEPA, argue that this method has a flaw. Doctor's reports are written for humans making quick decisions; they often skip tiny details or subtle changes because they aren't relevant to the immediate diagnosis. If you only teach the computer what the doctor says, the computer might miss the subtle visual clues that are actually there.

So, the team asked: Can we teach the computer to understand X-rays without ever reading a single word of a doctor's report?

The New Approach: The "Fill-in-the-Blank" Game

Instead of using text, RadJEPA uses a method called Joint Embedding Predictive Architecture (JEPA). Think of this like a high-stakes game of "Fill-in-the-Blank" or a puzzle, but played in the computer's "brain" rather than on paper.

The Setup: The computer looks at a chest X-ray.
The Mask: The computer covers up (masks) a random chunk of the image, like putting a piece of tape over part of the X-ray.
The Guess: The computer looks at the visible parts (the healthy lung, the ribs, the heart) and tries to predict what the hidden, masked part looks like.
The Twist: It doesn't try to redraw the picture pixel-by-pixel (like a photocopier). Instead, it tries to guess the meaning or the "essence" of the missing part. It asks, "Based on the rest of the body, what should be in this hidden spot?"

If the computer guesses the "essence" correctly, it learns. If it guesses wrong, it adjusts its internal understanding. Over time, by playing this game millions of times on thousands of X-rays, the computer builds a deep, intuitive understanding of anatomy and disease without ever needing a human to tell it what it's seeing.

Why This is Better Than the Old Ways

The paper compares RadJEPA to two other popular ways of teaching computers:

The "Text-Teacher" (Vision-Language Models): These rely on the doctor's reports. As mentioned, reports can be biased or incomplete. RadJEPA ignores the text entirely, forcing the computer to learn from the image itself.
The "Mirror-Teacher" (Self-Distillation/DINO): These methods show the computer the same image in different ways (cropped, flipped, brightened) and tell it, "These are all the same picture." The computer learns to ignore the changes. The authors argue this makes the computer focus on the look of the image rather than the meaning of the anatomy. RadJEPA, by predicting the missing parts, forces the computer to understand the structure and relationships between body parts.

The Results: A Smarter, Smaller Model

The team tested their new "Fill-in-the-Blank" teacher (RadJEPA) on three difficult tasks:

Diagnosing Diseases: Can it tell if a patient has pneumonia or an enlarged heart?
Drawing Boundaries: Can it draw a line around a lung or a rib to separate it from the rest of the body?
Writing Reports: Can it help a language model write a medical report based on the image?

The surprising finding: RadJEPA beat the current "champions" (the best existing models) in all these tasks.

Even more impressively, RadJEPA achieved this while being much smaller than its competitors. Imagine a small, highly trained detective solving a case better than a giant, slow-moving bureaucracy. While other models needed massive amounts of computing power and data to learn, RadJEPA learned more efficiently because its "Fill-in-the-Blank" method taught it the right things to look for.

The Bottom Line

The paper claims that you don't need to pair X-rays with text to build a smart medical AI. By simply asking the AI to predict missing pieces of the image, you can create a "radiology encoder" that understands the visual language of the human body better than models trained on text. The authors have made their code and the trained model available for others to use and verify.

Technical Summary: RadJEPA

Problem Statement

Recent advances in medical vision–language models have driven the learning of visual representations through textual supervision. However, this paradigm is constrained by the availability of paired image–text data and relies on the assumption that radiology reports provide a complete and unbiased account of visual content. In practice, clinical reports emphasize selective findings for decision-making, often omitting subtle variations or absent observations. Consequently, encoders trained to align images with text may inherit biases and fail to preserve the full semantic structure of medical images.

Furthermore, existing self-supervised alternatives, such as DINO-style self-distillation, learn invariances by aligning global representations across multiple augmented views. In the context of chest X-rays, where fine-grained anatomical differences and subtle pathological cues are diagnostically critical, enforcing global agreement across views may prioritize appearance-level consistency over semantic completeness. These methods often emphasize pixel-space consistency rather than predicting semantic representations in latent space.

Methodology: RadJEPA

To address these limitations, the authors introduce RadJEPA (Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture), a self-supervised framework that learns without language supervision.

Core Architecture

RadJEPA is built on a Joint Embedding Predictive Architecture (JEPA). Unlike methods that reconstruct pixels or align global view representations, RadJEPA explicitly models latent-space prediction:

Input Partitioning: An input chest X-ray $x$ is partitioned into visible regions ( $x_v$ ) and masked regions ( $x_m$ ).
Encoding: A visual encoder $f_\theta$ maps these regions into latent representations $z_v = f_\theta(x_v)$ and $z_m = f_\theta(x_m)$ .
Prediction: A predictor network $g_\phi$ is trained to predict the latent representation of the masked regions ( $\hat{z}_m$ ) based solely on the latent representation of the visible context ( $z_v$ ).
Objective: The model minimizes the squared error between the predicted latent representation and the actual (stop-gradient) latent representation of the masked region:
$L = \|g_\phi(z_v) - \text{stopgrad}(z_m)\|^2$
Training Dynamics: The target encoder parameters are updated via an exponential moving average (EMA) of the online encoder, similar to teacher-student formulations, but the objective operates entirely in representation space without pixel reconstruction, contrastive negatives, or explicit data augmentations.

Downstream Adaptation

After pretraining on unlabeled chest X-rays, the visual encoder $f$ is frozen. Task-specific heads are trained independently for:

Disease Classification: A linear classifier on frozen global embeddings.
Semantic Segmentation: A UPerNet decoder aggregating multi-scale features from the frozen encoder.
Report Generation: A LLaVA-style multimodal setup where frozen visual embeddings are projected into the language embedding space and fed into a language model (Vicuna-7B).

Key Contributions

Limitations of Existing Paradigms: The paper demonstrates that both language supervision and DINO-style self-distillation have limitations in radiological representation learning, particularly regarding semantic completeness and fine-grained anatomical detail.
RadJEPA Framework: The authors introduce a predictive self-supervised architecture that learns directly from unlabeled chest X-ray images. It achieves performance comparable to or exceeding state-of-the-art (SoTA) models trained on over 839k chest X-ray images, despite using a smaller backbone (ViT-B/14, 86M parameters) compared to larger models like DINOv2 (1.1B) or I-JEPA (0.6B).
Comprehensive Evaluation: The work provides empirical evaluations across three distinct downstream tasks: disease classification, semantic segmentation, and report generation, benchmarking against a diverse set of baselines including larger-capacity architectures.

Experimental Results

The model was pretrained on 839,364 chest X-ray images from open-source datasets (BRAX, CheXpert, MIMIC-CXR, ChestX-ray14, PadChest).

Image Classification: On VinDr-CXR and RSNA-Pneumonia benchmarks, RadJEPA consistently outperformed image-only and vision-language baselines.
- On VinDr-CXR, it achieved a mean AUPRC of 55.2, surpassing RAD-DINO (52.8) and I-JEPA (50.0).
- Notable improvements were observed on subtle and heterogeneous findings such as pulmonary fibrosis (+4.5 AUPRC over RAD-DINO) and aortic enlargement.
- On RSNA-Pneumonia, it attained the best AUPRC (72.7) and AUROC (89.2).
Semantic Segmentation: Using a UPerNet decoder, RadJEPA achieved the highest Dice scores across lung (98.3), lung zone (93.7), and rib (89.6) segmentation tasks. It showed particularly strong gains on structurally complex targets like ribs (+4.8 Dice over RAD-DINO).
Report Generation: On MIMIC-CXR and the out-of-distribution IU-Xray dataset, RadJEPA achieved the strongest performance across all metrics (ROUGE-L, BLEU-4, RGER, Macro-F1-14) among image-only and predictive encoders.
- On MIMIC-CXR, it achieved a ROUGE-L of 26.1 and Macro-F1-14 of 32.6.
- On IU-Xray, it improved ROUGE-L by +1.7 over the strongest non-JEPA baseline.
Statistical Significance: Single-tailed hypothesis tests confirmed that 16 out of 19 evaluated cases showed statistically significant improvements ( $p < 0.05$ ) over baselines.

Significance

The paper argues that chest X-rays are fundamentally semantic rather than view-centric. Clinical interpretation relies on global anatomical context and spatial relationships, suggesting that learning representations directly in pixel space or enforcing view-level invariance is suboptimal. RadJEPA demonstrates that predictive latent modeling allows for the learning of robust, general-purpose radiology encoders without reliance on paired image–text data or traditional distillation-type architectures.

The results indicate that performance gains in medical imaging are not solely driven by model scale or dataset size, but by the quality of the learned representations. By achieving superior downstream performance with a smaller parameter count (ViT-B/14) compared to massive models, RadJEPA highlights the sample efficiency and representational quality of JEPA-style architectures in the medical domain. The authors conclude that strong radiology encoders can be learned effectively without language supervision, offering a reproducible path forward via the public release of code and weights.