🧬 biology

JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures

JEPA-DNA introduces a model-agnostic continual training framework that integrates Joint-Embedding Predictive Architectures with traditional generative objectives to shift genomic foundation models from token-level reconstruction to semantic alignment, achieving state-of-the-art performance across diverse genomic benchmarks.

Original authors: Ariel Larey, Elay Dahan, Amit Bleiweiss, Raizy Kellerman, Guy Leib, Omri Nayshool, Dan Ofer, Tal Zinger, Dan Dominissini, Gideon Rechavi, Nicole Bussola, Simon Lee, Shane O'Connell, Dung Hoang, Mariss

Published 2026-08-12

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Ariel Larey, Elay Dahan, Amit Bleiweiss, Raizy Kellerman, Guy Leib, Omri Nayshool, Dan Ofer, Tal Zinger, Dan Dominissini, Gideon Rechavi, Nicole Bussola, Simon Lee, Shane O'Connell, Dung Hoang, Marissa Wirth, Alexander W. Charney, Nati Daniel, Yoli Shavit

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine the human body as a massive, ancient library. Inside this library, every single instruction for building and running a person is written in a code made of just four letters: A, C, G, and T. This code is DNA. For a long time, scientists have been trying to teach computers to read this library, hoping that if a computer understands the "grammar" of DNA, it can predict how our bodies work, why we get sick, or how to fix genetic glitches.

To do this, researchers use something called "Foundation Models." Think of these as super-smart students who have read millions of books (DNA sequences) and learned the rules of the language. Usually, these students learn by playing a game of "fill in the blank." If you cover up a word in a sentence, the student has to guess what it was based on the words around it. This works great for learning the local rules of the sentence, like how words fit together. But here's the problem: knowing the grammar doesn't always mean you understand the story. A student might know that "The cat sat on the..." is usually followed by "mat," but they might not understand that the cat is actually hiding from a dog, or that the whole scene is part of a larger mystery. In DNA, this means computers are good at spotting small patterns but often miss the big picture of how different parts of the code work together to control life.

This is where a new study called JEPA-DNA comes in. The researchers, a team from NVIDIA and medical centers in Israel and the US, decided to upgrade these DNA-reading students. They introduced a new training method that forces the computer to stop just guessing the missing letters and start guessing the meaning of the missing parts. Instead of asking, "What letter goes here?" they ask, "What is the job of this whole chunk of DNA?"

The team tested this new method on 17 different tasks, ranging from finding where genes start (promoters) to predicting how genetic changes affect health. They tried it on three different types of DNA-reading computers (DNABERT-2, NTv3, and HyenaDNA) to make sure it worked for everyone. The results were impressive: by adding this new "meaning-focused" training, the computers got better at almost every task. For example, on one task called "GUE Promoter," the new method improved performance by over 11%. On another task involving disease variants, it boosted the ability to spot problems by more than 12%.

The paper suggests that this approach works because it teaches the computer to look at the "forest" instead of just the "trees." While the old methods were great at memorizing the local syntax (the specific order of letters), JEPA-DNA helps the model understand the global functional context (what that sequence actually does in the body). The researchers found that this new way of learning creates a "world model" of biology, where the computer understands the rules of life, not just the spelling. They even showed that this method works better than the current best models available, setting a new high bar for what these AI systems can achieve. It's like taking a student who was good at spelling and teaching them to be a literary critic who understands the plot, the characters, and the theme all at once.

Technical Summary: JEPA-DNA

Problem Statement

Genomic Foundation Models (GFMs) have advanced the modeling of DNA sequences by adapting Large Language Model (LLM) architectures to genomic data. However, current GFMs predominantly rely on generative self-supervised objectives, such as Masked Language Modeling (MLM) or Next-Token Prediction (NTP). While effective at capturing local sequence syntax and identifying motifs, these token-level reconstruction paradigms often prioritize high-frequency sequence "noise" (e.g., repetitive elements) over global functional context. Consequently, models may achieve high syntactic fidelity without internalizing the high-level regulatory logic or long-range interactions that govern genomic function. There is a critical gap between genomic syntax and biological semantics that existing generative objectives fail to bridge.

Methodology: JEPA-DNA Framework

To address this limitation, the authors introduce JEPA-DNA, a model-agnostic continual training framework that integrates a Joint-Embedding Predictive Architecture (JEPA) with traditional generative objectives. Rather than predicting raw tokens, JEPA-DNA forces models to predict the latent functional representations of masked genomic segments.

Architecture Components

The framework augments a standard GFM backbone ( $E_\theta$ ) with a JEPA branch consisting of:

Context Encoder ( $E_\theta$ ): Processes a masked input sequence $x$ to produce contextual latent representations $h_{ctx}$ .
Target Encoder ( $E_{\bar{\theta}}$ ): A structural duplicate of the context encoder with weights updated via Exponential Moving Average (EMA). It processes the unmasked target sequence $y$ to yield latent targets $z$ .
Predictor Head ( $P_\phi$ ): Maps context representations to the target latent space, learning to align context-derived features with the target encoder's embeddings.
Sequence Aggregator ( $A$ ): Compresses latent sequences into global representations (e.g., using [CLS] tokens, last tokens, or mean pooling) to facilitate global semantic alignment.

Training Strategy

JEPA-DNA employs a multi-objective continual training procedure:

Dual-Masking: Utilizes a span-based masking strategy with a scheduler that gradually increases masking ratios and span lengths, challenging the model to infer long-range dependencies.
Re-masking: Replaces masked positions in the context encoder's output with learnable latent [MASK] embeddings to prevent trivial mappings.
Composite Loss Function: The model is optimized by minimizing a total loss ( $L_{total}$ $L_{t o t a l}$ ) that balances four components:
1. LLM Loss ( $L_{llm}$ ): Standard MLM or NTP loss to maintain token-level precision.
2. Latent Predictive Loss ( $L_{jepa}$ ): A cosine similarity loss forcing the predicted global latent representation ( $\hat{z}_{glob}$ ) to align with the target global representation ( $z_{glob}$ ).
3. Variance Loss ( $L_{var}$ ): Ensures embedding dimensions maintain sufficient variance across the batch to prevent informational collapse.
4. Covariance Loss ( $L_{cov}$ ): Encourages decorrelation between embedding dimensions to prevent redundant representations (based on VICReg).

The framework is designed to be compatible with diverse architectures (Transformers, State Space Models like HyenaDNA) and tokenization methods, requiring only an aggregation operator to compress sequences into latent representations.

Key Contributions

Novel Training Framework: Introduction of a continual training framework that shifts the learning objective from literal token reconstruction to latent feature alignment via JEPA.
Architecture Agnosticism: Demonstration that JEPA-DNA acts as a universal "representational uplift" mechanism, compatible with leading GFMs (DNABERT-2, NTv3, HyenaDNA) and both MLM and NTP objectives.
State-of-the-Art Performance: Establishment of new performance ceilings across supervised linear probing tasks by bridging generative precision with latent semantic grounding.
Synergistic Analysis: Characterization of the interplay between generative and latent objectives, showing that a hybrid approach outperforms pure reconstruction or pure contrastive methods.

Experimental Results

The framework was evaluated on 17 diverse genomic benchmark tasks using linear probing and zero-shot protocols.

Linear Probing: JEPA-DNA demonstrated consistent gains across all three tested backbone architectures.
- DNABERT-2: Improved performance in 6–8 out of 9 tasks, with AUROC gains up to +8.0%.
- NTv3: Improved performance in 8 out of 9 tasks.
- HyenaDNA: Improved performance in 7 out of 9 tasks, with notable gains in promoter recognition (+11.22%) and splice site classification (+10.28%).
- Task Specificity: Gains were most pronounced in tasks relying on broader regulatory context (promoters, splice sites, meQTL, causal eQTL), while tasks driven by fine-grained local changes (coding pathogenicity) showed more variable results.
Zero-Shot Evaluation: Using DNABERT-2, JEPA-DNA improved zero-shot performance in 6 out of 8 clinical and fitness-scoring benchmarks, including a +12.06% AUROC gain on disease variant prediction.
Ablation Studies:
- VICReg Regularization: Identified as essential for unlocking the benefits of the hybrid paradigm.
- Predictor Architecture: Attention-based predictors (4 layers) outperformed MLPs and deeper architectures.
- Loss Composition: Latent-space alignment provided a more informative signal (+3.86% gain) than token reconstruction alone (+2.06%), but the optimal configuration required a hybrid of both.

Significance

The paper claims that JEPA-DNA establishes a new state-of-the-art for Genomic Foundation Models by successfully grounding them in biological semantics. By shifting the learning signal from token recovery to semantic alignment, the framework forces models to learn abstract, functional features that are invariant to low-level sequence noise. This approach bridges the gap between genomic syntax and biological logic, offering a scalable path to grounding foundation models without the computational overhead of multi-sequence alignment or the data requirements of contrastive negative mining. The authors position JEPA-DNA not as a replacement for existing GFMs, but as a versatile continual training phase that enhances the representational quality of any existing genomic foundation model.