From nucleotides to semantics: genomic representation learning via joint-embedding predictive architecture

This paper introduces GenoJEPA, a genomic foundation model that leverages a joint-embedding predictive architecture to shift from computationally expensive nucleotide reconstruction to efficient semantic alignment in latent space, thereby achieving robust generalization across 55 downstream tasks with reduced parameters and enabling lightweight, GPU-free classification.

The Gist

Imagine you are trying to teach a computer to understand the "language of life" (DNA). For a long time, scientists have treated DNA like a human language, trying to teach computers to read it word-by-word, letter-by-letter, just like they teach them to read English or French.

But here's the problem: DNA isn't actually a language. It's more like a natural landscape, such as a forest or a mountain range.

• Human Language has clear rules: spaces between words, capital letters for names, and punctuation. It's dense with meaning.
• DNA is a continuous, messy stream of four letters (A, C, G, T). It has no spaces, no capital letters, and it's full of "noise"—random evolutionary changes that don't mean anything.

The Old Way: The "Pixel-by-Pixel" Painter

Previous AI models tried to learn DNA by acting like a painter trying to recreate a photo pixel-by-pixel. They would hide a part of the DNA sequence and ask the AI, "What letter was here?"

• The Flaw: Because DNA is so noisy, the AI spent all its energy memorizing the tiny, meaningless details (the "static" on a TV screen) instead of learning the big picture (the actual shape of the mountain).
• The Cost: To get the AI to be useful for a specific task (like finding a disease gene), scientists had to "fine-tune" it heavily. This is like hiring a master painter and then making them repaint the whole canvas from scratch for every single new picture. It requires massive computers and a lot of time, which many biology labs can't afford.

The New Way: GenoJEPA (The "Mood Ring" Approach)

This paper introduces GenoJEPA, a new way to teach AI about DNA. Instead of trying to recreate the exact letters, GenoJEPA learns the "vibe" or "mood" of the DNA.

Here is how it works, using a simple analogy:

1. The "Patch" Strategy (Looking at the Forest, not the Leaves)

Instead of looking at DNA one letter at a time, GenoJEPA looks at it in chunks (like looking at a patch of a forest rather than a single leaf). It treats these chunks as continuous signals, similar to how a computer vision AI looks at a patch of pixels in an image. This helps it ignore the tiny, noisy mutations and focus on the bigger biological patterns.

2. The "Semantic Alignment" (Matching the Vibe)

This is the core magic.

• Imagine you have two photos of the same forest: one taken in the morning and one in the afternoon. They look different (different lighting, shadows), but they are the same place.
• Old AI models tried to force the computer to match every single leaf in the morning photo to the afternoon photo.
• GenoJEPA says: "Don't worry about the leaves. Just make sure the computer understands that both photos represent a 'forest'."

It aligns the "meaning" (semantics) of the DNA in a hidden, high-dimensional space. It teaches the AI to recognize that a specific chunk of DNA is a "promoter" (a switch that turns genes on) or an "enhancer," regardless of the tiny random noise around it.

3. The "Frozen" Superpower (The Swiss Army Knife)

Because GenoJEPA learns the meaning rather than just memorizing the letters, it becomes incredibly efficient.

• The Analogy: Think of GenoJEPA as a Swiss Army Knife that is already fully sharpened and ready to use.
• Old Models: Were like a block of raw steel. You had to spend hours sharpening and shaping them (fine-tuning) just to turn them into a screwdriver or a knife.
• GenoJEPA: You can take it out of the box, attach a simple, cheap handle (a lightweight classifier), and it works immediately.

Why This Matters for Everyone

The paper shows that GenoJEPA is:

1. Smarter: It learns the "rules" of biology better than models 10 to 100 times its size.
2. Faster & Cheaper: It doesn't need supercomputers to be useful. A standard laptop can run the "frozen" version to analyze DNA.
3. Data Efficient: It can learn from very small amounts of data, which is great because biological data is often hard to get.

In a nutshell:
Previous AI models tried to memorize the dictionary of life letter-by-letter. GenoJEPA teaches the AI to understand the story of life by looking at the big picture, ignoring the noise, and allowing scientists to use powerful tools without needing a massive budget. It turns genomic research from a "supercomputer-only" club into something accessible to any biology lab.

Technical Summary

1. Problem Statement

Current genomic foundation models largely adopt paradigms from Natural Language Processing (NLP), treating DNA as a language and utilizing pretraining objectives like Masked Language Modeling (MLM) or Next-Token Prediction (NTP). The authors identify several critical limitations in this approach:

• Structural Mismatch: Unlike human language, which has high information density and explicit semantic boundaries, genomic sequences resemble natural images. They lack predefined boundaries, contain substantial evolutionary noise, and have a low signal-to-noise ratio.
• Inefficient Reconstruction: NLP-based objectives often force models to reconstruct low-dimensional, discrete nucleotides (pixel-level details). This encourages models to fit high-frequency evolutionary noise rather than learning biologically relevant regulatory syntax, leading to representations with limited discriminative power.
• Downstream Bottlenecks: Because pretraining often fails to filter noise effectively, downstream tasks require expensive, full-parameter fine-tuning. This restricts the practical application of large models in biology laboratories with limited computational resources.
• Tokenization Issues: Conventional tokenization (BPE, k-mers) introduces vocabulary redundancy and is sensitive to single-nucleotide mutations, while single-nucleotide tokenization incurs quadratic computational costs.

2. Methodology: GenoJEPA

The authors propose GenoJEPA, a framework based on Joint-Embedding Predictive Architecture (JEPA), specifically adapting the LeJEPA formulation. The core philosophy shifts optimization from input-space reconstruction to latent-space semantic alignment.

Key Architectural Components:

• Continuous Patching (Tokenization):
  • Instead of discrete tokens (BPE/k-mers) or single nucleotides, the input DNA sequence is partitioned into non-overlapping nucleotide patches (e.g., size 16).
  • These patches are mapped via linear projection to continuous dense feature vectors. This avoids vocabulary inflation, preserves local biochemical dependencies, and significantly shortens the effective sequence length.
• Backbone Architecture:
  • Based on ModernBERT, a bidirectional encoder incorporating Rotary Position Embeddings (RoPE) and bias-free designs.
  • Two scales are instantiated: GenoJEPA-T (6M parameters) and GenoJEPA-B (52M parameters).
• Pretraining Objective (LeJEPA):
  • Multi-View Augmentation: Sequences are augmented into multiple "views" (2 global views with 65–80% cropping; 6 local views with 35–40% cropping) to capture both global structure and local motifs.
  • Semantic Alignment: All view representations are aligned to the mean representation of the global views in a high-dimensional latent space using Mean Squared Error (MSE). This forces the model to learn invariant features robust to local noise.
  • Anti-Collapse Regularization (SIGReg): To prevent representation collapse (where all inputs map to the same vector), the model employs Sketched Isotropic Gaussian Regularization. This uses the Empirical Characteristic Function (ECF) to guide the latent feature distribution toward an isotropic Gaussian without relying on heuristic devices like momentum encoders or stop-gradients.
• Training Data: Pretrained on a multispecies corpus of 850 representative species (bacteria, fungi, invertebrates, vertebrates) totaling ~193 billion nucleotides.

3. Key Contributions

1. Paradigm Shift: Introduces the first JEPA-based framework for genomics, moving away from nucleotide reconstruction to latent semantic alignment, arguing that this better suits the "image-like" nature of DNA.
2. Continuous Patching Strategy: Proposes a CV-inspired continuous patching method for DNA that reduces sequence length and computational cost while preserving biochemical continuity, avoiding the pitfalls of discrete tokenization.
3. Resource Efficiency: Demonstrates that high-quality representations can be learned with significantly fewer parameters (52M vs. 494M for NT-v2) and less compute, making foundation models accessible to resource-constrained labs.
4. Frozen-Backbone Utility: Shows that GenoJEPA produces representations so discriminative that they can be used with frozen backbones and lightweight, GPU-free classifiers (e.g., Logistic Regression) to achieve competitive accuracy, eliminating the need for expensive fine-tuning in many scenarios.

4. Experimental Results

The model was evaluated on 55 downstream tasks across three benchmarks: Genomic Benchmarks, GUE Benchmarks, and Nucleotide Transformer (NT) Tasks.

• Probing Performance (Frozen Backbone):
  • GenoJEPA-B achieved the highest overall performance, winning the majority of tasks compared to baselines (HyenaDNA, CaduceusPh, GROVER, DNABERT-2, NT-v2).
  • GenoJEPA-T (6M params) outperformed NT-v2 (494M params, ~100x larger) in probing tasks, demonstrating superior representation quality per parameter.
  • The model excelled in tasks requiring structural motif recognition (e.g., transcription factor binding, splice site recognition, promoter detection).
• Fine-tuning Performance:
  • Even with full fine-tuning, GenoJEPA-B remained competitive, outperforming NT-v2 by 2.9% in average MCC despite having 10x fewer parameters.
• Data Efficiency (Few-Shot):
  • GenoJEPA maintained high performance with only 10% of the training data, approaching the performance of baselines trained on full datasets. This suggests the learned representations generalize well with limited labels.
• Computational Efficiency:
  • Training/Inference: GenoJEPA demonstrated lower runtime and memory usage compared to larger Transformers and even newer sub-quadratic architectures (HyenaDNA, CaduceusPh) across various sequence lengths.
  • Stability: Unlike some baselines that encountered Out-Of-Memory (OOM) errors at moderate sequence lengths, GenoJEPA maintained stable memory usage up to 16,384 bp.

5. Significance and Impact

• Democratizing Genomic AI: By enabling high-performance downstream analysis with frozen backbones and lightweight classifiers, GenoJEPA lowers the barrier to entry for biological laboratories lacking high-end GPU clusters.
• Theoretical Insight: The results validate that genomic sequences are better modeled as continuous signals with semantic boundaries (like images) rather than discrete linguistic tokens. This suggests that latent-space alignment is a more effective objective for extracting regulatory syntax than nucleotide reconstruction.
• Scalability: The framework provides a practical, efficient path for scaling genomic foundation models, balancing performance with computational feasibility.

In conclusion, GenoJEPA represents a significant step forward in genomic representation learning by aligning the architecture and objectives of foundation models with the intrinsic physical and biological properties of DNA, rather than forcing a linguistic analogy.