Technical Summary: The JEPA Paradox in Language

1. Problem Statement

Joint-Embedding Predictive Architectures (JEPAs) have achieved significant success in continuous modalities like images, video, and audio by learning to predict target representations in a latent space from a masked context. However, this deterministic latent-prediction paradigm has not become a standard objective for text encoders, where distributional objectives (e.g., Masked Language Modeling) remain dominant.

The paper argues that this gap stems from a fundamental statistical mismatch between squared-error latent prediction and the conditional structure of language.

• Conditional Concentration: JEPA relies on the assumption that, given a context and a target location, the target representation lies near a single, geometrically meaningful point. This holds for images due to spatial continuity and local correlations.
• The Text Mismatch: In language, a single context often admits multiple valid lexical or semantic completions. Consequently, the conditional distribution of a masked token is multimodal. A squared-error predictor, which must output a single vector, is forced to regress toward a centroid of these heterogeneous alternatives rather than a coherent target. This forces the model to either average out meaningful distinctions (leading to poor representations) or collapse the representation space to minimize loss.

2. Methodology

The authors formalize the mismatch through three necessary conditions for useful JEPA learning and validate them by constructing T-JEPA, a text-based analogue of the image-based I-JEPA (Assran et al., 2023).

Theoretical Framework

The paper posits three conditions that must be met for deterministic latent prediction to work:

1. Predictability: The target must have low conditional entropy given the context. (Satisfied by spatial smoothness in images; limited by lexical ambiguity in text).
2. Non-Collapse: The objective must force distinct inputs to occupy distinct regions of representation space. (Visual diversity provides this pressure; text allows low-rank collapse without geometric penalty).
3. Low Conditional Variance: The reducible loss component must dominate the irreducible one. (In images, residual uncertainty is local texture; in text, multiple valid completions dominate the loss).

Experimental Setup

• I-JEPA (Baseline): Trained on 100K ImageNet-1K images using a ViT-H/16 backbone.
• T-JEPA (Proposed): A direct text analogue trained on 100K English C4 sentences using a BERT-Large backbone.
  • Architecture: Context encoder (BERT-Large), Target encoder (EMA-updated BERT-Large), and a Transformer predictor.
  • Objective: Minimize squared error between the predicted context embedding and the target embedding of a masked span.
  • Protocol: Matched training schedules (15 epochs, AdamW, specific warmup/decay) to ensure comparability. Experiments were repeated across five independent data seeds to rule out sampling artifacts.

Diagnostics

The study tracks five specific metrics to diagnose the failure mode:

1. Mutual Information (MI) proxy (InfoNCE).
2. Effective rank of the representation covariance.
3. Pairwise cosine similarity (and its dispersion).
4. Train-validation loss divergence.
5. Irreducible conditional target variance (estimated via K=16 candidate completions).

3. Key Results

The empirical evidence reveals a distinct "temporal failure signature" for T-JEPA that does not appear in I-JEPA:

1. Predictability Saturation Precedes Collapse: T-JEPA's MI proxy saturates early (staying below 0.35 nats), indicating that the masked target is weakly predictable from the context. This occurs while the effective rank is still non-trivial. In contrast, I-JEPA shows steadily increasing MI.
2. Loss Divergence and Instability: T-JEPA training and validation losses initially decrease together but then diverge catastrophically around step 9,340. Validation loss spikes while training loss becomes erratic, indicating the optimizer is memorizing batch-specific noise rather than learning generalizable structure.
3. Effective Rank Degeneration: Following the instability point, T-JEPA's validation effective rank collapses (dropping to ~1.57), while the training rank artificially inflates. This confirms the model merges distinctions among plausible targets to minimize loss.
4. Irreducible Variance: T-JEPA maintains high irreducible variance throughout the stable phase (reflecting lexical ambiguity). The variance only drops sharply after the representational collapse, not because ambiguity is resolved, but because the encoder compresses all embeddings into a narrow region.
5. Downstream Failure: T-JEPA representations perform at chance levels on classification benchmarks (IMDB, SNLI) and achieve zero performance on retrieval tasks (FEVER, MSMARCO). This contrasts sharply with BERT (MLM) and other self-supervised baselines (Barlow Twins, VICReg, BYOL), which retain useful structure.

4. Key Contributions

1. Conditional Concentration: The paper introduces "conditional concentration" as a necessary condition for deterministic latent prediction to yield useful representations. It argues that language often fails this condition due to inherent multimodality.
2. Centroid Degeneracy: It identifies "centroid degeneracy" as a specific failure mode where squared-error prediction forces the model to regress toward a centroid of incompatible alternatives, leading to representational collapse.
3. Empirical Diagnosis: The study provides a comprehensive diagnostic framework (MI, rank, cosine similarity, variance) showing that information saturation precedes representation collapse in text-based JEPA. This establishes a causal chain where the lack of conditional concentration weakens the prediction signal, creating pressure toward collapse.

5. Significance and Claims

The paper claims that its results do not rule out predictive representation learning for language entirely. Instead, it identifies a missing inductive bias: language prediction objectives must preserve the multimodal structure of valid completions rather than compressing them into a single latent centroid.

The authors suggest that successful text-based predictive learning likely requires distributional, contrastive, mixture-based, or semantic-level alternatives that can handle multiple plausible continuations, rather than the deterministic squared-error regression used in standard JEPA. The work serves as a theoretical and empirical explanation for why JEPA has not become a standard recipe for text encoders, contrasting the geometric constraints of vision with the combinatorial nature of language.