Factorized Latent Dynamics for Video JEPA: An Empirical Study of Auxiliary Objectives

Technical Summary: Factorized Latent Dynamics for Video JEPA

Problem Statement
Joint-Embedding Predictive Architectures (JEPA) have emerged as a promising framework for self-supervised video representation learning, notably through Video-JEPA (V-JEPA), which predicts latent embeddings of masked spatiotemporal regions rather than reconstructing pixels. However, the behavior of auxiliary objectives in small-scale V-JEPA training remains poorly characterized. A critical gap exists in understanding whether auxiliary signals can simultaneously improve fine-grained temporal dynamics (essential for tasks like action recognition) and static appearance representation (essential for object recognition), or if they introduce capacity trade-offs where gains in one domain degrade performance in another.

Methodology
The authors conduct a small-scale empirical study evaluating 18 auxiliary objective variants across two pretraining regimes: single-dataset (UCF-101) and mixed-dataset (UCF-101 + Something-Something V2 + ImageNet-100). The study utilizes a frozen-encoder evaluation protocol on three complementary benchmarks:

• Diving-48: Fine-grained motion understanding.
• Something-Something V2 (SSv2): Temporal reasoning.
• ImageNet-100: Appearance recognition.

The 18 variants are categorized into six groups:

1. Kinematic Regularization: Penalties on temporal differences (L1, Huber, acceleration, split-channel, annealed).
2. Motion-Guided Masking: Biased sampling toward high-motion regions.
3. Anti-Collapse Regularization: SIGReg (enforcing Gaussian-distributed embeddings).
4. Physics-Inspired Dynamics: Hamiltonian and velocity-gated priors.
5. Latent Dynamics Prediction: Predicting temporal differences (Delta-JEPA, LD-JEPA, Spectral-JEPA, LTC-JEPA).
6. Factorized Latent Objectives: Structurally separating appearance and dynamics subspaces.

Key Contribution: FWM-HW-LD
The paper introduces FWM-HW-LD (Factorized World-Model with Hard-Region-Weighted Latent Dynamics) as a novel training-time objective. Unlike a new inference architecture, it modifies the training process of the standard V-JEPA student encoder by:

• Latent Factorization: Partitioning the encoder's output into two subspaces: $Z_{app}$ (appearance, encouraged to be temporally invariant) and $Z_{dyn}$ (dynamics, free to capture temporal variation).
• Orthogonality and Stability: Applying losses ($L_{static}$ and $L_{orth}$) to ensure $Z_{app}$ remains stable over time and that the two subspaces do not encode redundant information.
• Hard-Region Weighting: Re-weighting both the standard JEPA prediction error and the latent dynamics error toward tokens with the highest prediction errors, focusing the model on difficult regions.
• Latent Dynamics Head: A dedicated head predicts temporal differences ($h(t+1) - h(t)$) using only the $Z_{dyn}$ subspace during training.

Results

• Capacity Trade-offs: The study finds that many auxiliary objectives exhibit significant trade-offs. For instance, kinematic regularization variants improved ImageNet-100 (appearance) by 1.5–1.7 percentage points (pp) but degraded Diving-48 (motion) by 2.5–2.9 pp. Similarly, pixel-prediction objectives (AC-JEPA, FAC-JEPA) resulted in substantial drops (−13 to −16 pp) on ImageNet-100.
• FWM-HW-LD Performance: In the mixed-dataset setting, FWM-HW-LD achieved the most balanced results among the tested variants. It improved ImageNet-100 by +5.92 pp and SSv2 by +3.21 pp relative to the baseline, while remaining within −0.30 pp on Diving-48.
• Component Interaction: Ablation studies revealed that the components of FWM-HW-LD interact non-additively. While Latent Dynamics (LD) alone boosted SSv2 (+5.02 pp), it hurt other benchmarks. Factorization (FWM) alone improved ImageNet but hurt SSv2. The full combination with hard-weighting was necessary to achieve the balanced improvement.
• Synthetic vs. Real Motion: Kinematic regularization showed strong performance (+40–45 pp) on synthetic motion discrimination tasks but did not translate to real-world fine-grained motion benchmarks, suggesting a mismatch in the type of temporal structure learned.

Significance and Claims
The paper modestly claims that latent factorization is a useful direction for studying auxiliary-objective trade-offs in Video-JEPA. The authors emphasize that their findings are based on a small-scale, single-seed empirical study and should be viewed as preliminary signals rather than statistically established rankings.

Key takeaways include:

1. Trade-off Existence: Auxiliary objectives often force a competition between motion-discriminative and appearance-sensitive capabilities within a shared latent space.
2. Structural Separation: Separating the latent space into appearance and dynamics subspaces, combined with hard-region weighting, offers a pathway to mitigate these trade-offs without requiring pixel-level reconstruction.
3. Caution on Pixel Prediction: In this specific small-scale setup, objectives that predict pixel-level RGB deltas (AC-JEPA) were detrimental to semantic recognition tasks.

The authors conclude that while FWM-HW-LD shows promise, the evidence is preliminary. They position factorized latent dynamics not as a solved objective, but as a controlled mechanism to investigate how appearance and motion information compete or cooperate within frozen video representations, suggesting a need for larger-scale, multi-seed replication to validate these findings.