🤖 machine learning

Human-JEPA: A Human-Centric Vision Model that Perceives and Anticipates

Human-JEPA is a human-centric vision model trained on video via anchored forecasting that simultaneously achieves state-of-the-art static perception and future anticipation with significantly fewer parameters than existing specialists, overcoming previous limitations in motion understanding and model collapse.

Original authors: Hui Wei, Licai Sun, Guoying Zhao

Published 2026-08-24

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Hui Wei, Licai Sun, Guoying Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Human vision is a dual engine. It does not merely record what is happening right now; it constantly runs a simulation of what will happen next. When you watch someone reach for a cup, your brain simultaneously identifies the person, tracks their posture, and predicts exactly where their hand will land before it arrives. For decades, artificial intelligence has struggled to replicate this second half of the equation. While machines have become exceptionally good at analyzing a single photograph—identifying a person's face, their clothing, or their pose—they have remained largely blind to the flow of time. They can describe a still image with great detail, but they cannot anticipate the future movement within a video. This gap has left a significant portion of human understanding out of reach for computers, limiting their ability to interact with the world in a way that feels natural or safe.

A new study introduces a system called Human-JEPA, designed to bridge this divide. The researchers built a model that learns to perceive the present and anticipate the future at the same time, using only video footage without any human labels or manual annotations. The core of their discovery lies in how they trained the system. Previous attempts to teach machines about human movement often failed because the learning process itself was flawed. When the system tried to learn from video, it would quietly lose its ability to understand the details of the human body, such as the shape of a limb or the texture of clothing, while focusing too much on copying static patterns. The researchers found that by anchoring the system's memory of the human form to a fixed starting point and forcing it to predict the future rather than just fill in missing parts of the past, they could prevent this collapse. The result is a model that is significantly smaller than previous leaders in the field yet outperforms them in tracking human movement and identifying individuals, all while maintaining the ability to guess what happens next.

The challenge the team faced was that existing powerful models were trained on static images. These models are excellent at reading a photograph, but they do not understand motion. To create a system that understands people in motion, the researchers started with a video-based model that had already learned general patterns from the internet. They then specialized it to focus on humans. However, simply showing this model more videos of people did not work. The system would begin to degrade, losing its ability to recognize fine details like body parts or clothing, a failure so subtle that the training process did not even show an error signal. The model was essentially forgetting what a human looks like while trying to learn how they move.

To solve this, the researchers introduced a method they call anchored forecasting. Imagine a student trying to learn a new skill while being told to forget the basics they already know; the student would likely become confused and lose their foundation. The researchers prevented this by "pinning" the system's understanding of the human form to a frozen, unchanging copy of its original knowledge. This anchor ensured that as the model learned to predict future movements, it did not lose its ability to recognize the present. Additionally, they added a stream of training data using still images of people, which helped the system maintain a sharp sense of human appearance. This combination allowed the model to keep its detailed perception of the human body while learning to anticipate motion.

The second major change involved how the system was tested during training. Standard methods often ask the model to fill in missing blocks of a video, which encourages it to simply copy what it sees nearby in time and space. This is a shortcut that prevents the model from truly learning how a scene evolves. The researchers replaced this with a pure past-to-future split. The model was shown the first half of a video clip and asked to predict the second half. Because the future was completely hidden, the only way to succeed was to build a genuine model of how the scene changes over time. This forced the system to learn the dynamics of human movement rather than just memorizing static patterns.

The results of this approach were striking. When tested on frozen checkpoints—meaning the system was not allowed to learn further during the test—the new model outperformed the largest and most specialized human-vision models available, despite having 2.7 times fewer parameters. It achieved higher accuracy in tracking human poses and identifying individuals in a crowd. For example, on a standard test for re-identifying people, it improved its score by 2.7 points over its own base version and surpassed the leading image-based specialist. Perhaps most importantly, the model's ability to anticipate the future was not degraded by these changes. In fact, the system's predictor head was the first to improve anticipation without harming its other capabilities.

The study also rigorously tested what did not work, ruling out several intuitive ideas. The researchers tried making the person themselves the unit of prediction, masking out entire people to see if the model could learn to track their partners. This approach failed completely, collapsing the model's ability to understand interactions. They also found that simply adding more data or increasing the model's size did not fix the underlying issues caused by the training method. The failure was not due to a lack of data or computing power, but rather a fundamental flaw in how the learning objective was structured. The study concludes that the key to understanding humans in time is not just to watch them, but to anchor the system's perception of their form while forcing it to predict their future.

This work establishes a new direction for artificial intelligence. It demonstrates that a single model can successfully handle both the static perception of the present and the dynamic anticipation of the future, two tasks that were previously thought to require separate approaches. By preventing the silent collapse of detailed perception and removing the shortcuts that hinder true learning, the researchers have created a system that sees people as they are: entities that exist in time, moving and changing. The findings suggest that for machines to truly understand humans, they must be taught to look forward, not just to look back.

Technical Summary: Human-JEPA

1. Problem Statement

Current human-centric vision models, such as Sapiens2 and HAP, excel at static, dense perception (e.g., pose estimation, body parsing, and re-identification) by leveraging pixel-anchored encoders pretrained on massive human image corpora. However, these models are fundamentally limited to static images; they cannot represent motion or anticipate future states. Conversely, existing video-based self-supervised models (like V-JEPA) learn temporal dynamics but lack the specialized human-centric representations required for high-fidelity pose and identity tasks.

The core challenge addressed by this paper is the inability of current systems to simultaneously perceive the present (static human attributes) and anticipate the future (temporal evolution of human actions) within a single, unified model. Furthermore, attempts to simply continue pretraining a video JEPA on human data often lead to a "silent collapse" of dense perception capabilities, where the model loses its ability to resolve fine-grained human details despite improving on the training objective.

2. Methodology: Human-JEPA

Human-JEPA is built upon the architecture of the released V-JEPA 2.1 (a Joint Embedding Predictive Architecture pretrained on general internet video). The authors retain the original encoder and predictor architecture but introduce specific modifications to the training objective and data strategy to specialize the model for humans without destroying its temporal capabilities.

2.1 Anchored Continued Pretraining

To prevent the "silent collapse" of dense perception observed during continued pretraining, the authors introduce an anchoring mechanism:

Frozen Initialization Anchor: Instead of using an Exponential Moving Average (EMA) teacher that drifts with the student (which causes dense targets to drift and collapse), the dense context loss is supervised by a frozen copy of the initialization ( $f_0$ ). This pins the dense perception targets to the original high-quality features.
Image Co-training Branch: To further preserve appearance features, an auxiliary branch trains on single-frame person crops (from the LUPerson-T dataset) rendered as one-frame clips. This branch carries full-body appearance information that video-only training often fails to maintain.

2.2 Forecasting Masks (Past-to-Future Split)

Standard JEPA training often uses block masking (inpainting), where spatial blocks are masked across all frames. The authors argue this encourages the model to perform a "copy task" (copying appearance across time) rather than learning motion dynamics.

Pure Past-to-Future Split: Human-JEPA replaces block masking with a temporal split where the context consists of the first half of the video tubelets, and the target is the second half.
Temporal Specialization: This forces the predictor to model scene evolution rather than spatial interpolation. This design avoids the "action tax" (performance drop on action recognition) and "re-identification collapse" associated with block masking.

2.3 Omissions

The model explicitly omits person-level masking and pose inputs. The authors found that treating people as the unit of prediction or providing skeleton priors degrades performance by allowing the model to rely on static body configurations rather than learning scene-level dynamics.

3. Key Contributions

First Simultaneous Perception and Anticipation: Human-JEPA is the first human-centric vision model capable of both dense static perception and future anticipation.
Anchored Forecasting: The paper introduces a training recipe that specializes a video JEPA for humans without destroying its dense features. This is achieved through a frozen-initialization anchor (preventing collapse) and a past-to-future masking strategy (specializing temporal representation).
Systematic Evaluation: The authors provide a comprehensive evaluation under a unified frozen-probe protocol, including a causal "partner-ablation" probe that falsifies the effectiveness of person-level prediction objectives.

4. Experimental Results

The model was evaluated using frozen probes across seven tracks: dense parsing, body part segmentation, pose, identity (ReID), and anticipation.

4.1 Perception (Pose and Identity)

Pose Estimation: Human-JEPA achieves a mean of 0.620 AP on pose estimation (averaged over two pretraining seeds, ±0.003), outperforming the pixel-anchored specialist Sapiens2-0.8B (0.591 AP) by 2.9 AP, despite having 2.7 times fewer parameters (0.3B vs. 0.8B).
Person Re-Identification (ReID): On Market-1501, Human-JEPA achieves 0.4635 mAP (seed 1), surpassing Sapiens2 (0.4522 mAP) and improving by 2.7 mAP over its own base (V-JEPA 2.1, seed 1: 0.4370 mAP).
The Cost: This specialization comes with a trade-off. Human-JEPA concedes performance in high-resolution dense parsing (ATR mIoU drops from ~0.767 for Sapiens2 to ~0.739) and DensePose. The authors frame this as a "division of labor": pixel-anchored models own static appearance, while latent video prediction owns dynamics.

4.2 Anticipation

Predictor Head: The released predictor head for Human-JEPA is the first that does not degrade anticipation performance. In contrast, the base model's shipped head actually decreased anticipation performance by 2.98 points on NTU-120 early action.
Performance: The full Human-JEPA bundle gains 4.1 points over the base's released bundle on anticipation tasks.
Forecasting Capability: Direct evaluation shows the predictor agrees with true future latents at 0.873 cosine similarity, significantly outperforming static references (0.798) and demonstrating genuine forecasting rather than copying.

4.3 Ablation Findings

Mask Family: Replacing block masks with the forecasting split recovers action performance and prevents ReID collapse. Block masks cause a 17-point drop in ReID mAP.
Anchoring: Without the frozen anchor and image branch, continued pretraining leads to a silent collapse of dense perception (Pose AP drops from 0.614 to 0.110) while training loss continues to decrease.
Person-Level Objectives: A causal probe falsified all nine person-level prediction objectives constructed by the authors, showing they fail to learn relational dynamics and often degrade scene-level forecasting.

5. Significance and Claims

The paper claims that Human-JEPA represents a shift in how human vision models are specialized. It demonstrates that:

Specialization requires structural changes, not just more data: Simply training longer on human data causes collapse; specific architectural changes (anchoring and masking) are necessary.
Static and Dynamic understanding are distinct: The paper posits a division of labor where pixel-anchored pretraining is superior for static high-resolution appearance, while latent video prediction (Human-JEPA) is superior for temporal dynamics and anticipation.
Causal Validation: The authors introduce a causal partner-ablation probe to rigorously test relational learning, finding that current "person-centric" masking strategies do not actually teach models to understand human interactions.

The authors conclude that Human-JEPA successfully unifies the two halves of understanding humans—perceiving the present and anticipating the future—within a single model, setting a new standard for human-centric video representation learning.