Technical Summary: Behavioral Geometric Supervision Aligns Video Foundation Models with Human Social Perception

1. Problem Statement

Current video foundation models, even state-of-the-art self-supervised architectures like V-JEPA 2, fail to capture how humans organize social information in dynamic scenes. While these models excel at action classification (e.g., distinguishing "dancing" from "cooking"), they systematically under-explain human social similarity judgments compared to language models. Specifically, caption-based sentence embeddings (e.g., MPNet) consistently outperform pretrained video encoders in predicting human similarity judgments for social video clips. This indicates a "representation gap": video pretraining objectives miss the relational and affective structure inherent in human social perception. Furthermore, existing alignment methods often rely on expensive fMRI data ("brain-tuning") or focus on static images, leaving the question open whether scalable behavioral supervision can induce human-like social visual structure in video models.

2. Methodology

2.1 Dataset Construction

The authors constructed a large-scale human similarity benchmark for dynamic social video:

• Stimuli: 250 naturalistic, 3-second two-person social interaction clips from the Moments in Time dataset.
• Task: A triplet odd-one-out (OOO) paradigm where participants identify the "odd" video among three, implicitly defining the remaining pair as more similar.
• Scale: 49,484 judgments collected from 245 participants.
• Structure: A greedy set-cover algorithm ensured every video pair appeared in at least one triplet. The data was split into 200 training videos and 50 test videos (stimulus-disjoint).
• Ground Truth: Human similarity is represented as a Representational Similarity Matrix (RSM) based on the frequency of pairwise similarity judgments.

2.2 Behavioral Geometric Supervision (BGS)

The core contribution is Behavioral Geometric Supervision (BGS), a hybrid training objective designed to align the embedding geometry of video models with human similarity judgments.

• Architecture: Applied via Low-Rank Adaptation (LoRA) to four heterogeneous Vision Transformer (ViT) backbones: V-JEPA 2, V-JEPA 2.1, TimeSformer, VideoMAE, and CLIP. This updates fewer than 2% of parameters.
• Hybrid Loss Function ($L_{hybrid}$): Combines local and global constraints:
  1. Triplet Loss ($L_{triplet}$): Enforces local constraints where the distance between an anchor and a positive pair is smaller than the distance to a negative (odd-one-out) pair by a margin $\gamma$.
  2. RSA Loss ($L_{RSA}$): Enforces global geometry by aligning the model's pairwise distance matrix with the human RSM using Representational Similarity Analysis (RSA). This term is applied 6 times per epoch on a subset of videos to manage memory.
• Training Schedule: The model is fine-tuned for 50 epochs. The weight of the RSA loss ($\beta$) is linearly ramped from 0.3 to 0.7, prioritizing local triplet correctness early and global structure later.

2.3 Controls and Baselines

To isolate the mechanism of improvement, the authors compared BGS against:

• Pretrained Baselines: 12 video encoders, 2 image encoders, 2 VLMs, and 22 sentence-embedding language models (using video captions).
• Ablations: Triplet-only loss, RSA-only loss, and a budget-matched triplet variant.
• Language Distillation Control: A model trained with the same hybrid loss but using MPNet caption embeddings as the supervisory target instead of human behavioral judgments.

2.4 Evaluation Metrics

• Primary: Spearman correlation squared ($R^2$) between model RSMs and human RSMs (split-half reliability ceiling $R^2 = 0.213$).
• Secondary:
  • Variance Partitioning: Decomposing explained variance into unique video, unique language, and shared components.
  • Linear Probes: Testing for emergent social-affective attributes (valence, arousal, dominance, intimacy, communication) without explicit supervision.
  • OOD Generalization: Zero-shot transfer to the PHASE dataset (abstract geometric social interactions).
  • Action Recognition: Preservation of performance on UCF101.
  • Attention Analysis: Spatial attention rollout to observe shifts in focus.

3. Key Results

3.1 Closing the Performance Gap

• Pretrained Deficit: No pretrained video model matched the performance of the best sentence encoder (MPNet, $R^2 = 0.134$). The best pretrained video model (X3D-M) reached only $R^2 = 0.124$.
• BGS Success: Fine-tuning with BGS significantly improved performance. The best model, V-JEPA 2.1 (distilled), reached $R^2 = 0.165$ (78% of the noise ceiling), and V-JEPA 2 (large) reached $R^2 = 0.172$. Both exceeded the MPNet baseline.
• Necessity of Hybrid Objective: The hybrid loss outperformed triplet-only ($R^2 = 0.158$) and RSA-only ($R^2 = 0.131$) variants. Crucially, the language distillation control failed to reproduce these gains ($R^2 = 0.123$), ruling out simple caption transfer as the mechanism.

3.2 Emergent Social-Affective Structure

• Unique Variance: Variance partitioning revealed that fine-tuned models captured unique variance in human judgments that caption-based embeddings could not ($R^2$ unique video variance doubled from 0.02 to 0.04).
• Unsupervised Attributes: Despite never being trained on attribute labels, BGS-aligned models developed linearly decodable representations of valence, arousal, and dominance. A language distillation control failed to recover these attributes on 4 out of 5 dimensions.

3.3 Generalization and Preservation

• Zero-Shot Transfer: BGS-aligned models achieved 81% accuracy on the PHASE dataset (abstract social interactions), matching human agreement levels and significantly outperforming the pretrained baseline (55%). This demonstrates generalization to out-of-distribution (OOD) stimuli.
• Action Recognition: Fine-tuning preserved (and slightly improved for V-JEPA 2.1) action recognition performance on UCF101, indicating no catastrophic forgetting of pretraining capabilities.

3.4 Mechanistic Shifts

• Attention Reorganization: Attention rollout analysis showed a systematic shift in spatial focus. Pretrained models concentrated on single focal regions (e.g., a torso), while fine-tuned models redistributed attention to socially informative regions: faces, gaze directions, interacting hands, and both agents. This shift occurred without any spatial supervision during training.

4. Significance and Claims

The paper claims that social perception in video models is a latent property that becomes accessible through lightweight behavioral supervision. The key contributions are:

1. Demonstration of a Representation Gap: Current video pretraining misses social-visual structure, which is better captured by language models until specific behavioral alignment is applied.
2. Scalable Alignment via BGS: Behavioral Geometric Supervision successfully induces human-like social similarity structure in diverse video backbones using a compact signal (LoRA, <2% parameters) and behavioral data, without the infrastructure costs of fMRI-based "brain-tuning."
3. Beyond Language Distillation: The gains are not merely due to learning from captions. BGS captures unique visual-social structure (e.g., gaze, interaction dynamics) that language embeddings miss, and it recovers interpretable social-affective dimensions without explicit labeling.
4. Robustness: The method generalizes to abstract, out-of-distribution social interactions and preserves standard action recognition capabilities.

The authors conclude that relational behavioral data offers a distinct form of alignment, steering video models toward human-like social visual understanding by shaping the geometric structure of embeddings to match human similarity judgments.