SocialFusion: Addressing Social Degradation in Pre-trained Vision-Language Models

Technical Summary: SocialFusion: Addressing Social Degradation in Pre-trained Vision-Language Models

Problem Statement
While large-scale pre-trained Vision-Language Models (VLMs) demonstrate remarkable general capabilities, they fail to unify multiple visual social interaction understanding tasks simultaneously. Instead of leveraging synergies between tasks, existing VLMs exhibit "negative transfer," where joint training on diverse social tasks degrades performance compared to single-task training. The authors hypothesize that this failure stems from a phenomenon they term "social degradation." They posit that the standard visual-linguistic pre-training process of VLMs impairs the visual encoder's ability to represent nuanced social information, rendering the encoded features less decodable and less compatible for social perception tasks.

Methodology
The paper proposes SocialFusion, a unified framework designed to bypass social degradation by learning a minimal connection between a frozen visual encoder and a language model, rather than relying on the degraded representations of standard VLMs.

• Architecture:
  • Visual Encoder: SocialFusion utilizes a pre-trained foundation model (specifically CLIP in the primary experiments) that has not undergone the typical VLM pre-training pipeline. The visual encoder weights are frozen to preserve the quality of the original visual representations.
  • Connector: A lightweight, three-layer MLP projects the visual feature maps into the embedding space of the Large Language Model (LLM).
  • Language Model: The LLM (Llama 3.2 1B) is frozen, with only a Low-Rank Adaptation (LoRA) adapter applied to its linear layers to allow for task-specific learning.
  • Inputs & Outputs: The model accepts images and optional bounding boxes (converted to learned position embeddings). For classification tasks (gesture, expression, relationship, gaze direction), it outputs text tokens. For gaze target estimation, it projects the final visual features into a 2D heatmap via a linear projection and transposed convolution.
• Training Objective: The model is trained jointly on five distinct social tasks using a combined loss function: cross-entropy loss for text-based tasks and pixel-wise binary cross-entropy for gaze heatmap prediction.
• Investigation of Social Degradation: To validate the hypothesis, the authors conducted controlled experiments comparing visual encoders before and after VLM pre-training. They analyzed two mechanisms:
  1. Decodability: Using linear representation probing to measure how well social features can be linearly separated after pre-training.
  2. Compatibility: Using gradient conflict analysis (Gradient Conflict Degree) to measure the alignment of task gradients during joint training.

Key Contributions

1. Identification of Social Degradation: The paper identifies and empirically demonstrates that VLM pre-training degrades visual encoders' representations specifically for social interaction tasks. Controlled experiments show that encoders perform worse on social tasks after undergoing visual-linguistic pre-training compared to their pre-training counterparts.
2. Mechanism Analysis: Through linear probing and gradient conflict analysis, the authors reveal that the primary driver of social degradation is reduced linear decodability of the visual features, with gradient conflict playing a secondary, less significant role.
3. SocialFusion Framework: The proposal of a minimal fusion architecture that freezes the visual encoder and learns a lightweight connection to an LLM. This approach avoids the degradation inherent in standard VLMs.
4. Unified Positive Transfer: SocialFusion is the first model to achieve positive transfer across all five tested visual social interaction tasks (Gesture Analysis, Gaze/Attention, Facial Expression, Conversation Dynamics, and Social Situation Analysis) when trained jointly.

Results

• Baseline Comparison: When tested on popular open-source VLMs (Qwen2-VL, Sail-VL 1.5, and MolmoE) using LoRA fine-tuning, these models exhibited stark negative transfer in joint training scenarios, often performing worse than single-task training.
• SocialFusion Performance:
  • SocialFusion achieved positive transfer on all 10 tested metrics (two per task) in joint training.
  • It set new State-of-the-Art (SOTA) results on the HaGRIDv2 gesture recognition benchmark (99.8 mAP) and the PISC social situation analysis benchmark (94.2 mAP for Domain, 88.5 mAP for Relation).
  • It remained competitive on GazeFollow, LAM (Looking at Me), and AffectNet, though it did not surpass the SOTA on AffectNet or GazeFollow (where specialized models like DINOv2-based approaches still hold an edge).
• Synergy Analysis: Pairwise training experiments indicated that while some task pairs showed synergy, the full positive transfer effect was most pronounced when training on all diverse tasks simultaneously, suggesting the model learns generalizable "atomic social competencies."

Significance and Claims
The paper claims that current VLM pre-training strategies, while effective for general multimodal understanding, may be detrimental to acquiring general social competence. The findings suggest that the visual representations learned during large-scale pre-training do not preserve the specific visual cues necessary for social perception.

The significance of SocialFusion lies in demonstrating that positive transfer in complex, multi-faceted social domains is achievable not by building larger or more complex models, but by avoiding the representation degradation caused by standard pre-training pipelines. The authors argue that this highlights a critical need for socially-aware training paradigms in future VLM development, suggesting that specialized domains like social interaction may require distinct pre-training or fine-tuning strategies to preserve fine-grained visual cues. The work serves as a foundation for building unified, socially competent AI systems capable of reasoning about human interactions in real-world scenarios.