GAViD: A Large-Scale Multimodal Dataset for Context-Aware Group Affect Recognition from Videos

1. Problem Statement

Group Affect Recognition (GAR) aims to analyze collective emotional states arising from human-human interactions. While individual emotion recognition is well-studied, GAR in real-world ("in-the-wild") environments faces significant challenges:

• Data Scarcity: There is a lack of large-scale, annotated datasets that include diverse group-level labels (valence and discrete emotions) alongside rich contextual metadata.
• Complexity: Group affect is shaped by dynamic interactions, occlusions, varying lighting, and the interplay between individual emotions and scene context.
• Contextual Gap: Existing methods often rely heavily on visual features and fail to capture the full context (e.g., event type, environment, interaction dynamics) necessary for robust analysis. Most existing datasets lack a comprehensive combination of video, audio, and contextual annotations.

2. Key Contributions

The authors introduce two primary contributions to bridge these gaps:

A. The GAViD Dataset

GAViD (Group Affect from ViDeos) is a large-scale, in-the-wild multimodal dataset designed specifically for GAR.

• Scale: Contains 5,091 high-quality video clips (average duration 5 seconds) derived from 321 source videos.
• Multimodal Inputs: Each sample includes synchronized Video, Audio, and Contextual Metadata.
• Annotations:
  • Affect Labels: Ternary valence (Positive, Neutral, Negative) and five discrete emotions (Happy, Sad, Fear, Anger, Neutral).
  • Intensity & Interaction: Emotion intensity (High, Medium, Low) and interaction types (Cooperative, Hostile, Neutral).
  • Action Cues: Human-annotated cues such as smiling, clapping, shouting, dancing, etc.
  • Contextual Metadata: Generated by VideoGPT (a Multimodal Large Language Model) to describe scene types and event details, then verified by human annotators.
• Quality Control: Data was curated from Creative Commons (CC BY) licensed YouTube videos. A rigorous annotation process involved 108 annotators using a majority-vote system to ensure high inter-annotator agreement (Cohen's $\kappa$ = 0.72 for valence, 0.65 for discrete emotions).

B. The CAGNet Model

The authors propose the Context-Aware Group Affect Recognition Network (CAGNet), a multimodal architecture designed to fuse visual, audio, and contextual features.

• Architecture:
  1. Modality-Specific Encoding: Uses frozen backbones: DINOv2 (Vision), Wav2Vec 2.0 (Audio), and XLM-RoBERTa (Text/Context).
  2. Cross-Modal Alignment: Employs masked multi-head cross-attention blocks for every bimodal pair (Vision-Audio, Vision-Context, Audio-Context). This allows each modality to query others for missing cues.
  3. Gated Fusion: A Squeeze-and-Excitation gate dynamically re-scales the concatenated embeddings. This allows the model to up-weight specific modalities based on context (e.g., prioritizing audio if faces are occluded).
  4. Classification: A lightweight MLP predicts valence and discrete emotion labels.
• Robustness: The model handles missing modalities gracefully via modality dropout during training and zero-vector injection during inference.

3. Methodology Details

• Data Compilation: Videos were segmented into 5-second clips (matching the typical duration of expressive behaviors). Clips were filtered to ensure clear group structures and face detectability.
• Training Strategy: The model was trained using AdamW optimizer with a learning rate of $1 \times 10^{-4}$. Only the cross-attention blocks, gating mechanism, and MLP head were trained, while the feature backbones remained frozen.
• Baseline Comparisons: The authors compared CAGNet against:
  • Baseline 1 (Merged Fusion): Simple concatenation of cross-attention embeddings without gating.
  • Baseline 2 (Hierarchical Fusion): Sequential fusion of modalities.
  • State-of-the-Art (SOTA): Video-GPT and LLaVA-NeXT (generic video-language models).

4. Experimental Results

Experiments were conducted on the GAViD dataset split into Train (3,503), Validation (542), and Test (1,046) sets.

• Valence Classification Performance:
  • CAGNet (Tri-modal V+A+C): Achieved 63.20% accuracy and 0.614 F1-score on the test set.
  • Comparison: CAGNet significantly outperformed both baselines and SOTA video-LLMs (Video-GPT: 51.92%, LLaVA-NeXT: 54.82%).
  • Modality Impact: The tri-modal setup (V+A+C) performed best, confirming the value of context. Visual+Audio was the strongest bimodal pair, but adding context consistently improved performance, especially in ambiguous scenarios.
• Discrete Emotion Recognition:
  • CAGNet achieved 61.33% accuracy and 0.458 F1-score, outperforming baselines (Baseline 1: 58.33%, Baseline 2: 60.01%).
• Qualitative Analysis:
  • The inclusion of context (V+A+C) proved critical in distinguishing subtle interactions (e.g., "speaking in a harsh tone" vs. "chatting warmly") where visual and audio cues alone were insufficient.

5. Significance and Impact

• Filling a Critical Gap: GAViD is the first dataset to provide a comprehensive combination of valence, discrete emotions, and rich contextual metadata for group affect, addressing the limitations of existing image-based or context-poor video datasets.
• Advancing Context-Aware AI: The results demonstrate that integrating contextual metadata (generated by MLLMs and verified by humans) significantly boosts GAR performance, validating the hypothesis that group affect cannot be understood in isolation from the scene.
• Ethical Framework: The paper establishes a strong ethical protocol, utilizing only CC-licensed public domain data, removing PII, and restricting usage to research/education to prevent surveillance or profiling.
• Future Directions: The dataset and CAGNet serve as a baseline for future research in computational social systems, human-computer interaction, and real-time group emotion monitoring. The authors plan to expand the dataset to include frame-level and individual-level annotations.

Availability: The dataset and code are publicly available at github.com/deepakkumar-iitr/GAViD and zenodo.org/records/15448846.