Technical Summary: VIDEONORMS

Problem Statement

As Video Large Language Models (VideoLLMs) are deployed globally, there is a critical need to assess their ability to reason across differing cultural contexts. While general object recognition, temporal reasoning, and narrative understanding have been extensively benchmarked, the cultural competence of VideoLLMs—specifically their understanding of socio-cultural norms—remains underexplored. Socio-cultural norms are defined as societal rules delineating accepted and appropriate behavior within a culture. Evaluating this in a multimodal context is particularly challenging because it requires the correct interpretation of implicit meaning expressed through both non-verbal cues (e.g., gaze, gestures) and verbal features (e.g., sarcasm), while accounting for documented differences in values and norms across cultures. Existing benchmarks largely focus on textual social norms or culturally aligned image generation, lacking a systematic framework for video-based cross-cultural norm understanding.

Methodology

Dataset Construction: VIDEONORMS

The authors introduce VIDEONORMS, a dataset comprising over 3,000 annotations of socio-cultural norms derived from popular US and Chinese television shows. The construction process utilizes a human-AI collaboration framework consisting of three stages:

1. Video Selection: Eight popular TV shows were selected (four US, four Chinese), covering both formal (workplace) and informal social settings across drama and comedy genres. The selection relies on sociological research indicating that scripted television reflects real-world cultural expectations. From these, 15-second clips were sampled to isolate distinct social interactions.
2. Candidate Annotation (AI-First): A large VideoLLM (Gemini-2.0-Pro) was used to generate initial candidate annotations. Given a video clip, the model extracted candidate norm categories, adherence/violation labels, and corresponding verbal and non-verbal evidence. The model was prompted using a taxonomy of social norms (e.g., Greeting, Apology, Agreement) adapted for both English and Mandarin contexts.
3. Human Refinement: Each candidate annotation was reviewed by at least three trained monocultural annotators with significant lived experience in the target culture (US or China). Annotators verified timestamps, norm categories, adherence labels, and the correctness of evidence, editing errors as necessary.

To ensure high reliability for benchmarking, a consensus-based subset named VIDEONORMS-BENCHMARK was created, containing 542 instances where all three annotators agreed on the context, category, subject, specific norm, and adherence/violation label.

Evaluation Tasks and Metrics

The paper proposes two evaluation tasks for open-weight VideoLLMs:

1. Binary Classification: Predicting whether a specific cultural norm is adhered to or violated given a video clip, transcript, norm category, and specific norm.
2. Explainable Prediction: Predicting the adherence/violation label and generating supporting verbal and non-verbal evidence.

Performance is measured using Macro F1 scores for classification. For the explanation task, an LLM-as-a-Judge protocol (using GPT-5) evaluates the quality of generated evidence against reference annotations using a 5-point rubric that distinguishes between content perception and cultural reasoning. Additionally, Hierarchical Linear Modeling (HLM) is employed to analyze performance disparities across models, cultural contexts, and norm categories, controlling for variables such as show difficulty and item type.

Ablation Studies

The authors conduct ablation studies to isolate the contributions of input modality (video vs. transcript), model size, and the number of sampled frames to task performance.

Key Contributions

1. Human-AI Collaboration Framework: A novel two-stage framework for annotating socio-cultural norms in videos, leveraging a strong multimodal model for initial generation and monocultural experts for refinement. This approach addresses the inefficiency of manual annotation from scratch.
2. VIDEONORMS Dataset: A high-quality dataset of over 3,000 annotations and a filtered benchmark of 542 instances covering US and Chinese cultural contexts, including detailed verbal and non-verbal evidence.
3. Empirical Analysis of VideoLLMs: The first systematic comparison of open-weight VideoLLMs regarding cross-cultural norm understanding, revealing specific performance disparities and reasoning bottlenecks.

Results and Findings

The evaluation of seven open-weight VideoLLMs (including LLaVA-family, InternVL, Qwen-VL, and VideoChatR1) yielded the following findings:

• Cultural Performance Disparity: Models consistently perform worse in Chinese cultural contexts compared to US contexts. Hierarchical linear modeling indicates that the odds of correct prediction for norm adherence are 57% higher for US contexts than for Chinese contexts (OR=1.57, p=0.006). This suggests that norm adherence prediction requires culturally specific reasoning that current models lack.
• Non-Verbal Evidence Bottleneck: Models struggle significantly more with generating non-verbal evidence compared to verbal evidence. While language backbones can analyze transcripts effectively, multimodal capabilities for interpreting visual social cues (gaze, gestures) remain a bottleneck.
• Modality Necessity: Ablation studies confirm that the video modality is necessary for high performance. Transcript-only inputs result in significantly lower classification scores, particularly for Chinese contexts where meaning is often conveyed through non-verbal cues (high-context culture).
• Scaling Limitations: Increasing model size (from 4B to 14B parameters) does not reliably improve classification scores. While larger models may show slight gains in evidence quality, the performance gap in cultural reasoning persists, suggesting the bottleneck lies in the lack of culturally informed training data rather than raw model capacity.
• Annotation Disparity: Human verification revealed a high rate of edits to AI-generated annotations for Chinese content (42–53% change rate) compared to US content (18–26%), cautioning against fully automatic approaches for cultures under-represented in training data.

Significance and Limitations

The paper positions VIDEONORMS as a practical starting point for cross-cultural norm benchmarking in videos, a task previously unaddressed. The authors argue that their findings highlight the need for more diverse cultural data in video model training to achieve true cultural competence.

The authors explicitly acknowledge limitations:

• Scope: The study is limited to two countries (US and China) and does not claim to measure overall cultural competence or capture the full intra-cultural diversity of these nations.
• Data Source: The reliance on scripted TV shows as "cultural mirrors" rather than natural interactions is a necessary simplification, though prior work suggests familiarity with scripts does not substitute for the reasoning required in this task.
• Annotator Representation: Due to budget constraints, only three annotators per country were used, which may not fully represent heterogeneous populations.
• Bias Risks: The authors caution against using the benchmark for stereotyping (ecological fallacy) and note the potential for over-optimization to the metric, where high scores do not necessarily equate to genuine cultural competence.

Ultimately, the work serves as a foundational step toward understanding and improving the cultural competence of video models through rigorous, human-verified evaluation.