ClimateVID -- Social Media Videos Analysis and Challenges Involved

1. Problem Statement

The proliferation of short-form videos on social media has fundamentally altered public discourse on climate change. While visual theme analysis is crucial for understanding these narratives, existing methods face significant challenges:

• Data Complexity: Social media videos contain temporal redundancy, changing viewpoints, and multiple distinct scenes, making them harder to analyze than static images.
• Model Limitations: Current Vision-Language Models (VLMs) often struggle with zero-shot classification on social media content, particularly regarding specific climate categories, memes, and out-of-distribution content.
• Clustering Granularity: Traditional clustering methods often produce coarse results that fail to capture fine-grained visual themes or distinct sub-topics within broader categories.
• Lack of Guidelines: There is a scarcity of practical guidance for researchers on how to preprocess video data (frame selection, embedding combination) and select appropriate models for large-scale, unsupervised theme discovery.

2. Methodology

The authors propose a dual-strategy framework for analyzing a dataset of ~78,000 unique Twitter videos related to climate change (collected 2019–2022).

A. Zero-Shot Classification (Supervised Approach)

The study evaluates the capability of three open-source Video Large Language Models (VLMs) against a frame-wise baseline:

• Models Evaluated: VideoChatGPT, PandaGPT, VideoLLaVA, and a baseline using CLIP (frame-wise).
• Prompting Strategy: A hierarchical prompting approach was used. First, binary "Yes/No" prompts screened for broad groups (Animals, Consequences, Climate Action). If positive, detailed prompts requested specific category numbers.
• Categories: 59 distinct categories across 5 groups: Animals, Consequences, Climate Action, Setting, and Type.
• Evaluation: Models were tested on semantic coherence and faithfulness (adherence to output format).

B. Unsupervised Clustering (Discovery Approach)

To uncover latent themes without predefined labels, the authors adapted the Minimum Cost Multicut (MP) problem for video data.

• Pipeline:
  1. Frame Selection: Compared Static Selection (uniform sampling at 1 fps) vs. Dynamic Selection (clustering-based keyframe extraction).
  2. Embedding Generation: Used two foundation models: DINOv2 (ViT architecture) and ConvNeXt V2 (masked autoencoder).
  3. Frame Combination: Aggregated frame embeddings into a single video-level vector using methods like Simple Average, Weighted Diversity, and Temporal Coherence.
  4. Clustering: Constructed a similarity graph and partitioned it using the MP algorithm to identify visually similar content groups.
• Metrics: Evaluated using standard clustering metrics (Silhouette, Davies-Bouldin, Calinski-Harabasz) and pragmatic metrics (Gini coefficient for balance, Singleton ratio for over-fragmentation, and Top-10 coverage).

3. Key Contributions

1. Comprehensive VLM Evaluation: A critical assessment of VideoChatGPT, PandaGPT, and VideoLLaVA for social media climate analysis, revealing significant limitations in zero-shot performance compared to frame-wise CLIP.
2. Methodological Extension: Successfully extended Minimum Cost Multicut clustering to video data, providing ablation studies on frame selection and embedding combination strategies.
3. Practical Guidelines: Established best practices for social media video analysis, recommending specific model choices (ConvNeXt V2) and processing pipelines (static sampling + simple averaging) for thematic discovery.
4. Open Resources: Released a curated dataset of tweet IDs and code for reproducibility.

4. Key Results

Zero-Shot Classification Findings

• VLM Failures: VideoChatGPT and PandaGPT produced highly unreliable results, characterized by:
  • Positional Bias: Strong tendency to select the first category listed in the prompt.
  • Format Errors: Inconsistent adherence to output formats (e.g., returning text instead of category numbers).
  • Over-classification: Tendency to assign multiple, often contradictory, labels to a single video (e.g., labeling a video as containing "All" animal types).
• CLIP Success: Frame-wise CLIP emerged as the most reliable model. While it lacks temporal reasoning, it provided consistent, interpretable probability distributions across frames.
• Recommendation: For social media video classification with predefined categories, frame-wise analysis using CLIP is currently superior to end-to-end VLMs.

Clustering Findings

• Embedding Space Differences:
  • DINOv2: Produced fewer, broader clusters (50 total). It grouped videos based on high-level visual cues (color palette, composition, broad themes like "Arctic Ice"). Top 5 clusters contained 94.8% of all videos.
  • ConvNeXt V2: Produced more granular clusters (71 total). It distinguished fine-grained details such as specific objects, production styles, and contexts (e.g., separating "domestic pets" from "wildlife"). It showed a more balanced distribution, with only 13.1% of videos in the top 5 clusters.
• Frame Selection Impact: The choice between static and dynamic frame selection had a benign effect on clustering quality. Static sampling (uniform) was sufficient and computationally efficient.
• Optimal Configuration: The authors recommend ConvNeXt V2 embeddings with static frame selection and simple averaging for video clustering. This setup yielded the most meaningful and interpretable visual themes.

Content Insights

• Dominant Themes: "Climate Action" (politics, protests, energy) was the most frequent content (65%), followed by "Consequences" (56%) and "Animals" (53%).
• Visual Trends: "Memes" were a dominant video type. "Outer Space" (Earth from space) was a surprisingly prevalent setting, often used for abstract climate discussions.
• Temporal Shifts: Over time, there was a shift from abstract visualizations toward more realistic content (residential/commercial settings).

5. Significance and Conclusion

This paper highlights a critical gap in current AI capabilities: while VLMs excel at general dialogue, they struggle with the specific, noisy, and meme-heavy nature of social media climate discourse. The study demonstrates that simpler, frame-based approaches (CLIP) combined with robust unsupervised clustering (ConvNeXt V2 + Multicut) are currently more effective for large-scale thematic analysis than complex generative VLMs.

The work provides a vital roadmap for researchers and practitioners, offering validated methodologies to extract actionable insights from massive volumes of social media video data, thereby aiding in the monitoring of public discourse and the development of targeted climate communication strategies.