💻 computer science

From Codebooks to VLMs: Evaluating Automated Visual Discourse Analysis for Climate Change on Social Media

This paper evaluates the effectiveness of various vision-language models for automated visual discourse analysis of climate change on social media, demonstrating that while Gemini-3.1-flash-lite achieves the highest instance-level accuracy, even moderate-performing models can reliably recover population-level trends, thereby offering a scalable alternative to manual annotation for large-scale research.

Original authors: Katharina Prasse, Steffen Jung, Isaac Bravo, Stefanie Walter, Patrick Knab, Christian Bartelt, Margret Keuper

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Katharina Prasse, Steffen Jung, Isaac Bravo, Stefanie Walter, Patrick Knab, Christian Bartelt, Margret Keuper

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a massive, chaotic conversation happening in a giant, noisy town square. This town square is social media, and the topic everyone is shouting about is climate change.

For years, researchers have been trying to listen to this crowd. But they've mostly only been reading the words people are shouting. They've ignored the pictures people are holding up—photos of polar bears, memes about heatwaves, infographics about rising sea levels, and protest signs. These pictures tell a huge part of the story, but there are millions of them. It's impossible for a team of human experts to look at every single one; they would need a lifetime and a fortune to do it.

This paper is like a team of scientists asking: "Can we teach a super-smart robot to look at these millions of pictures for us, and will it actually understand what it's seeing?"

Here is the breakdown of their adventure, using some simple analogies:

1. The Problem: The "Needle in a Haystack"

The researchers had two piles of pictures:

The "Gold Standard" Pile (ClimateCT): A small, carefully curated box of 1,000 pictures that human experts had already labeled perfectly. Think of this as a teacher's answer key.
The "Wild West" Pile (ClimateTV): A massive, messy mountain of over 1.2 million pictures scraped from X (Twitter). This is the real world—full of weird angles, blurry photos, memes, and confusing images.

2. The Tools: The Robot Brains

They didn't just use one robot; they brought in a whole team of Vision-Language Models (VLMs).

Think of these models as super-intelligent interns. They have read billions of books and seen billions of photos.
Some are "Open-Source" interns (free to use, like Qwen or Gemma).
Some are "Big Tech" interns (expensive, like Google's Gemini or OpenAI's GPT).
They also tested older, simpler robots (like CLIP) that are good at spotting objects but bad at understanding context.

3. The Test: The "Codebook" Challenge

The researchers gave these robots a Codebook. In the old days, humans used codebooks to manually sort pictures into categories like:

Animals: (Is there a polar bear? A pet? A farm animal?)
Consequences: (Is this a flood? A wildfire? Or just a sad face?)
Action: (Is someone protesting? Installing solar panels?)
Setting: (Is this in a city, a forest, or outer space?)
Type: (Is this a photo, a cartoon, or a meme?)

The robots had to look at the pictures and sort them into these buckets without being specifically trained on climate change beforehand. They had to figure it out just by reading the instructions (prompts).

4. The Big Discoveries (The "Aha!" Moments)

🏆 The Winner: The "Flash" Robot

They found that Gemini-3.1-flash-lite was the clear champion. It was like the smartest intern in the room. It didn't need to be the biggest or most expensive model to win; it just needed to be the right tool for the job. It beat the other robots, even the huge 30-billion-parameter ones, which sometimes got confused by their own complexity.

📉 The "Good Enough" Secret

Here is the most surprising finding: The robots didn't have to be perfect to be useful.

If you ask a robot to label one specific picture, it might get it wrong 30% of the time.
BUT, if you ask it to label 1 million pictures, the mistakes cancel each other out. The robot gets the overall trend right.
Analogy: Imagine trying to guess the average height of everyone in a stadium. If you ask a slightly clumsy robot to measure every person, it might get some wrong. But if you ask it to measure the whole crowd, the average it calculates will be incredibly accurate. For social scientists, knowing the trend (e.g., "more people are posting about floods than polar bears") is often more important than knowing if one specific photo was labeled correctly.

🧠 The "Over-Thinker" Trap

The researchers tried telling the robots to "think step-by-step" (Chain of Thought), like a human solving a math problem.

Result: This actually made the robots worse at this specific task.
Analogy: It's like asking a chef, "Before you chop this onion, please write a 5-paragraph essay on the history of onions." By the time they finish writing, they've forgotten how to chop. For visual tasks, sometimes you just need to "see" and "act," not over-analyze.

📝 The "Prompt" Matters

How you ask the robot matters.

For simple things (like "Is this a photo or a drawing?"), a short instruction works best.
For complex things (like "Is this a drought or a flood?"), the robot needed a longer, detailed explanation with examples.
Lesson: You can't use the same instruction for every job. You have to tailor the request.

5. The Conclusion: Why This Matters

This paper is a bridge. It tells social scientists, "You don't need to hire 1,000 humans to analyze climate images anymore."

The Old Way: Manual, slow, expensive, and limited to small samples.
The New Way: Automated, fast, cheap, and capable of analyzing the entire conversation.

While the robots aren't perfect (they still get confused between similar things, like "sea level rise" vs. "floods"), they are reliable enough to spot the big picture. They can tell us if the world is shifting from talking about "distant polar bears" to "local floods," which is exactly what policymakers and scientists need to know to fight climate change effectively.

In short: We finally have a robot assistant that can read the visual news of the world, and it's ready to help us understand how we are talking about our planet's future.

1. Problem Statement

Climate change communication on social media is increasingly visual, yet computational research has historically focused on text analysis, leaving a "critical blind spot" regarding the millions of images shared daily. Manual content analysis of these images is prohibitively expensive and limited to small, non-representative samples. While recent advances in Vision-Language Models (VLMs) offer a scalable alternative, their reliability for social science discourse analysis remains unproven.

The core problem is determining whether VLMs can:

Reproduce human-quality annotations based on complex, theoretically grounded social science codebooks (which differ from standard computer vision benchmarks).
Reliably capture population-level discourse trends even if per-image accuracy is imperfect.
Handle the specific challenges of climate imagery, such as hierarchical category entanglement (e.g., distinguishing "polar bears" from "land mammals") and cultural context.

2. Methodology

Datasets

The authors utilized two distinct datasets derived from X (formerly Twitter) posts containing "climate change" keywords from 2019–2022:

ClimateCT (Climate Change Twitter): A curated set of 1,038 images representing the most popular tweets. These were manually annotated by domain experts (social scientists) to serve as a high-quality ground truth.
ClimateTV (Climate Twitter Visuals): A large-scale corpus of 1.22 million images (first image per tweet). A subset of 50,000 images was manually validated by crowd workers (QualityMatch) to assess automatic labeling quality.

Annotation Scheme

The study employs a five-dimensional annotation taxonomy, distinct from standard computer vision classes due to its hierarchical and overlapping nature:

Animals: (e.g., polar bear, land mammal, no animals).
Climate Action: (e.g., protest, politics, sustainable energy).
Consequences: (e.g., floods, biodiversity loss, rising temperature).
Setting: (e.g., residential, arctic, industrial).
Type: (e.g., photo, infographic, meme).

Experimental Setup

Model Benchmarking: The authors evaluated 6 promptable VLMs (including Gemini-3.1-flash-lite, GPT-5.4-mini, Qwen3 variants, and Moondream) and 15 zero-shot CLIP-like models (OpenCLIP library).
Prompt Engineering Ablation: They systematically tested prompt components:
- Category descriptions (labels only vs. extended explanations with examples).
- Super-category definitions.
- Chain-of-thought (CoT) reasoning triggers.
- Spatial localization instructions (foreground vs. background).
- Paraphrasing to test robustness.
Evaluation Metrics:
- Instance-level: Macro-accuracy, Precision, Recall, F1-score.
- Distributional: Jensen-Shannon Divergence (JSD), Total Variation (TV), and $\chi^2$ distance to measure how well predicted label distributions match ground truth at the corpus level.

3. Key Contributions

Comprehensive VLM Benchmarking for Social Science: The first systematic evaluation of VLMs against a multi-dimensional, theory-grounded social science codebook for climate imagery, moving beyond simple object detection to discourse analysis.
Distributional Evaluation Framework: The authors advocate for distributional metrics over instance-level accuracy. They demonstrate that VLMs can reliably recover population-level trends (e.g., the proportion of "polar bear" images) even when individual image classification is imperfect, provided errors are non-systematic.
Prompt Engineering Guidelines: The study provides actionable, dimension-specific advice:
- Chain-of-Thought (CoT): Explicitly triggering CoT reasoning reduces performance for visual classification tasks.
- Prompt Length: Simple categories (e.g., "Type") require short prompts; complex categories (e.g., "Consequences") benefit significantly from extended explanations and examples (+0.3 accuracy boost).
- Localization: Spatial instructions generally do not improve performance.
Taxonomy Design Insights: The research highlights that "other" and "none" categories create systematic confusion hotspots and that hierarchical entanglement (e.g., specific vs. general animals) leads to asymmetric confusion patterns.
Open Resources: Release of 50k validated labels, 1.2M image IDs, and code to ensure reproducibility.

4. Key Results

Model Performance

Best Performer: Gemini-3.1-flash-lite outperformed all other models across all five super-categories and both datasets.
Open vs. Closed Models: The gap between the best closed model (Gemini) and the best open-weight model (Qwen3-8B) was relatively small (0–8%), suggesting open models are viable for cost-sensitive research.
CLIP Limitations: Zero-shot CLIP-like models struggled significantly, particularly with negative classes (e.g., "no consequences"), achieving macro-accuracies up to 0.29 points lower than VLMs.
Dataset Difficulty: All models performed better on the curated ClimateCT dataset than the noisy, heterogeneous ClimateTV dataset.

Specific Findings by Research Question

RQ1 (Annotation Replacement): VLMs achieve moderate macro-accuracy (0.61–0.76). While instance-level errors exist, the Jensen-Shannon Divergence for categories like "Animals" and "Type" was negligible ( $\le 0.01$ ), confirming VLMs are viable for trend analysis.
RQ2 (Taxonomy): Confusion is highest between semantically proximate classes (e.g., "Floods" vs. "Sea level rise") and between specific classes and "Other" categories. Confusion is often asymmetric (e.g., "Pets" confused as "Farm animals," but not vice versa).
RQ3 (Model Selection): Model size does not guarantee better performance; the 30B parameter Qwen3 variant showed higher variance and lower mean accuracy than the 8B variant.
RQ4 (Prompting):
- CoT: Harmful for this task.
- Examples: Critical for complex categories (Consequences, Climate Action).
- Robustness: Performance remained stable across prompt paraphrases.

5. Significance and Implications

Methodological Shift: The paper argues that automated visual discourse analysis is now technically feasible for social scientists, but it requires a shift from "uploading a codebook and running" to a nuanced approach involving distributional evaluation and tailored prompt engineering.
Scalability: Researchers can now analyze millions of images to track how visual framing of climate change evolves over time and across regions, a task previously impossible due to manual annotation costs.
Cost-Efficiency: The findings suggest that smaller, open-weight models combined with optimized prompts can achieve near-state-of-the-art performance, making large-scale analysis accessible without expensive API costs.
Cautionary Note: While VLMs are powerful, they are not a "drop-in" replacement for human experts. They are best suited for aggregate trend analysis rather than precise instance-level labeling, and human validation of a representative subsample remains essential for high-stakes research.

In conclusion, this work bridges the gap between computer vision and social science, providing a rigorous framework for leveraging VLMs to understand the visual dimension of the climate crisis on social media.