💻 computer science

Seeing Candidates at Scale: Multimodal LLMs for Visual Political Communication on Instagram

This paper evaluates the effectiveness of multimodal large language models, specifically GPT-4o, against traditional computer vision models for analyzing visual political communication on Instagram during the 2021 German federal election, demonstrating that GPT-4o significantly outperforms existing methods in identifying politicians and counting individuals.

Original authors: Michael Achmann-Denkler, Mario Haim, Christian Wolff

Published 2026-04-22

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Michael Achmann-Denkler, Mario Haim, Christian Wolff

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: Who is the real star of the show in a political campaign?

In 2021, Germany held a big election. During this time, politicians flooded Instagram with thousands of photos and short videos (called "Stories") to win votes. But there were too many pictures for any human to look at one by one. It would take years!

So, three researchers from German universities decided to build a team of digital detectives (AI models) to do the job for them. Their goal was to answer two big questions:

Can computers get better at spotting politicians than humans?
How do politicians use Instagram to make themselves look like the main character?

Here is the story of their investigation, explained simply.

1. The Contest: Old School vs. The New Super-Brain

The researchers set up a race between three different types of AI "detectives" to see who could do the best job on Instagram photos.

The Old School Detectives (FaceNet & RetinaFace): These are like specialized security guards. They are trained only to look at faces. They are good at finding a face in a crowd, but they are a bit rigid. If a face is blurry, turned sideways, or covered by a mask, they might get confused.
The Cloud Detective (Google Cloud Vision): This is a powerful, pre-made tool from a tech giant. It's like hiring a professional security firm. It's good, but it's a "black box"—you don't always know exactly how it makes its decisions.
The New Super-Brain (GPT-4o): This is a Multimodal Large Language Model. Think of this as a super-smart intern who has read every book in the library and seen every movie ever made. Unlike the security guards who just look for "face shapes," this intern can understand the picture. It can tell the difference between a real person standing in a crowd and a picture of a person on a poster. It can also understand context, like "Is this person wearing a mask?" or "Is this a blurry selfie?"

The Result:
The Super-Brain (GPT-4o) won the race by a landslide.

It was much better at recognizing the politicians, even when they were wearing masks or standing far away.
It was also much better at counting how many people were in a photo. The old-school detectors kept getting tricked by faces in the background (like on a billboard) and thought there were 10 people when there were only 2. The Super-Brain knew to only count the main people.

The Analogy:
Imagine you are at a party.

The Old School Detector is like a bouncer who only checks IDs. If your face is covered by a hat, he says, "I can't see you, you're not here."
The Super-Brain is like a witty host who knows everyone. Even if you are wearing a hat or standing in the back, the host says, "Ah, that's the Mayor! And I see three other guests with him."

2. The Mystery: Who is the "Main Character"?

Once they had the best AI detective (GPT-4o), they used it to analyze the 2021 German election. They were looking for a concept called "Concentrated Visibility."

Think of a movie. Sometimes the camera focuses only on the hero (the Front-Runner). Sometimes it shows the whole cast (the Party).

Concentrated Visibility means the camera is zoomed in tight on just one person, making them look like the most important person in the world.

What did they find?

The Candidates vs. The Parties: The individual politicians (the candidates) loved to be the "Main Character." On their personal Instagram accounts, they posted lots of photos of themselves, often alone or with just a few people. They wanted you to fall in love with them.
The Parties were more shy: The official party accounts (like the CDU or SPD pages) were more balanced. They showed the candidates, but they also showed groups of people, rallies, and crowds. They didn't focus as intensely on just one face.
Stories vs. Permanent Posts:
- Posts (Permanent): These are like polished magazine covers. They are high-quality, professional photos. Both candidates and parties used these to look serious and official.
- Stories (24-hour vanishing content): These are like "behind-the-scenes" vlogs. They are messy, spontaneous, and casual. The researchers found that candidates used Stories to show their "real" selves, while parties didn't use Stories as much to highlight their leaders.

The Takeaway:
Politicians are playing a two-part game. They use their personal accounts to say, "Look at me, I am the star!" and they use their party accounts to say, "We are a team." But on Instagram Stories, the candidates really leaned into being the "Main Character," while the parties were a bit more reserved.

3. Why Does This Matter?

This study is a big deal for two reasons:

The Tool is Ready: We used to think analyzing millions of photos was impossible without a huge army of humans. Now, we have a "Super-Brain" AI that can do it quickly, cheaply, and accurately. This means researchers can study politics on a massive scale without spending years on manual work.
The Strategy is Clear: We now know exactly how politicians are trying to win our hearts. They are using "Concentrated Visibility" to make us feel like we know them personally, even though they are just a face on a screen.

The Catch (Limitations)

The researchers admitted a few things:

The "Black Box" Problem: The Super-Brain (GPT-4o) is owned by a private company. We can't see its code. It's like hiring a genius detective who won't tell you how they solved the case.
The Mask Problem: Because this election happened during the pandemic, many politicians wore masks. The old AI models struggled with this, but the Super-Brain handled it better. However, if a candidate wore a mask and stood far away, even the Super-Brain sometimes got confused.
One Female Candidate: There was only one female candidate in their study (Annalena Baerbock). The old AI models were worse at recognizing her face, which suggests AI might have a bias against women. The Super-Brain was better, but we need more data to be sure.

In a Nutshell

This paper is about teaching computers to watch political TV shows (Instagram) and tell us who the stars are. They found that a new, smart AI (GPT-4o) is much better at this than older tools. They also discovered that politicians are very strategic: they use their personal accounts to shine the spotlight solely on themselves, trying to make us feel like they are our friends, while their party accounts play it a bit safer.

It's a mix of high-tech detective work and political psychology, showing us that in the digital age, being "seen" is just as important as being heard.

1. Problem Statement

The study addresses two primary challenges in the field of Computational Social Science (CSS) and Visual Political Communication (VPC):

Methodological Scalability: Traditional manual coding of visual political content is time-consuming and limits the scale of analysis. While computer vision (CV) models exist, their application to political communication often struggles with the "sensory gap"—mapping raw algorithmic outputs to complex theoretical constructs like "concentrated visibility."
The "Story" Gap: Most existing research focuses on permanent Instagram posts, neglecting ephemeral content (Stories), which offers unique affordances for spontaneity and immediacy. There is a lack of understanding regarding how political actors utilize Stories versus Posts for visual self-presentation.
Model Performance: It remains unclear whether specialized, open-source deep learning models (e.g., face detection/recognition) or emerging Multimodal Large Language Models (MLLMs) are more effective for identifying specific political figures and counting individuals in complex social media imagery.

2. Methodology

The authors conducted a computational case study focusing on the 2021 German federal election campaign.

Data Collection

Corpus: 1,424 Instagram Stories and 547 Posts collected between September 12 and September 25, 2021.
Subjects: Content from five major parties (CDU, CSU, SPD, B90/Die Grünen, FDP) and their respective "front-runner" candidates (e.g., Olaf Scholz, Annalena Baerbock).
Preprocessing: Video content was converted to static images (first frame) to maintain a manageable image-level unit of analysis.

Operationalization: Concentrated Visibility

The study measured "concentrated visibility" (a sub-dimension of political personalization) through two classification tasks:

Face Verification: Determining if a specific front-runner candidate is present in an image.
Person Counting: Counting the number of people in the focus of the image (categorized as 0, 1, 2, or 3+).

Computational Models Evaluated

The authors compared three distinct approaches:

Open-Source Deep Learning Pipeline: A combination of RetinaFace (for face detection) and FaceNet512 (for face recognition/verification). This represents the traditional CV approach.
Commercial API: Google Cloud Vision API (GCV), utilizing object detection and face recognition capabilities.
Multimodal LLM: GPT-4o (specifically gpt-4o-2024-05-13). The authors used prompt engineering to instruct the model to verify candidate presence and count people, asking for JSON output with reasoning comments.

Ground Truth & Evaluation

Annotation: Human annotators (students and staff) created a gold-standard dataset.
- Face Verification: 13 annotators labeled 1,171 stories and 735 posts.
- Person Counting: 5 annotators labeled a subset of images containing front-runners.
Metrics: Models were evaluated against human annotations using Precision, Recall, Macro F1-score, and Weighted F1-score.
Reliability: Krippendorff's Alpha ( $\alpha$ ) was calculated to measure inter-rater agreement, treating the AI models as "additional annotators" to see if they increased or decreased consistency.

3. Key Contributions

Benchmarking MLLMs in CSS: The paper provides empirical evidence that Multimodal LLMs (specifically GPT-4o) outperform specialized, state-of-the-art computer vision models in political visual analysis tasks.
Prompt Engineering for VPC: It demonstrates that carefully crafted prompts can guide MLLMs to handle theoretical nuances (e.g., distinguishing between a "face on a poster" vs. a "real person") better than raw CV model outputs.
Stories vs. Posts Analysis: It offers the first large-scale computational comparison of concentrated visibility strategies between permanent posts and ephemeral Stories in a German political context.
Open Resources: The authors released prompts, evaluation notebooks, and model outputs (excluding the copyrighted image corpus) to facilitate reproducibility.

4. Key Results

A. Model Performance (RQ1)

Face Recognition:
- GPT-4o achieved a Macro F1-score of 0.89 for Stories and 0.91 for Posts.
- FaceNet512 scored significantly lower (0.74 for Stories, 0.87 for Posts).
- Key Insight: GPT-4o was particularly superior in identifying female candidates (Annalena Baerbock), where FaceNet512 struggled (F1=0.66 vs. GPT-4o F1=0.91), suggesting potential gender bias in traditional CV models.
Person Counting:
- GPT-4o achieved a Macro F1-score of 0.86 (Stories) and 0.93 (Posts).
- RetinaFace and Google Cloud Vision performed poorly (F1 scores ranging from 0.44 to 0.58).
- Key Insight: Traditional CV models tended to overcount by detecting faces in backgrounds, on posters, or in screenshots. GPT-4o successfully interpreted the "focus" of the image, distinguishing between main subjects and background elements.
Agreement: GPT-4o showed the highest agreement with human annotators (Krippendorff's $\alpha$ = 0.85 for Stories), outperforming both specialized CV models and commercial APIs.

B. Concentrated Visibility Patterns (RQ2)

Account Type Differences:
- Candidate Accounts: Heavily utilized concentrated visibility. In Stories, candidates appeared alone (C1) in 33.5% of images, compared to only 12.0% on Party accounts.
- Party Accounts: Focused more on group shots or no candidates (C0), with only 14.7% of Stories featuring a candidate alone.
- Statistical Significance: Significant differences were found between party and candidate accounts ( $\chi^2$ tests, $p < .001$ ), indicating distinct visual strategies.
Format Differences (Stories vs. Posts):
- Stories: Showed a higher tendency for "Candidate Only" (C1) content on candidate accounts, suggesting a strategy of personal branding and intimacy.
- Posts: Showed a more balanced mix of "Candidate + Others" (C+), often used for formal campaign events.
- Conclusion: Parties and candidates use Stories for personalization and "backstage" access, while Posts are used for more formal, collective messaging.

5. Significance and Implications

Paradigm Shift in Analysis: The study suggests a shift from relying on specialized, hardware-intensive CV pipelines to using accessible, prompt-driven MLLMs for social science research. GPT-4o offers a "zero-shot" capability that requires less technical setup while delivering higher accuracy.
Methodological Trade-offs: The authors highlight a trade-off between accessibility/performance (GPT-4o) and transparency/privacy (Open-source models). While GPT-4o is superior in performance, its closed-source nature raises concerns about data privacy and replicability.
Political Strategy Insights: The findings confirm that political personalization is not uniform; it is strategically adapted to platform affordances. Candidates leverage Stories to humanize themselves (high concentrated visibility), while Party accounts maintain a more collective image.
Bias Detection: The study underscores the importance of human-in-the-loop validation, as traditional models exhibited gender bias (underperforming on female candidates) and context blindness (counting faces on posters as real people).

In summary, the paper validates GPT-4o as a superior tool for scaling visual political communication analysis, offering a robust, accessible method for researchers to study "concentrated visibility" and personalization strategies across different social media formats.