💬 NLP

Measuring Negative Campaigning across Languages with Large Language Models: A Study of 18 Million Tweets in 19 Countries

This study demonstrates that zero-shot large language models can effectively and scalably classify negative campaigning across 19 languages, revealing that governing parties are less negative while ideologically non-centrist parties, particularly those on the radical right, engage in significantly more confrontational rhetoric.

Original authors: Victor Hartman, Petter Törnberg

Published 2026-07-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Victor Hartman, Petter Törnberg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of politics as a giant, noisy stadium where everyone is trying to convince the crowd to cheer for their team. For decades, scientists studying this stadium have been obsessed with one specific behavior: "negative campaigning." Think of this not just as a politician saying, "I'm great," but as them pointing at a rival and shouting, "Look at that guy, he's terrible!" It's the art of the political roast. While we know this happens everywhere, from ancient Greek pottery shards to modern TV ads, figuring out why some teams do it more than others has been a massive headache for researchers. The problem? Counting these insults manually is like trying to count every grain of sand on a beach while wearing oven mitts. It takes forever, costs a fortune, and if you try to do it in twenty different languages, you'd need a translator for every single sentence. Because of this, most studies have been stuck looking at just one country (usually the US) or relying on experts guessing the "vibe" of a campaign, which isn't very precise. But recently, a new tool has arrived on the scene: Large Language Models (LLMs). You can think of these as super-smart, tireless robots that have read almost everything ever written on the internet. They can understand nuance, sarcasm, and insults in dozens of languages without needing to be taught the rules from scratch. This paper asks a big question: Can these AI robots be the new referees for counting political insults across the whole world, and what do they tell us about who actually likes to throw the mud?

The authors of this study decided to put these AI robots to the test on a massive scale. First, they had to prove the robots were good at their job. They fed the AI millions of tweets and asked it to spot negative attacks, then compared the robot's answers to those of human experts who had spent years manually reading the same posts. The result? The robots were shockingly good. In fact, in some cases, the AI was even more consistent and accurate than the human experts, and it could do the job in ten different languages without breaking a sweat. It was like swapping a team of tired, overworked grad students for a fleet of super-fast, multilingual drones that never get tired or confused.

Once they trusted the robots, the researchers unleashed them on a mountain of data: 18 million tweets posted by politicians in 19 European countries between 2017 and 2022. They wanted to see if the old rules about who gets nasty still held up when you looked at the whole continent. They tested a few big ideas: Do politicians in charge (the "government" team) act nicer because they have to look responsible? Do politicians on the far edges of the political spectrum (the radicals) act meaner because they have nothing to lose? And do "populist" politicians, who claim to fight against the "corrupt elite," naturally throw more insults?

The findings were clear and surprisingly consistent. The AI data showed that politicians currently sitting in the government are indeed much less negative. They seem to hold back the insults, perhaps because they know that if they start a fight, they might hurt their chances of staying in power or working with other parties later. It's like a team captain trying to keep the peace so the team doesn't fall apart. On the other hand, politicians who are not in charge, especially those on the far right and far left of the political spectrum, were the ones throwing the most mud. The study suggests that being far away from the political center makes a party more likely to be aggressive. Specifically, the "radical right" parties were the most negative of all, using confrontational language at a much higher rate than anyone else. While the study also looked at whether being a "populist" made a difference, it found that the real driver was actually how far to the extreme a party was; the nastiness seemed to come from being an outsider rather than just being a populist.

Interestingly, the study also looked at whether election time made people nastier. The results were a bit mixed; sometimes elections made things worse, but not always. The biggest takeaway wasn't about the timing, but about the players themselves. The paper suggests that the rise in negative campaigning we see on social media isn't just because Twitter (now X) makes people angry. Instead, it's a strategic choice by certain types of parties—specifically those on the fringes and those out of power—to use negativity as a weapon. By using these AI robots, the researchers were able to see patterns across 19 countries that would have been impossible to find with old methods, proving that in the game of politics, the outsiders are the ones most likely to play dirty, while the people in charge try to keep their hands clean.

Technical Summary: Measuring Negative Campaigning across Languages with Large Language Models

Problem Statement
Comparative research on negative campaigning—defined as criticism leveled by one candidate against another—has been constrained by methodological limitations. Traditional approaches rely on expert surveys, which suffer from subjectivity, lack of transparency, and poor replicability, or manual content analysis, which is labor-intensive and difficult to scale across multilingual contexts. While supervised machine learning offers a partial solution, it requires extensive, language-specific labeled training data, limiting its applicability for large-scale cross-national studies. Consequently, existing literature is disproportionately focused on single-country settings (particularly the U.S.) and lacks direct behavioral data linking party characteristics to campaign negativity in multiparty systems.

Methodology
The study employs a two-stage methodological approach:

Validation of Zero-Shot LLMs: The authors evaluate the capacity of zero-shot Large Language Models (LLMs) to classify negative campaigning across ten languages (English, German, Croatian, French, Hungarian, Italian, Dutch, Polish, Spanish, and Swedish). They benchmark two OpenAI models (gpt-4o and gpt-4o-mini) against two established, manually coded datasets:
- Petkevic and Nai (2022): 2018 U.S. Senate election tweets.
- Klinger et al. (2023): 2014 and 2019 European Parliament campaign Facebook posts.
  The models were prompted with natural language instructions (zero-shot) to identify "explicit attack or critique toward opponent party or candidate." Performance was measured using weighted F1 scores and inter-rater reliability (IRR) metrics (Krippendorff's $\alpha$ and Brennan-Prediger coefficient) to compare LLM outputs against human annotations.
Large-Scale Empirical Analysis: Leveraging the validated LLM approach, the authors analyzed 18,066,672 tweets posted by 5,439 parliamentarians across 19 European countries between 2017 and 2022.
- Unit of Analysis: Data were aggregated to the party level to reflect strategic party behavior rather than individual candidate idiosyncrasies.
- Dependent Variable: The percentage of negative tweets per party.
- Independent Variables: Derived from expert surveys (CHES, POPPA) and parliamentary records (ParlGov), including:
  - Cabinet status (governing vs. opposition).
  - Ideological positioning (Left-Right scale and squared distance from the center).
  - Populism scores.
  - Election proximity (within 3 months of an election).
  - Party family classification.
- Statistical Model: OLS regression with country fixed effects and clustered standard errors was used to test hypotheses regarding strategic incentives.

Key Contributions

Methodological: The paper provides the first systematic evaluation of zero-shot LLMs for cross-lingual classification of negative campaigning. It demonstrates that LLMs can achieve performance comparable to, and in some cases exceeding, human coders and conventional supervised models without requiring language-specific training data.
Substantive: It conducts the largest cross-national study of negative campaigning to date, moving beyond single-country analyses to examine party-level drivers in multiparty systems using direct behavioral data (tweets) rather than expert perceptions.

Results

LLM Performance: In the U.S. Senate dataset, the LLM outperformed both human coders (IRR $\alpha$ = 0.860 vs. 0.790) and the original supervised model (Weighted F1 increased from 0.810 to 0.930). In the multilingual European dataset, the LLM maintained consistent performance across all ten languages, achieving weighted F1 scores of at least 0.90 and high agreement with human coders, even under stricter definitions of negativity.
Governing Status: Hypothesis 1 is supported: parties in cabinet engage in significantly less negative campaigning (approximately 7–8 percentage points lower) than opposition parties. This suggests governing parties face stronger reputational and coalition-management constraints.
Ideological Positioning: Hypothesis 2 is supported: parties further from the ideological center engage in more negative campaigning. This effect is most pronounced for radical right parties, which exhibit the highest predicted levels of negativity (approx. 36–37%), followed by radical left parties. Centrist parties show the lowest levels.
Populism: While populist parties initially appeared more negative, this effect was largely accounted for by their non-centrist ideological positioning when controlling for ideological extremism.
Election Proximity: The effect of election proximity was weaker and less consistent than governing status or ideology, varying significantly by context.

Significance and Claims
The paper claims that its findings provide new comparative evidence on the party-level foundations of campaign negativity, challenging the notion that rising negativity is solely a byproduct of social media logics. Instead, it reflects deeper dynamics of party competition, coalition incentives, and the strategic behavior of specific political movements (particularly the radical right).

Methodologically, the study argues that LLMs can transform the scale, consistency, and replicability of political discourse research. By enabling zero-shot classification across languages, LLMs lower the barriers to large-scale comparative analysis, allowing researchers to move beyond the "depth vs. breadth" trade-off that has historically limited the field. The authors emphasize that while LLMs improve measurement scalability, they require rigorous validation against human benchmarks and careful prompt design to ensure validity, positioning them as a complement to, rather than a replacement for, established methodological standards.

Technical Summary: Measuring Negative Campaigning across Languages with Large Language Models

More like this