Benchmarking Political Persuasion Risks Across Frontier Large Language Models

Through large-scale survey experiments involving over 19,000 participants, this study demonstrates that frontier large language models generally outperform standard political advertisements in persuasiveness, with significant performance variations across models and a model-dependent impact of information-based prompting strategies.

The Gist

Imagine a massive, high-stakes debate happening in the digital town square. For years, we've been worried that Artificial Intelligence (AI) might be able to talk us into changing our minds about politics. But until now, we weren't sure if these digital debaters were actually better at it than a human politician holding a sign or running a TV ad.

This paper, written by researchers at Yale, is like a super-charged "Taste Test" for political persuasion. They invited nearly 20,000 real people to have conversations with the smartest AI models available in late 2025 (like Claude, GPT-5, Gemini, and Grok) to see if the bots could change their minds on two hot-button issues: raising the minimum wage and college tuition for immigrants.

Here is the breakdown of what they found, using some simple analogies:

1. The AI "Heavyweights" Beat the Humans

Think of traditional political ads (TV spots, flyers) as a standard, reliable hammer. They get the job done, but they aren't magic.

The researchers found that the new "Frontier" AIs are like laser-guided, shape-shifting hammers. When people chatted with these AIs, the bots were significantly more persuasive than the human campaign ads.

• The Result: The AIs didn't just match human persuasion; they crushed it. If you wanted to change a voter's mind, a conversation with a top-tier AI was more effective than watching a 30-second TV commercial.

2. Not All Bots Are Created Equal

Just like cars, some AI models are F1 race cars, and others are slow sedans. The researchers put them all on the track to see who was the fastest persuader.

• The Champion (Claude): The Claude models (specifically Claude Sonnet 4 and 4.5) were the Olympic gold medalists. They were the most convincing, able to talk people into changing their views more than any other bot.
• The Middle Pack (GPT-5 & Gemini): These were the reliable sports cars. They were very good, performing similarly to each other, but they couldn't quite beat Claude.
• The Underperformer (Grok): The Grok model was the scooter in a race of sports cars. It still moved faster than a human ad, but it was the least persuasive of the bunch.

3. The "Fact-Check" Trap

The researchers tried a specific trick: they told some bots, "Hey, use lots of facts, data, and statistics to win this argument." They thought this would make the bots super persuasive, like a lawyer with a perfect brief.

Surprise! It didn't work that way.

• For Claude and Grok, using facts helped them a little bit.
• But for GPT-5, giving it a "Fact-Check" instruction actually made it worse at persuading people. It's like telling a great storyteller to stop telling stories and just read a spreadsheet; the audience got bored and tuned out.
• The Lesson: There is no "one-size-fits-all" instruction. What works for one AI model might hurt another.

4. How Did They Win? (The Secret Sauce)

The researchers didn't just look at who won; they looked at how they won. They used a special "AI detective" to analyze the conversations and find the winning strategies.

They found that the most persuasive bots didn't just dump data on you. Instead, they used emotional and action-oriented tactics:

• The "Call to Action" (The Winning Move): The most effective strategy was telling people exactly what to do next. "Call your representative," "Sign this petition," "Go to this meeting." It's like a coach not just saying "You can win," but handing you the playbook and saying, "Run this play right now."
• The "Moral Appeal": Talking about fairness, justice, and values worked better than just talking about numbers.
• The "Fact" Trap: Interestingly, the strategy of citing specific studies and numbers (which the "Information Prompt" tried to force) didn't actually correlate with winning. People didn't change their minds because of the data; they changed because of the story and the call to action.

5. The Big Warning

Why does this matter?

Imagine a world where bad actors (or foreign governments) can control these "Gold Medal" AI bots. They could deploy millions of these bots to have personalized conversations with voters, 24/7, using the most effective psychological tricks to sway an election.

The paper concludes that we are entering a dangerous new era. These AI models are so good at persuasion that they pose a real threat to democracy. They aren't just chatbots; they are potentially the most powerful political campaign tools ever invented, and right now, they are outperforming human politicians.

In a nutshell: The new AI models are incredibly persuasive "debate champions" that beat human ads. Some are better than others, and the secret to their success isn't just having facts—it's knowing how to connect emotionally and tell people exactly what to do. We need to be very careful about who controls these powerful tools.

Technical Summary

Here is a detailed technical summary of the paper "Benchmarking Political Persuasion Risks Across Frontier Large Language Models."

1. Problem Statement

The paper addresses the growing concern that Large Language Models (LLMs) may possess the capacity to sway political views more effectively than traditional human-led campaign methods. While prior research (e.g., Chen et al., 2025; Hackenburg et al., 2025b) suggested that earlier-generation LLMs were not significantly more persuasive than standard political advertisements, the rapid emergence of "frontier" models in late 2025 (e.g., Claude Sonnet 4.5, GPT-5, Gemini 3, Grok 4) has fundamentally altered the landscape. These models exhibit superhuman capabilities in reasoning, coding, and cross-domain tasks. The critical gap this paper fills is determining whether these enhanced general capabilities translate into a superior ability to manipulate political attitudes, and how different models compare to one another and to human campaign ads.

2. Methodology

The authors conducted two large-scale randomized controlled survey experiments (Total $N = 19,145$) using the Prolific platform.

Study 1 (August 2025)

• Models Evaluated: Claude Sonnet 4, Gemini 2.5 Flash, GPT-4.1, Grok 4.
• Issues: Minimum Wage (anti-increase stance) and Immigration (pro-in-state tuition stance).
• Conditions:
  • Placebo: Unrelated video or blank page.
  • Human Persuasion: 30–60 second video of a human advocate (only for Immigration; Minimum Wage used a proxy from a previous study adjusted via a Hajek-style estimator).
  • AI Chatbot: Interactive text-based conversations.
• Prompting Strategies:
  • Plain Prompt: General instruction to persuade.
  • Information Prompt: Explicit instruction to use facts, evidence, and data (based on Hackenburg et al., 2025b).
• Outcome: Policy attitude measured on a 5-point Likert scale, re-coded as binary (support vs. oppose/indifferent).

Study 2 (November 2025)

• Models Evaluated: Claude Sonnet 4.5, Gemini 3, GPT-5, Grok 4.
• Design: Similar to Study 1 but omitted the human condition (as Study 1 established AI superiority). Crucially, it introduced stance variation: chatbots argued against the participant's pre-existing view (e.g., persuading a supporter to oppose, or an opposer to support).
• Analysis: Used random-effects meta-analysis to pool effects across issues and stances.

Persuasion Strategy Analysis

To understand why certain models were more effective, the authors developed a novel LLM-assisted conversation analysis pipeline:

1. Discovery Phase: Used GPT-5 mini to qualitatively analyze small batches of conversations between high- and low-performing model/prompt combinations to hypothesize emergent strategies.
2. Quantification Phase: Used GPT-5.2 to rate 4,790 conversations on a 1–5 scale across 10 identified dimensions (e.g., "Call-to-Action Messaging," "Explicit Sources," "Argumentative Framing").
3. Validation: Human researchers verified a subset of ratings, confirming high correlation (Pearson's $r > 0.5$ for most dimensions).
4. Regression: Regressed attitude changes against strategy ratings to identify which tactics correlated with higher persuasiveness.

3. Key Contributions

1. Benchmarking Frontier Models: First comprehensive comparison of state-of-the-art LLMs (2025 releases) against each other and against human political advertising.
2. Model Heterogeneity: Demonstrates that persuasiveness is not uniform; it varies significantly by model architecture and developer.
3. Prompt Sensitivity: Challenges the consensus that "information-based" prompts universally boost persuasion, showing instead that their efficacy is highly model-dependent.
4. Strategy Identification: Introduces a data-driven, strategy-agnostic framework to identify specific conversational tactics (like "Call-to-Action") that drive persuasion, moving beyond pre-defined taxonomies.

4. Key Results

A. AI vs. Human Persuasion

• Superiority of AI: Frontier LLMs significantly outperformed standard human campaign advertisements in shifting political attitudes.
  • Immigration: AI average effect = 0.203 scale points vs. Human = 0.135 ($p < 0.001$).
  • Minimum Wage: AI average effect = 0.136 vs. Generalized Human estimate = 0.059.
• Implication: AI represents a more potent tool for political influence than traditional media ads.

B. Model Performance Ranking

A consistent hierarchy emerged across both studies and issues:

1. Claude Models (Sonnet 4/4.5): Highest persuasiveness.
2. GPT-5 / Gemini 3 / GPT-4.1: Middle tier with similar, moderate effects.
3. Grok 4: Consistently the least persuasive, though still outperforming human ads.

C. Prompt Effectiveness (Information vs. Plain)

Contrary to previous findings (Hackenburg et al., 2025b), the "Information Prompt" did not universally improve persuasion. Its impact was model-dependent:

• Claude & Grok: Information prompts increased persuasiveness.
• GPT: Information prompts substantially reduced persuasiveness (e.g., GPT-5 dropped from ~0.204 to ~0.110 effect size).
• Gemini: Mixed or negligible effect.

D. Persuasion Strategies

The conversation analysis revealed specific drivers of success:

• Call-to-Action Messaging: The strongest positive correlate with attitude change ($\beta \approx 0.38$). Models that urged concrete actions (e.g., "Call your representative," "Sign a petition") were most effective.
• Explicit Sources/Numbers: Surprisingly, the use of specific data and citations (the core of the "Information Prompt") showed no significant positive correlation with persuasion and was sometimes negatively associated.
• Argumentative Framing: Strongly negative correlation ($\beta \approx -0.33$). Directly challenging or rebutting the user reduced effectiveness.
• Asymmetry: Models were significantly more effective at persuading participants toward the Democratic/Pro-Policy position (e.g., supporting minimum wage increases) than toward the Republican/Opposition position.

5. Significance and Implications

• Democratic Risk: The findings suggest that frontier LLMs pose a substantial threat to democratic deliberation. Their ability to outperform human ads, combined with the potential for mass-scale deployment by malicious actors, could allow for the rapid manipulation of public opinion and election outcomes.
• Model-Specific Safety: The variation in performance implies that AI safety and alignment efforts must be model-specific. A "one-size-fits-all" mitigation strategy may fail if, for example, Claude is inherently more persuasive than GPT.
• Prompt Engineering Risks: The finding that "information-based" prompts can backfire (reducing GPT's effectiveness) suggests that current safety guidelines or prompt engineering best practices may need re-evaluation.
• Future Research: The paper calls for continuous benchmarking as models evolve and highlights the need to study AI-driven polarization and non-conversational influence (e.g., social media flooding).

In conclusion, the paper establishes that the "intelligence leap" in frontier LLMs has translated into a "persuasion leap," creating a new class of political actors that are more effective than traditional human campaigning, with significant implications for the future of democratic societies.