Persuasion with Large Language Models: A Survey of Empirical Evidence, Study Methodologies, and Ethical Implications

1. Problem Statement

The rapid emergence of Large Language Models (LLMs) has fundamentally altered the landscape of persuasive communication. Unlike previous eras where persuasion relied on human-generated content or limited mass-media targeting, LLMs enable fully automated, hyper-personalized, and interactive content generation at an unprecedented scale.

The core problem addressed by this survey is the lack of a comprehensive synthesis regarding:

• The empirical effectiveness of LLM systems in changing human attitudes, beliefs, and behaviors compared to human persuaders.
• The methodological rigor of current studies (e.g., experimental designs, metrics).
• The key technical and contextual factors (e.g., model scale, personalization, prompt engineering) that drive persuasive success.
• The ethical and societal risks associated with deploying such systems, including misinformation, manipulation, and threats to autonomy.

2. Methodology

The authors conducted a systematic literature review of research published between January 2022 and October 2025.

• Search Strategy: A query was run on Scopus using keywords related to "Large Language Models," "Generative AI," "Persuasion," and "Human/User Studies." This yielded ~4,783 peer-reviewed articles and ~4,171 preprints.
• Eligibility Criteria:
  • Must use an LLM to generate text with the intention to persuade.
  • Must report a human-facing outcome (e.g., attitude shift, behavioral compliance, validated persuasion scale).
  • Must exclude theoretical pieces, studies measuring only intrinsic text properties, or systems that only recommend rather than generate text.
  • Must be in English and published within the specified timeframe.
• Synthesis: The final corpus was analyzed to extract data on application domains, influencing factors, experimental designs, success metrics, and comparative persuasiveness (LLM vs. Human).

3. Key Contributions

The paper provides a structured taxonomy of the field, organized into four main pillars:

A. Application Domains

The survey categorizes LLM persuasion applications into five primary domains:

1. Public Health: Generating vaccination messages, addiction cessation aids, and lifestyle coaching.
2. Politics: Creating policy arguments, campaign ads, propaganda, and interactive debate agents.
3. E-commerce & Marketing: Shopping chatbots, sales negotiations, and personalized advertising.
4. Misinformation: Both the adversarial generation of disinformation and the defensive deployment of AI for debunking/fact-checking.
5. Charity: Soliciting donations through interactive dialogue and rewriting campaign materials.

B. Factors Influencing Persuasiveness

The authors identify five critical technical and design factors:

1. Interaction Approach: While static (one-shot) messages are common, interactive, multi-turn dialogues often yield stronger persuasion by allowing real-time adaptation to user counter-arguments, though results vary by domain.
2. Model Scale & Capability: Larger models generally perform better, but diminishing returns are observed beyond a certain parameter count. Crucially, specialized fine-tuning can allow smaller models to outperform larger, generic frontier models.
3. AI Source Labeling: A significant "AI penalty" exists; content explicitly labeled as AI-generated is often perceived as less competent and persuasive than identical content labeled as human-written. However, high-quality personalization can mitigate this penalty.
4. Prompt Design: The "persona" and rhetorical strategy are vital. Prompts emphasizing facts and evidence generally outperform emotional appeals for LLMs (unlike humans, who often respond better to narratives). Structural constraints and few-shot prompting significantly enhance output quality.
5. Personalization: Tailoring messages to user psychological profiles (e.g., Big Five traits) can boost persuasion, but results are mixed. Some studies suggest generic high-quality AI messages are nearly as effective as micro-targeted ones, while others show significant gains in specific contexts (e.g., conspiracy debunking).

C. Experimental Design & Metrics

The survey critiques the current state of evaluation:

• Design: There is a shift toward Randomized Controlled Trials (RCTs), often with pre-registration. However, longitudinal studies (measuring effects over weeks/months) remain rare.
• Metrics:
  • Direct Outcomes: Belief change, attitude shift, and behavioral intention/action (e.g., purchasing, donating).
  • Proxy Metrics: Perceived message quality, source credibility, and AI detection accuracy. The paper notes that high perceived quality does not always correlate with actual behavioral change.
• Synthetic Simulations: An emerging trend involves using LLM-driven "user agents" to simulate human responses, allowing for rapid prototyping, though the validity of these agents in modeling human psychology is still debated.

D. Comparative Performance (LLM vs. Human)

• On Par: In many domains (politics, health), LLMs achieve persuasiveness comparable to human experts.
• Superhuman: Under specific conditions (e.g., optimized multi-step strategies, personalization, or in interactive debates), LLMs have demonstrated superhuman performance, outperforming human experts and even paid human persuaders.
• Inferior: LLMs generally underperform when the "AI label" is disclosed, triggering a bias against non-human sources, or when the task requires deep domain expertise that the model lacks.

4. Results & Findings

• Effectiveness: LLM systems are capable of shifting attitudes and behaviors at a scale and speed impossible for human teams. In political microtargeting and interactive debates, they have shown the ability to shift candidate preferences and policy support significantly.
• The "Human Penalty": The most consistent negative factor is the disclosure of AI authorship. When users know they are interacting with an AI, trust and persuasiveness drop, even if the content quality is high.
• Scalability: The ability to generate hyper-personalized content at scale without proportional resource increases is the primary disruptive advantage of LLMs.
• Ethical Risks: The paper highlights severe risks regarding information integrity (generation of convincing disinformation), societal polarization (echo chambers via microtargeting), privacy (exploitation of personal data for manipulation), and autonomy (covert manipulation).

5. Significance

This survey is significant for several reasons:

1. State-of-the-Art Inventory: It serves as the first comprehensive mapping of the rapidly evolving field of LLM-based persuasion, synthesizing hundreds of disparate studies.
2. Methodological Critique: It exposes gaps in current research, specifically the lack of longitudinal data and the over-reliance on proxy metrics (perceived quality) rather than behavioral outcomes.
3. Policy & Regulation: It provides evidence-based arguments for the urgent need for updated regulatory frameworks (e.g., the EU AI Act) and ethical guidelines. It argues that current regulations are insufficient to handle the speed of AI advancement and the specific risks of automated, personalized manipulation.
4. Future Directions: It calls for the development of standardized benchmarks that measure actual behavioral change and advocates for research into the long-term societal impacts of pervasive AI persuasion.

In conclusion, the paper posits that LLMs have already reached a level of persuasive capability that rivals or exceeds human communicators in specific contexts. Without robust ethical guidelines and regulatory oversight, the widespread deployment of these systems poses profound risks to democratic processes, individual autonomy, and social cohesion.