Prosocial Persuasion at Scale? Large Language Models Outperform Humans in Donation Appeals Across Levels of Personalization

1. Problem Statement

While Large Language Models (LLMs) are widely studied for their ability to generate persuasive misinformation, their potential to drive costly prosocial behavior (specifically charitable giving) remains underexplored. Key research gaps include:

• Human vs. LLM Benchmarking: It is unclear if LLMs can match or exceed human persuaders in motivating actual financial donations, which involve real costs and emotional engagement, unlike "cheap talk" attitude changes.
• The Role of Personalization: Theoretical frameworks (e.g., Elaboration Likelihood Model, Social Identity Theory) suggest personalized messages are more effective. However, empirical evidence on whether LLM-generated personalized content outperforms generic content or human-written content is mixed.
• False Personalization: The impact of "falsely personalized" content (where the message claims to target a demographic the recipient does not belong to) on trust and persuasion is not well understood in the context of AI-generated text.

2. Methodology

The authors conducted two preregistered online experiments ($N_1 = 658$; $N_2 = 642$) using a 2 (Content Source: Human vs. LLM) × 3 (Personalization: Generic, Personalized, Falsely Personalized) within-subjects factorial design.

Experimental Design & Procedure:

• Participants: US-based participants recruited via Prolific, quota-balanced for age, gender, political affiliation, and religiosity.
• Task: Participants viewed six social media posts (modeled after Twitter/X) from fictional charities. They were asked to:
  1. Allocate a portion of a $0.10 bonus to the charities.
  2. Rate engagement (Like/Neutral/Dislike).
  3. Rate perceived persuasiveness (7-point Likert scale).
• Stimuli Generation:
  • LLM: All LLM posts were generated using GPT-4.5-preview.
  • Study 1 (Human Writers): Posts were written by 16 graduate/postgraduate students with academic training in behavioral sciences.
  • Study 2 (Human Writers): To test generalizability and improve human baseline performance, 164 US laypeople (Prolific users) wrote posts. They were incentivized ($10 bonus for top 10) to write for their own demographic cluster, creating a "self-targeted" condition.
• Personalization Logic:
  • Personalized: Tailored to the participant's actual cluster (Age, Gender, Political Ideology, Religiosity).
  • Falsely Personalized: Tailored to a mismatched cluster (e.g., a religious participant receiving a message for a non-religious persona).
  • Generic: No demographic tailoring.

Analysis:

• Linear mixed-effects models (LMM) were used with fixed effects for Content Source, Personalization, and their interaction.
• Random intercepts were included for participants and charities to account for data dependence.
• Dependent variables: Donation Amount (% of bonus), Post Engagement, and Perceived Persuasiveness.

3. Key Results

A. Content Source Effect (LLM vs. Human)

In both studies, LLM-generated content significantly outperformed human-authored content across all metrics:

• Donation Amount: Participants donated a higher percentage of their bonus to LLM posts ($M \approx 13-14\%$) compared to human posts ($M \approx 10-13\%$).
• Engagement: LLM posts received significantly more "Likes" and fewer "Dislikes."
• Persuasiveness: LLM posts were rated as more convincing and compelling.
• Note: This superiority held even in Study 2, where human writers were incentivized and wrote for their own demographic groups.

B. Personalization Effects

• Study 1: Personalization did not significantly increase donations over generic messages. However, falsely personalized messages resulted in significantly lower donations and engagement compared to both generic and personalized messages.
• Study 2: True Personalization significantly outperformed both generic and falsely personalized messages in generating donations.
  • Interaction: The "penalty" for false personalization was consistent. In Study 2, personalized appeals generated the highest donations ($13.7\%$), followed by generic ($11.1\%$) and falsely personalized ($11.0\%$).
• LLM Sensitivity: LLM-generated content was particularly sensitive to false personalization; mismatched LLM messages were rated significantly less persuasive than generic LLM messages.

C. Interaction Effects

• There was generally no significant interaction between Content Source and Personalization regarding donation amounts, suggesting LLMs outperform humans regardless of the personalization level.
• However, for engagement and persuasiveness, interactions were observed where LLMs maintained high performance in generic/personalized conditions but suffered a sharp drop in performance when falsely personalized.

4. Key Contributions

1. First Empirical Evidence of LLM Superiority in Prosocial Behavior: This is the first study to demonstrate that LLMs can outperform humans in eliciting costly prosocial behavior (monetary donations), challenging the assumption that human empathy is required for effective fundraising.
2. Nuanced View on Personalization: The study clarifies that while accurate personalization can boost prosocial outcomes (especially when writers understand the target audience), false personalization is detrimental. It acts as a "trust penalty," reducing effectiveness below even generic levels.
3. Robustness of Findings: The results held across different human writer populations (academic experts vs. incentivized laypeople), suggesting the LLM advantage is robust and not merely a function of human writer quality.
4. Mechanism Exploration: The authors propose that LLMs may excel due to:
   • Higher grammatical fluency and stylistic polish (increasing perceived credibility).
   • Access to vast corpora of persuasive patterns and rhetorical strategies.
   • Effective use of morally charged and emotional semantics.

5. Significance and Implications

• For Fundraising and NGOs: LLMs represent a scalable, high-efficiency tool for generating donation appeals. Organizations should prioritize accurate data for personalization; if data reliability is low, generic appeals are safer than risking mismatched personalization.
• For AI Ethics and Safety: While much research focuses on LLM risks (misinformation), this paper highlights a "positive" use case. However, it also warns that the ability to generate highly persuasive, emotionally resonant content at scale could be misused if not regulated.
• For Future Research:
  • Disclosure: The study did not disclose AI authorship. Future work must investigate if the "LLM advantage" disappears when users know the content is AI-generated (the "AI disclosure effect").
  • Cultural Generalizability: Current LLMs are trained heavily on Western (WEIRD) data. Their persuasive superiority may not translate to non-Western cultures or languages with lower data coverage.
  • Interactive Contexts: Future studies should examine LLM persuasion in bidirectional, interactive dialogues rather than one-way static posts.

Conclusion: The paper concludes that LLMs are currently superior to humans in generating persuasive donation appeals, provided that personalization is executed accurately. False personalization undermines this advantage, suggesting that data integrity is the critical bottleneck for scaling AI-driven prosocial interventions.