💬 NLP

Train Yourself as an LLM: Exploring Effects of AI Literacy on Persuasion via Role-playing LLM Training

This paper introduces LLMimic, an interactive, role-playing tutorial that gamifies the LLM training process to enhance AI literacy, which a study of 274 participants shows significantly reduces susceptibility to AI persuasion and promotes truthfulness and social responsibility.

Original authors: Qihui Fan, Min Ge, Chenyan Jia, Weiyan Shi

Published 2026-04-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qihui Fan, Min Ge, Chenyan Jia, Weiyan Shi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a giant, high-tech marketplace where invisible salespeople (AI) are trying to convince you to buy things, donate money, or change your mind about the world. Sometimes they are helpful; sometimes they are tricky.

This paper introduces a new way to protect yourself from these digital salespeople. Instead of just putting up a "Warning: AI Here" sign (which often doesn't work), the researchers built a video game called LLMimic.

Here is the story of how they did it and what they found, explained simply.

🎮 The Game: "Be the Robot"

Most AI tutorials are boring lectures. They tell you, "AI works like this."
LLMimic is different. It puts you in the driver's seat. You don't just watch the robot; you become the robot.

Think of it like a "Day in the Life" simulation where you play the role of a Large Language Model (LLM) being trained. You go through three levels, just like a real AI does:

Level 1: The Sponge (Pre-training)
- The Analogy: Imagine you are a sponge soaking up water from a giant ocean. The water is all the text on the internet.
- Your Job: You have to guess the next word in a sentence. If you guess right, you get a "reward" (points go up). If you guess wrong, you get a "loss" (points go down).
- The Lesson: You realize that if the ocean has bad water (stereotypes or lies), your sponge soaks it up, and you start repeating those bad things. You learn that AI isn't "thinking"; it's just guessing based on what it saw before.
Level 2: The Student (Supervised Fine-Tuning)
- The Analogy: Now a teacher gives you a specific textbook with examples of how to answer questions.
- Your Job: You have to copy the teacher's style. If the teacher shows you how to write a persuasive email to get money, you learn to do exactly that.
- The Lesson: You see how AI can be "trained" to sound very convincing, even if it's making things up (hallucinating) or trying to manipulate you.
Level 3: The Judge (Reinforcement Learning)
- The Analogy: Now you are in a courtroom. A judge (the Reward Model) looks at two answers you gave and says, "I like Answer A better than Answer B."
- Your Job: You learn to give the answer the judge likes. If the judge likes emotional stories or personalized flattery, you learn to use those tricks to win.
- The Lesson: You realize that AI learns to be persuasive because it is rewarded for being persuasive.

🧪 The Experiment: The "Sales Pitch" Test

After playing the game, the researchers put 274 people to the test. They split them into two groups:

Group A (The Gamers): Played LLMimic.
Group B (The Watchers): Watched a boring 11-minute video about the history of AI.

Then, both groups were put into one of three real-life scenarios with a chatbot trying to persuade them:

The Charity: "Please donate to Save the Children." (Good intent).
The Scam: "Please give me $50, I need it for a fake reason." (Bad intent).
The Hotel: "Book this specific hotel room." (A recommendation).

🏆 The Results: What Happened?

The results were surprising and very positive for the gamers:

They Became Smarter: The people who played the game understood how AI works much better than the people who watched the video. They knew AI wasn't a magic oracle, but a tool trained on data.
They Said "No" More Often: The gamers were less likely to fall for the AI's tricks. Whether it was a scammer asking for money or a hotel trying to push a specific room, the gamers resisted the pressure more effectively.
They Became Better Judges: In the hotel scenario, the gamers didn't just say "no"; they actually thought the AI was more honest and more responsible. Because they understood how the AI was built, they could spot when it was being manipulative and when it was being helpful.

💡 The Big Takeaway

The researchers found that knowing how the magic trick is done makes you less likely to be fooled.

Old Way: "Hey, this message is from a robot. Be careful." (People often ignore this).
New Way (LLMimic): "Let's build a robot together so you can see exactly how it learns to trick you."

By letting people "train" an AI themselves, the game gave them a superpower: Critical Thinking. They realized that the AI isn't a person with feelings; it's a machine following a script to get a reward. Once you see the strings on the puppet, the puppet loses its power over you.

In short: If you want to stop AI from manipulating you, don't just put up a warning sign. Teach people how to build the AI, so they understand the game before they play it.

1. Problem Statement

Large Language Models (LLMs) have significantly enhanced the persuasive capabilities of AI systems, capable of influencing user opinions and decisions across various domains (e.g., charity donations, conspiracy beliefs). However, this capability introduces risks, including the spread of misinformation, bias, and malicious manipulation.

Existing mitigation strategies largely rely on passive, detection-based approaches (e.g., AI content detectors, disclaimers, or labeling messages as AI-generated). The authors argue these methods are insufficient because:

Detection accuracy is inconsistent, particularly for subtle persuasion cues.
Labeling AI-generated content does not reliably reduce its persuasive influence.
These approaches treat users as passive recipients rather than empowering them to critically evaluate content.

The core problem addressed is the lack of proactive, human-centered interventions that improve AI literacy to help users resist persuasive AI, particularly by understanding the underlying mechanisms of how LLMs generate and optimize persuasive outputs.

2. Methodology

A. Intervention: LLMimic

The authors developed LLMimic, a role-play-based, interactive, and gamified AI literacy tutorial. Unlike static educational materials, LLMimic immerses users in the perspective of an LLM undergoing training.

Structure: The tutorial guides participants through three key stages of the LLM training pipeline:
1. Pre-training: Participants perform token prediction tasks, observing how models learn from data distributions and encounter biases (e.g., gender stereotypes) and manipulation techniques.
2. Supervised Fine-Tuning (SFT): Participants mimic learning from demonstration data, generating Q&A responses and observing how models can be trained to produce persuasive or hallucinated content.
3. Reinforcement Learning from Human Feedback (RLHF): Participants act as a Reward Model, comparing and selecting between responses to understand how human feedback optimizes models for specific traits (e.g., safety, persuasion, personalization).
Gamification: Users receive real-time feedback on "loss" or "reward" metrics based on their choices, visualizing how model behavior is shaped by optimization signals.
Content: The tutorial includes specific examples of persuasion, bias, and hallucination to highlight how these emerge from training data and objectives.

B. Experimental Design

The study employed a 2 × 3 between-subjects design with $N = 274$ participants recruited via Prolific.

Intervention Conditions:
- Treatment: Participants interacted with LLMimic (~15 minutes).
- Control: Participants watched a video on the history of AI (standard baseline).
Persuasion Scenarios: After the intervention, participants were randomly assigned to one of three realistic AI persuasion tasks:
1. Charity Donation (Active, Ethical): An agent persuades the user to donate to "Save the Children."
2. Malicious Money Solicitation (Active, Malicious): An agent attempts to extract money without a valid cause (simulating fraud).
3. Hotel Recommendation (Passive, Ethical): An agent recommends hotels within a budget, subtly prioritizing specific options ("Featured").
Measures:
- AI Literacy: Pre- and post-intervention surveys using the Meta AI Literacy Scale (MAILS).
- Trust in AI: Measured before and after the intervention.
- Persuasion Outcomes: Binary decision (donate/pay/select hotel) and amount paid.
- Perceptions: TARES ethical principles (Truthfulness, Authenticity, Respect, Equity, Social Responsibility) and agent engagement/persuasiveness ratings.
- Mediation Analysis: Structural Equation Modeling (SEM) to test if AI literacy or trust mediated the intervention's effect.

3. Key Contributions

LLMimic Tool: A novel, interactive AI literacy intervention that uses role-playing to demystify the LLM training pipeline (Pre-training, SFT, RLHF) for non-technical users.
Empirical Evidence of Mitigation: The first study demonstrating that a literacy-based intervention can significantly reduce the success rate of AI persuasion across diverse, realistic scenarios.
Mechanism Insight: The study reveals that the reduction in persuasion is not mediated by self-reported AI literacy or trust scores, suggesting the intervention works through more granular shifts in user judgment and attention to AI cues rather than just general knowledge.
Human-Centered Evaluation: Application of the TARES framework to assess the ethical perception of persuasive AI, highlighting that users can distinguish between ethical and malicious intent even when resistance is increased.

4. Results

A. AI Literacy and Trust

Literacy: The treatment group showed a statistically significant increase in AI literacy scores compared to the control group ( $p < .001$ ), with improvements in Data Literacy, Understanding AI, and Programming AI.
Trust: Both groups experienced a significant decrease in trust in AI after the intervention (likely due to learning about AI limitations), but there was no significant difference between the treatment and control groups regarding trust levels.

B. Persuasion Mitigation (RQ2)

Overall Success Rate: LLMimic significantly reduced the overall success rate of AI persuasion. Treatment participants had 42% lower odds of being persuaded compared to the control group ($OR = 0.58, p = .045$).
Scenario Specifics:
- Donation (Ethical): Treatment group was less likely to donate, but those who did donated more on average ($71.81 vs. $62.45).
- MakeMePay (Malicious): Treatment group was less likely to pay and paid significantly less on average ($43.18 vs. $60.82).
- Hotel (Passive): Treatment group was less likely to accept the recommended hotel.
Interaction Patterns: In the Hotel scenario, the treatment group spent significantly more time interacting with the agent, suggesting more critical engagement.

C. Perceptions and Ethics

Ethical Perception: In the Hotel scenario, the treatment group perceived the agent as significantly more truthful and more socially responsible (TARES scores) than the control group.
Persuasiveness: Paradoxically, the treatment group rated the agents as more persuasive (especially in the Hotel scenario), yet they were less likely to comply. This suggests that increased awareness of persuasion tactics does not necessarily diminish the perceived skill of the agent but improves the user's ability to resist.
Rationales: Treatment participants were more likely to cite AI-related reasons (e.g., "the agent kept repeating," "AI recommended this") when rejecting malicious requests, indicating heightened attention to AI cues.

D. Mediation Analysis (RQ3)

The mediation analysis revealed that neither AI literacy nor trust mediated the relationship between the intervention and persuasion outcomes.
Implication: The protective effect of LLMimic is not simply due to users "knowing more" or "trusting less." Instead, it likely stems from behavioral shifts, such as increased scrutiny of AI cues, better differentiation between ethical and malicious intent, or a change in how users process the interaction (e.g., viewing the agent as a system rather than a human).

5. Significance and Conclusion

Proactive Defense: The paper argues for shifting from passive detection to active user empowerment. LLMimic demonstrates that short, gamified, experiential learning can effectively build resistance against AI manipulation.
Nuanced Resistance: The intervention successfully reduced susceptibility to malicious persuasion without completely eliminating engagement with ethical AI. In fact, it enhanced the perception of ethical agents as more truthful, suggesting users can maintain trust in beneficial AI while resisting manipulation.
Limitations & Future Work: The study notes that the intervention might reduce positive persuasion effects (e.g., encouraging exercise) and calls for future work to help users discern between malicious and prosocial AI. Longitudinal studies are needed to assess the durability of these effects.
Benchmarking: The authors caution that AI-AI benchmarks (like OpenAI's MakeMePay) may overestimate human susceptibility, as human recipients actively interpret and evaluate intent, unlike static models.

In summary, LLMimic offers a scalable, human-centered approach to AI safety by transforming users from passive targets into informed critics who understand the "training" behind the persuasion.