🤖 AI

Evaluating Online Moderation Via LLM-Powered Counterfactual Simulations

This paper introduces an LLM-powered simulator that enables counterfactual evaluation of online moderation strategies, demonstrating their effectiveness in curbing toxic discourse and revealing insights into social contagion and personalized intervention.

Original authors: Giacomo Fidone, Lucia Passaro, Riccardo Guidotti

Published 2026-04-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Giacomo Fidone, Lucia Passaro, Riccardo Guidotti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the mayor of a bustling, chaotic digital town square called "The Feed." Every day, people (agents) gather there to chat, share news, and sometimes, unfortunately, scream insults at one another. As the mayor, you want to stop the fighting, but you face a massive problem: How do you test a new rule without actually ruining the town?

If you try a new rule in the real world, you might accidentally make things worse, or you might not know if the rule worked because other things changed at the same time (like a sudden storm or a celebrity visiting). It's too risky and expensive to run these experiments on real people.

This is where the paper "Evaluating Online Moderation Via LLM-Powered Counterfactual Simulations" comes in. The authors built a digital twin of this town square called COSMOS.

Here is the simple breakdown of how it works, using some creative analogies:

1. The Digital Twin: A "Butterfly Effect" Simulator

Think of COSMOS as a time-traveling video game where you can run two versions of the same day simultaneously:

The "Real" Day (Factual): People chat, get angry, and insult each other naturally. No rules are enforced.
The "What-If" Day (Counterfactual): This is an exact copy of the "Real" Day. The same people, the same topics, the same moods. The only difference is that in this version, the "Mayor" (the moderator) steps in to stop the fights.

Because the two days are identical except for the intervention, you can see exactly what changed. Did the insults stop? Did the anger spread differently? It's like watching a movie, then rewinding it and changing one scene to see how the ending changes.

2. The Actors: AI Avatars with Personalities

The "people" in this simulation aren't random bots. They are LLM-powered agents (AI characters) given detailed "character sheets."

The Profile: Each agent has a fake name, age, job, political views, and a personality test score (like the "Big Five" traits: are they grumpy? are they kind? are they impulsive?).
The Magic: The AI is so good at role-playing that if you give an agent a "grumpy" personality, it naturally acts grumpy in the chat. If you give it a "kind" personality, it acts nice. This makes the simulation feel real, not robotic.

3. The Experiment: Testing Different "Mayor" Strategies

The researchers used COSMOS to test three different ways the "Mayor" could stop the fighting:

Strategy A: The "One-Size-Fits-All" Warning (OSFA)
- The Analogy: Imagine the Mayor puts up a generic sign that says, "No Fighting! Be Nice!" for everyone.
- The Result: It helps a little, but it's like using a sledgehammer to crack a nut. It doesn't fit everyone's specific problem.
Strategy B: The "Personalized" Note (PMI)
- The Analogy: The Mayor looks at the grumpy person's profile and writes a specific note: "Hey, I know you're having a bad day and you're usually very kind, but please calm down."
- The Result: This worked much better. Because the message was tailored to the person's personality, they actually listened. It's like a doctor prescribing the right medicine for a specific patient rather than giving everyone the same pill.
Strategy C: The "Ban Hammer" (Ex Post)
- The Analogy: If someone fights, the Mayor kicks them out of the town square immediately.
- The Result: This stopped the fighting from that specific person, but it also meant the town square became empty. You lose the conversation entirely. It's effective at stopping noise, but you lose the community.

4. The Big Discovery: The "Contagion" Effect

One of the coolest things the simulation found was Toxic Contagion.

The Analogy: Imagine a drop of red dye in a glass of water. If one person starts yelling, the person they yell at gets angry and yells back, and then that person yells at someone else. The anger spreads like a virus.
The simulation showed that when the "Mayor" stopped the first person from yelling (using the personalized note), the "virus" stopped spreading. The whole town became calmer, even for people who never got a warning.

Why Does This Matter?

In the real world, social media companies (like Facebook or X) often guess what rules work. They might ban people too quickly or send generic warnings that no one reads.

COSMOS allows them to run a "stress test" before changing the rules.

They can ask: "If we change the rule to be more empathetic, will the fighting stop?"
They can see the answer in a safe, simulated environment without hurting real people or ruining real conversations.

The Catch (Limitations)

The authors are honest about the flaws:

The Actors aren't perfect: Sometimes the AI actors get confused or say weird things (hallucinations), though this happens rarely.
It's not everything: The simulation focuses on chatting. It doesn't simulate "likes," "shares," or complex social networks yet. It's a chat room simulator, not a full social media platform.
Cost: Running these simulations takes a lot of computer power, like running a massive server farm just to watch a fake town argue.

The Bottom Line

This paper introduces a virtual laboratory for social media. By creating a digital twin of an online community with realistic AI people, researchers can test moderation strategies safely. They discovered that personalized, empathetic warnings work better than generic rules or just banning people, and that stopping the first spark of anger can prevent a whole forest fire of toxicity.

It's like having a crystal ball that lets you see the future of an online argument before it even happens.

1. Problem Statement

Online Social Networks (OSNs) face significant challenges in mitigating toxic discourse and affective polarization. While platforms employ content moderation strategies (e.g., bans, warning messages), evaluating their true effectiveness is difficult due to:

Data Scarcity: Private OSNs restrict API access, and toxic behavior is sparse in real-world datasets.
Lack of Control: Field observations lack experimental control, making it impossible to isolate the impact of moderation from unknown confounders.
Cost: Collecting large-scale empirical evidence for different moderation scenarios is prohibitively expensive and time-consuming.

Existing Agent-Based Modeling (ABM) tools lack the capability to simulate human-like social behavior with sufficient psychological realism to test these strategies effectively.

2. Methodology: COSMOS

The authors introduce COSMOS (COunterfactual Simulations of MOderation Strategies), a simulator designed to evaluate moderation strategies by running parallel, counterfactual experiments.

Core Architecture

Dual Simulation Environment: COSMOS runs two parallel simulations for every scenario:
1. Factual Simulation ( $F$ ): Agents interact freely without moderation.
2. Counterfactual Simulation ( $\hat{F}$ ): A replica of the factual run where moderation interventions are applied, while all other variables (agent profiles, initial topics, random seeds) remain constant.
LLM-Based Agents: Agents are powered by the open-source, uncensored SOLAR-10B model. Each agent is initialized with a Profile Module containing:
- Demographics: Age, gender, race, income, education, political leaning (derived from the General Social Survey).
- Psychology: Big Five personality traits (OCEAN) (derived from the PANDORA dataset).
Interaction Dynamics:
- Agents select actions (Post, Comment, Do Nothing) based on probability distributions.
- Sensory Modules: Agents receive context (topic for posts; parent thread for comments).
- Memory Modules: Agents store past moderation interventions, which influence future behavior.
Moderation Mechanisms:
- Toxicity Detection: Uses Google's Perspective API with a threshold ($THR=0.6$).
- Ex Post (Ban): Agents are banned if their violation count exceeds a tolerance $e$ .
- Ex Ante (Pre-emptive):
  - One-Size-Fits-All (OSFA): A generic warning message.
  - Personalized Moderation Intervention (PMI): A message generated by an LLM acting as a moderator, tailored to the specific agent's profile and the toxic content. Variants include Neutral, Empathizing, and Prescriptive.

Experimental Setup

Topics: Includes neutral topics and potentially contentious issues (e.g., abortion, climate change, fake news).
Metrics:
- Realism: Correlation between toxicity and psychological traits; consistency of agent behavior over time.
- Effectiveness: Mass divergence ( $\Delta M$ ) and Quantile divergence ( $\Delta q$ ) between factual and counterfactual toxicity distributions.
- Cost: Content Loss Ratio (CLR) to measure how much content is removed (banned).

3. Key Contributions

Novel Simulation Framework: COSMOS is the first tool to enable parallel, counterfactual simulation of OSN conversations specifically for evaluating moderation strategies, isolating the causal effect of interventions.
Psychological Realism: The system demonstrates that LLM agents, when prompted with socio-demographic and psychological profiles, exhibit toxic behaviors that statistically mirror real-world data (PANDORA), including correlations with traits like low agreeableness and high neuroticism.
Discovery of Toxicity Contagion: The simulation reveals that toxicity spreads through social interactions (parent-child node influence), validating the "contagion" hypothesis in a controlled environment.
Evaluation of Personalization: The framework provides empirical evidence that personalized moderation is superior to generic approaches.

4. Key Results

Realism Validation:
- Simulated agents showed significant Spearman correlations between toxicity and Big Five traits (e.g., low agreeableness $\rho = -0.32$ ), matching real-world patterns.
- Toxicity was found to propagate across threads, with a significant correlation ( $\rho = +0.39$ ) between parent and child node toxicity.
Moderation Effectiveness:
- Personalized vs. Generic: PMI (Personalized) strategies, particularly the Neutral variant, achieved the most significant reductions in toxicity mass ( $\Delta M$ ) without removing agents. OSFA (One-Size-Fits-All) showed minimal impact.
- Ban Trade-offs: Strict banning strategies (low tolerance $e=1$ ) reduced toxicity significantly but resulted in a high Content Loss Ratio (CLR $\approx$ 45%), meaning nearly half the content (including "healthy" content) was lost. This suggests bans are a blunt instrument that removes users rather than correcting behavior.
- Psychological Sensitivity: Personalized interventions were most effective against "prototypical toxic" agents (low agreeableness, high neuroticism).
- Targeting Extremes: Moderation strategies primarily reduced extreme toxicity ( $q \geq 0.8$ ), with less impact on mild toxicity.

5. Significance and Limitations

Significance:

Research: COSMOS offers a low-cost, high-control environment for social scientists to test hypotheses about online behavior and the efficacy of policy interventions before real-world deployment.
Industry: It provides a testing ground for automated moderation systems, allowing platforms to optimize strategies (e.g., choosing between bans and personalized warnings) to maximize safety while minimizing content loss.
Methodological Shift: It demonstrates that LLMs can serve as valid proxies for human agents in complex social simulations, moving beyond simple rule-based models.

Limitations:

Hallucinations: Approximately 7% of generated content was identified as "redundant hallucinations" (unbelievable OSN content).
Scope: The simulator focuses on conversational dynamics and excludes other OSN features like "likes," "follows," or complex recommendation algorithms, which limits the simulation of phenomena like echo chambers driven by algorithmic curation.
Scalability: Running LLM-based simulations for real-sized populations is computationally expensive.
Bias: LLMs may amplify societal biases present in their training data, and the "self-preference" bias of LLMs acting as judges/moderators remains an open concern.

Conclusion

The paper establishes that LLM-powered counterfactual simulations are a viable and powerful method for evaluating online moderation. The study concludes that personalized, ex-ante interventions are significantly more effective at mitigating toxicity than generic warnings or strict bans, which often result in excessive content loss. COSMOS bridges the gap between theoretical social science and practical platform governance.