💬 NLP

Who Decides What Is Harmful? Content Moderation Policy Through A Multi-Agent Personalised Inference Framework

This paper proposes an LLM-based multi-agent framework that personalizes content moderation by simulating individual user sensitivities through specialized agents, achieving a 32% accuracy improvement over traditional centralized systems while offering a scalable approach to balancing platform governance with user autonomy.

Original authors: Ewelina Gajewska, Michal Wawer, Katarzyna Budzynska, Jaroslaw A. Chudziak

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ewelina Gajewska, Michal Wawer, Katarzyna Budzynska, Jaroslaw A. Chudziak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a massive, chaotic city square where millions of people are shouting, sharing stories, arguing, and posting memes all at once. Currently, the "security guards" (content moderation systems) trying to keep this square safe use a single, giant rulebook for everyone. They decide what is "harmful" based on a one-size-fits-all list. If a post breaks a rule, it gets removed. If it doesn't, it stays.

The problem, as the authors of this paper point out, is that people are different. What one person finds funny, another finds deeply hurtful. What one person sees as a bold political opinion, another sees as a personal attack. The current system ignores these individual feelings, often leaving people overwhelmed by content they find toxic, or conversely, removing things they actually enjoy.

This paper proposes a new way to run the security guard station. Instead of one rigid rulebook, they suggest a personalized, team-based approach called PRISM.

The Core Idea: A Personalized Security Team

Think of the PRISM system not as a single robot, but as a customized security team assigned to every single user. Here is how this team works, using simple analogies:

1. The "Sensitivity Profile" (Your Personal Rulebook)
Instead of starting with a blank slate, the system learns your specific "pain points."

The Analogy: Imagine you are very sensitive to loud noises but don't mind messy rooms. Your neighbor, however, hates mess but loves loud music. The PRISM system learns your specific thresholds. It builds a profile that says, "This user gets upset easily by insults, but they are fine with strong political arguments."
How it works: As you interact with the system (telling it, "I don't like this post" or "This is fine"), it updates your profile. It learns exactly what you find harmful, not what a generic algorithm thinks you should find harmful.

2. The "Manager Agent" (The Team Leader)
When a new post arrives, a central "Manager" looks at it and your profile.

The Analogy: The Manager is like a conductor in an orchestra. They look at the post and say, "This post has some aggressive language and mentions a specific group. We don't need the whole orchestra; let's just call in the Sociologist (who knows about group dynamics) and the Psychologist (who knows about emotional impact)."
The Twist: The Manager also whispers your specific preferences to the experts. They tell the Psychologist, "Remember, this user is extremely sensitive to dehumanizing language, so look for that specifically."

3. The "Expert Agents" (The Specialists)
The system uses different AI "experts" to analyze the content from different angles.

The Sociologist: Looks for discrimination or unfair treatment of groups.
The Linguist: Looks for rude words, insults, or aggressive tone.
The Psychologist: Looks for threats or things that might cause emotional distress.
The "Ghost" Agent: This is a special agent that acts exactly like you. If the experts disagree, the Ghost Agent steps in to say, "Based on what this user has told us before, they would find this harmful."

4. The "Synthesis Agent" (The Decision Maker)
Finally, a Synthesis Agent gathers all the opinions from the experts and the Ghost. It weighs their arguments against your personal profile and makes the final call: "Show this to the user" or "Hide this."

What Did They Find?

The researchers tested this system against the old "one-size-fits-all" methods and some of the smartest standard AI models currently used.

Better Accuracy: Their personalized team was 32% more accurate at predicting what a specific user would find harmful compared to the standard, generic systems.
Fewer Mistakes: The multi-agent team made fewer mistakes than a single AI trying to do everything alone. By breaking the problem down into smaller tasks (like having a specialist for insults and a specialist for threats), they avoided confusing the two.
Fast Learning: The system didn't need years to learn. Even after a user gave just a few pieces of feedback, the system started working much better than the generic version. It didn't take long to figure out your "taste."

The Big Picture: Who Decides?

The paper asks a big question: "Who decides what is harmful?"

Currently, the answer is "The Platform." They decide for everyone.
The PRISM system suggests the answer should be: "The User, with help from a smart team."

The authors argue that while platforms still need to keep the basics safe (like preventing violence), the day-to-day decisions about what is "too much" for a specific person should be up to that person. This system allows users to have a "filter" that fits their unique mental well-being, rather than forcing everyone to live under the same rigid rules.

In short: Instead of a security guard shouting "No one can say that!" to the whole crowd, PRISM gives every person a personal bodyguard who knows exactly what they can handle and what they can't, ensuring everyone feels safe in their own way.

Technical Summary: "Who Decides What Is Harmful? Content Moderation Policy Through a Multi-Agent Personalised Inference Framework"

Problem Statement
Current automated content moderation systems rely on centralised, top-down rules that apply universal thresholds to all users. This approach fails to account for the subjective nature of harm perception, where the same content may be perceived as hateful, humorous, or acceptable depending on an individual's cultural, social, and ideological background. While recommender systems have advanced personalisation for engagement, they often optimise for metrics that conflict with user well-being or fail to provide granular control over content exposure. The paper identifies a critical gap: the lack of scalable, user-driven filtering mechanisms that allow individuals to define their own "sensitivity profiles" regarding harmful content, thereby shifting the locus of control from the platform to the user.

Methodology: The PRISM Framework
The authors propose PRISM (Personalised Reasoning and Inference System for Moderation), a user-in-the-loop framework that dynamically adapts content filtering based on individual sensitivity profiles. The system architecture consists of three primary stages:

User Profile Construction and Management:
- Each user profile ( $P_u$ ) maintains dimension-specific thresholds and weights across ten dimensions of harmful content (e.g., sentiment, respect, insult, dehumanisation, violence, genocide) derived from established hate speech taxonomies.
- New users begin with population-level median thresholds. As users provide explicit feedback (flagging content as offensive or neutral), the system updates their profile using an Exponential Moving Average (EMA) learning algorithm.
- The system calculates a confidence score ( $\kappa_u$ ) based on the volume of feedback, transitioning from reliance on population priors to learned user preferences as confidence approaches unity (after approximately 100 interactions).
- Dimension weights are computed based on the variance of severity scores, prioritising dimensions where the user demonstrates consistent sensitivity.
Multi-Agent Inference Engine:
The core reasoning is distributed among a coordinated set of Large Language Model (LLM) agents to mimic human deliberative judgment:
- Manager Agent: Orchestrates the pipeline by analysing content, selecting 1–3 relevant Expert Agents based on content characteristics, and generating dynamic, personalised prompts. It translates numerical thresholds into semantic descriptors (e.g., "extremely sensitive") to guide the experts.
- Expert Agents: Specialised agents (Sociologist, Linguist, Psychologist) provide domain-specific analysis. For instance, the Sociologist Agent focuses on identity-based discrimination, while the Psychologist Agent evaluates potential for emotional distress.
- Ghost Profile Agent: A dynamic agent created to simulate the specific user's perspective when expert agents disagree. It acts as a tie-breaker aligned strictly with the user's learned sensitivity thresholds.
- Synthesis Agent: Aggregates the reasoning, decisions, and confidence scores from all invoked agents to produce a final binary classification (hate/neutral).
Content Filtering Workflow:
Upon receiving new content, the system loads the user's profile, orchestrates the multi-agent evaluation, and compares the final weighted decision score against the user's personalised thresholds. Content exceeding the threshold is hidden; otherwise, it is displayed.

Key Contributions

Perspectivist Ground Truth: The paper reframes harmfulness not as an objective property but as a user-dependent construct, utilising individual-level annotations to define ground truth rather than aggregated consensus.
Multi-Agent Personalised Architecture: It introduces a novel architecture that combines domain-specific expert reasoning with a "Ghost Profile" agent to simulate user perspectives, addressing the limitations of monolithic classifiers in capturing multidimensional harm.
Dynamic Learning Mechanism: The system employs an adaptive learning rate and online learning to continuously refine user profiles, allowing the system to evolve with changing user sensitivities and contexts.
Policy Framework: The authors propose design principles for platform governance that balance user autonomy with platform safety, suggesting mechanisms like "opt-out" for severe toxicity and the delegation of profile creation to trusted community entities to reduce user burden.

Results
The system was evaluated using the Measuring Hate Speech dataset, which contains 39,565 comments annotated by 7,912 users, allowing for the construction of 100 distinct synthetic user profiles representing varying severity levels (from "very strict" to "very lenient").

Experiment 1 (Universal vs. Personalised): The PRISM multi-agent system achieved a macro-averaged F1 score of 81.5%, representing a 31.9% relative improvement over the universal baseline (61.8%) and a 7.0 percentage point improvement over a single-agent personalised baseline (74.5%). The multi-agent approach notably improved precision (85.2%), suggesting that expert decomposition effectively reduces false positives.
Experiment 2 (Comparison with SOTA): PRISM outperformed state-of-the-art non-personalised transformer models (BERTweet, TimeLMs, RoBERTa, BERT) on the same dataset. It surpassed the closest competitor, BERTweet (80.5%), by 1.0 percentage point, demonstrating that user-specific adaptation offers value beyond architectural sophistication or domain-specific pre-training.
Experiment 3 (Learning Dynamics): The system demonstrated rapid convergence, surpassing the universal baseline with as few as two feedback interactions. This indicates that the system mitigates "cold-start" concerns, providing immediate benefits without extended calibration periods.

Significance and Claims
The paper claims that personalised, user-in-the-loop moderation is both technically feasible and socially desirable. By anchoring moderation decisions in subjective sensitivity profiles, the framework bridges the gap between platform governance and individual empowerment. The authors argue that this approach challenges the assumption that harmfulness must be adjudicated through universal policies. Empirically, the results suggest that integrating user-specific information significantly enhances model performance and fairness across demographic groups. Theoretically, the work advances the understanding of hybrid, human-centric content moderation, offering a scalable pathway to reconcile moderation policies with individual digital rights and well-being. The authors remain modest regarding long-term convergence, noting that their evaluation focused on the early-to-mid adaptation regime, and highlight the need for future work on balancing user autonomy with the cognitive burden of safety management.

The Core Idea: A Personalized Security Team

What Did They Find?

The Big Picture: Who Decides?

Technical Summary: "Who Decides What Is Harmful? Content Moderation Policy Through a Multi-Agent Personalised Inference Framework"

More like this