🤖 machine learning

Learning Sexism Detection Using Multi-Agent Perspectivist Preference Optimization

This paper introduces the Multi-Agent Perspectivist Preference Optimization (MAP-PO) framework, which addresses annotator disagreement in sexism detection by clustering labeling behaviors and fine-tuning specialized language model agents that are coordinated through preference optimization to preserve diverse perspectives while maintaining alignment with majority labels.

Original authors: Hadi Mohammadi, Tina Shahedi, Robert A. Bagheri, Mehdi Dastani, Masoume M. Raeissi

Published 2026-08-06

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Hadi Mohammadi, Tina Shahedi, Robert A. Bagheri, Mehdi Dastani, Masoume M. Raeissi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to spot a mean joke. You ask ten different people to read a joke and decide if it's sexist. Some say "Yes, that's terrible," while others say "No, that's just a joke." In the old days of computer science, if the majority said "Yes," the computer would just ignore the "No" votes and call it a fact. But what if those "No" votes aren't mistakes? What if they are just different people seeing the world through different lenses? This is the heart of a field called "perspectivist" science. It argues that when humans disagree, that disagreement isn't noise to be deleted; it's a signal to be studied. The big question is: how do you build a computer that doesn't just pick a winner, but actually understands why different people see things differently?

This paper introduces a clever new system called MAP-PO (Multi-Agent Perspectivist Preference Optimization) to solve exactly that problem. Instead of trying to force one robot to be the "perfect" judge, the researchers built a team of three specialized robot agents. Here's how they did it: First, they looked at the people who labeled the data (the humans) and didn't group them by age or gender. Instead, they grouped them by how they voted. Some humans were very strict and said "Yes" often; others were more lenient. The researchers found that these "voting styles" were the real key to understanding the data, not the humans' backgrounds.

Next, they trained one robot agent to mimic each of these three voting styles. But here was the tricky part: if they just told each robot, "Only listen to your own group," the robots went crazy. They became extreme caricatures, with one robot saying "Yes" to everything and another saying "No" to everything, losing touch with the real humans they were supposed to represent. The paper found that to fix this, the robots needed a "team signal." They had to be rewarded not just for being true to their own style, but also for working together to get the right overall answer. When they added this team reward, the robots stayed balanced. They learned to be distinct personalities without losing their minds. In the end, this team of three robots was better at spotting sexism than any single robot or any old-fashioned computer program, proving that keeping different perspectives alive makes for a smarter, more honest system.

Technical Summary: Learning Sexism Detection Using Multi-Agent Perspectivist Preference Optimization

Problem Statement

Sexism detection in natural language processing (NLP) is inherently subjective. Annotators often disagree on whether a specific text contains sexism, not due to error, but because they genuinely perceive the content differently. Standard NLP approaches typically discard this disagreement by collapsing multiple annotations into a single "ground truth" via majority vote, treating minority opinions as noise. However, perspectivist research argues that this disagreement is a signal that should be modeled rather than averaged away. The challenge lies in creating systems that can represent these diverse annotator perspectives without forcing a single consensus, while ensuring the system remains calibrated to the actual distribution of human judgments.

Methodology: MAP-PO

The authors propose Multi-Agent Perspectivist Preference Optimization (MAP-PO), a framework designed to preserve diverse annotator perspectives through a multi-agent system. The methodology proceeds in three main stages:

Behavioral Clustering:
Instead of grouping annotators by demographic attributes (age, gender, etc.), the authors cluster them based on labeling behavior. Using the EXIST 2024 dataset (English and Spanish tweets), annotators are characterized by three features:
- YES rate: The frequency with which an annotator labels text as sexist.
- Agreement rate: The frequency of agreement with the annotator majority.
- Label entropy: The balance of YES/NO decisions.
  K-means clustering (k=3) is applied separately for each language, resulting in three distinct behavioral clusters per language.
Agent Training (Fine-Tuning):
One Large Language Model (LLM) agent is trained per cluster. The training involves two phases:
- Phase 1 (SFT): Agents undergo Supervised Fine-Tuning (SFT) on the majority labels of their specific cluster. To prevent agents from drifting too far apart, the training data is mixed with "shared positive examples" (texts where all clusters agree) and "team-alignment examples" (texts labeled with the overall annotator majority).
- Phase 2 (Preference Optimization): The authors explore three optimization objectives to refine agent behavior:
  - DPO (Direct Preference Optimization): Trains agents to prefer their cluster's label over the opposite label on texts where clusters disagree.
  - Mars-PO: Adapts DPO for multi-agent systems by including shared preference pairs (unanimous texts) alongside individual pairs.
  - GRPO (Group Relative Policy Optimization): Uses a convex reward function combining an individual reward (matching the cluster's label) and a team reward (matching the overall annotator majority). The team weight ( $\alpha$ ) balances these objectives.
Inference:
At inference, the three agents predict independently. The final team prediction is determined by a majority vote among the agents. The system also utilizes agent disagreement as a confidence signal; high disagreement indicates texts where human annotators also disagree.

Key Contributions

Behavioral vs. Demographic Clustering: The paper demonstrates that behavioral features (YES rate, agreement, entropy) cluster annotators more effectively than demographic attributes for sexism detection. Statistical tests show no significant association between cluster membership and demographics like gender or age.
Multi-Agent Preference Optimization: The authors adapt the Mars-PO framework to fine-tune specialized LLM agents for a task where "ground truth" is perspective-dependent.
Reward Design Analysis: Through extensive experimentation across four settings (two languages $\times$ two backbone models: GPT-4.1-mini and Qwen3-8B), the paper identifies that training agents only on their own cluster's labels leads to extreme polarization (agents moving far beyond the behaviors they represent). Conversely, introducing a shared team-level signal (either implicit via data mixing in SFT or explicit via Mars-PO/GRPO rewards) is necessary to keep agents calibrated to their specific clusters while maintaining team accuracy.

Results

The evaluation covers four settings: GPT-EN, GPT-ES, Qwen-EN, and Qwen-ES. Key findings include:

Necessity of Fine-Tuning: Without fine-tuning, zero-shot agents behave almost identically (near 100% agreement), failing to capture diverse perspectives.
Polarization of Individual-Only Training: Pure individual preference optimization (e.g., DPO without team signals) causes agents to overshoot, with Cluster 1 agents labeling almost 0% of texts as sexist and Cluster 3 agents labeling nearly 100%, regardless of the actual cluster distribution. This leads to high disagreement and poor calibration.
Effectiveness of Team Signals: Adding a team signal consistently restores calibration.
- In the GPT-EN setting, the best system (GRPO with $\alpha=0.17$ ) achieves 90.3% Team Accuracy and 89.9 Team F1, outperforming all baselines.
- The SFT-mixed approach (implicit team signal) also performs robustly, achieving high calibration with minimal error.
- On the smaller Qwen backbone, SFT slightly outperforms GRPO, suggesting a capacity effect where rejection sampling in GRPO introduces noise relative to supervised labels.
Generalization: The core finding—that individual-only optimization polarizes agents and team signals repair them—holds true across both languages and both backbone models.

Significance and Claims

The paper claims that MAP-PO successfully models annotator disagreement as a signal rather than noise. Its primary significance lies in demonstrating that:

Perspectives are behavioral, not demographic: Effective agent identities for subjective tasks are derived from labeling behavior, not demographic personas.
Calibration requires a team anchor: To prevent agents from drifting into unrealistic extremes, a shared team-level training signal is essential. This signal ensures that while agents represent specific viewpoints, they remain grounded in the broader distribution of human judgment.
Disagreement is informative: The system's ability to produce divergent outputs serves as a useful confidence signal, flagging texts that are genuinely contested and may require human review, rather than forcing a potentially false consensus.

The authors position MAP-PO as an assistive tool for content moderation, intended to highlight contested cases rather than make autonomous moderation decisions, acknowledging limitations regarding dataset scope (EXIST 2024) and the specific block design of the data.