💬 NLP

PluriHarms: Benchmarking the Full Spectrum of Human Judgments on AI Harm

This paper introduces PluriHarms, a novel benchmark designed to move beyond binary safety assessments by systematically analyzing the full spectrum of human judgments on AI harm, specifically focusing on the dimensions of harm severity and inter-annotator disagreement to enable the development of more pluralistic and personalized AI safety systems.

Original authors: Jing-Jing Li, Joel Mire, Eve Fleisig, Valentina Pyatkin, Anne Collins, Maarten Sap, Sydney Levine

Published 2026-02-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jing-Jing Li, Joel Mire, Eve Fleisig, Valentina Pyatkin, Anne Collins, Maarten Sap, Sydney Levine

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to be "good." Most current safety systems work like a strict traffic cop with only two signals: Green (Go, this is safe) and Red (Stop, this is dangerous).

The problem, as this paper explains, is that real life isn't just green and red. It's full of yellow lights, foggy intersections, and confusing detours where people genuinely disagree. One person might think a joke is funny; another might think it's hurtful. One person might see a political question as a debate; another sees it as a threat.

The authors of this paper, PLURIHARMS, argue that by only looking for clear-cut "bad" things, we are missing the messy middle ground where AI safety actually breaks down. They built a new tool to study this gray area.

Here is a breakdown of their work using simple analogies:

1. The Problem: The "One-Size-Fits-All" Safety Net

Current AI safety is like a cookie cutter. It tries to cut every piece of human conversation into a perfect circle (Safe) or a jagged square (Unsafe).

The Flaw: If you ask a group of people, "Is this joke mean?" you won't get one answer. You'll get a spectrum. Some will laugh, some will cringe, and some will be offended.
The Mistake: Current AI treats these different opinions as "noise" or mistakes to be averaged out until everyone agrees on a single "safe" answer. The authors say this is wrong. The disagreement is the signal. It tells us that people have different values, backgrounds, and experiences.

2. The Solution: The "Harm Spectrum" Benchmark

The team created a new dataset called PLURIHARMS. Think of this as a color wheel instead of a stoplight.

The Setup: They generated 150 different questions (prompts) that range from "completely harmless" (like asking about the weather) to "clearly dangerous" (like asking how to traffic children).
The Middle Ground: Crucially, they focused heavily on the middle colors—the yellow and orange prompts. These are the tricky questions where people disagree the most (e.g., "Is it okay to write a story about a controversial historical figure?").
The Human Element: They didn't just ask the AI to judge these. They asked 100 real humans to rate every single prompt on a scale of 0 to 100.
The Data: They also asked these humans about their lives: their age, education, political views, how much they use social media, and how much they've been bullied online.

3. What They Discovered: Why We Disagree

When they analyzed the data, they found that "harm" isn't just about the words in the question; it's about who is asking and who is answering.

The "Tangible Danger" Rule: Humans generally agree that things causing immediate, physical harm (like hurting a child or committing a crime) are bad. This is the "Red" zone.
The "Identity" Factor: Where people disagree is in the "Yellow" zone.
- Analogy: Imagine a question about a specific type of music. A person who has been bullied online might rate a question about that music as "very harmful" because it reminds them of trauma. A person with a different background might rate the same question as "harmless."
- The Finding: The paper shows that your background (like your education level or how often you use social media) and your past experiences (like being a victim of online toxicity) act like sunglasses. They change the color of the world you see. If you've been hurt before, you see more danger in the same prompt than someone who hasn't.

4. Testing the AI: Can Robots Learn to See Through Different Eyes?

The authors tested various AI safety models on this new "color wheel" dataset to see if they could predict how specific humans would react.

The Old Way (Consensus): They tried to train the AI to predict the average human opinion.
- Result: The AI was okay at this, but it missed the nuance. It was like trying to guess what a whole crowd thinks by asking one person.
The New Way (Personalization): They tried to train the AI to predict what one specific person would think, based on that person's profile.
- Result: This worked much better! When the AI was "personalized" to understand a specific user's values and history, it could predict their safety ratings much more accurately.
- The Takeaway: A "one-size-fits-all" safety rule is weak. A safety system that can adapt to who is using it is much stronger.

Summary

PLURIHARMS is a new tool that says: "Stop trying to force everyone to agree on what is safe."

Instead, it treats disagreement as a feature, not a bug. It shows that safety is a personal experience. Just as you might wear different shoes for hiking than for dancing, AI safety shouldn't be a single rigid rulebook. It needs to be flexible enough to understand that a "harmful" question for one person might be a "harmless" question for another, and that the best AI safety systems will be the ones that can understand and adapt to those different perspectives.

Technical Summary: PLURIHARMS – Benchmarking the Full Spectrum of Human Judgments on AI Harm

Problem Statement

Current AI safety frameworks predominantly treat harmfulness as a binary classification (benign vs. harmful). This monolithic approach fails to address the "ambiguous middle ground" where human communities and individuals hold diverse values and perspectives. By focusing on clear-cut extremes, existing safety datasets obscure the borderline cases where systems are most likely to fail. Furthermore, current practices often treat annotator disagreement in these borderline cases as statistical noise to be averaged out, rather than as a critical signal of legitimate, diverse viewpoints. This binary strategy creates a barrier to developing pluralistic AI systems capable of navigating complex, context-dependent safety considerations.

Methodology

The authors introduce PLURIHARMS, a benchmark designed to systematically study human harm judgments across two key dimensions: the harm axis (ranging from benign to harmful) and the agreement axis (ranging from consensus to disagreement).

1. Data Curation Framework

The dataset was constructed using a scalable, multi-stage automated framework:

Prompt Generation: Starting with 5,692 seed prompts from AIR-Bench 2024, the authors used DeepSeek-V3 to generate 11 variants per seed, spanning an ordinal harm scale from 0.0 (completely benign) to 1.0 (unambiguously harmful). This resulted in 62,612 candidate prompts.
Feature Extraction: Specialized models were employed to extract human-interpretable features:
- SafetyAnalyst generated "harm trees" decomposing prompts into 16 harmful action categories and 7 harmful effect categories.
- KALEIDO annotated prompts with 39 distinct human value categories (e.g., rights, duties, freedoms).
Strategic Curation: A genetic algorithm selected a final set of 150 prompts. The selection criteria prioritized over-sampling intermediate harm levels (0.4–0.8) to maximize disagreement rates while ensuring diversity across actions, effects, and values.

2. Human Annotation

Participants: 100 human annotators recruited via Prolific.
Task: Each annotator rated all 150 prompts on a continuous 0–100 scale (0 = benign, 100 = maximum harm).
Annotator Traits: The study collected comprehensive demographic data (age, gender, race, education, income, political affiliation, etc.) and psychological measures (Schwartz Value Survey, Moral Foundations Questionnaire, Interpersonal Reactivity Index, AI literacy).

3. Analysis and Evaluation

Statistical Modeling: Mixed-effects linear regression models with Lasso regularization were used to analyze the contributions of prompt features (actions, effects, values) and annotator traits (demographics, psychology) to harm judgments. Interactions between traits and prompt features were explicitly modeled.
AI Benchmarking: The authors evaluated various AI safety models (WildGuard, SafetyAnalyst, GPT-4.1, Claude, Qwen) and alignment methods (zero-shot, value profile steering, k-shot steering) on their ability to predict human ratings. Comparisons were made between models aligned to aggregated (mean) ratings versus personalized (individual) ratings.

Key Results

1. Validation of the Harm Spectrum

Human ratings strongly correlated with the synthetic harm levels assigned during generation (Spearman $r = 0.59$ ). Crucially, intermediate-to-high harm levels (0.4–0.8) exhibited significantly higher response variance and entropy, confirming that the dataset successfully targets cases with high human disagreement.

2. Drivers of Harm Judgments

Prompt Features: Features related to imminent risks and tangible harms (e.g., Child Harm, Self Harm, Criminal Activities, Physical Harm) exerted the strongest positive influence on perceived harmfulness. Conversely, features related to indirect, non-physical, or institutional domains (e.g., Psychological Harm, Social Harm, Cultural Diversity) were associated with lower perceived harmfulness.
Annotator Traits: Annotator traits significantly predicted ratings. Higher exposure to online toxicity and social media frequency correlated with higher harmfulness ratings. Conversely, higher education, male gender, and liberal political affiliation correlated with lower ratings.
Interactions: Disagreement is not merely additive; it emerges from structured interactions. For instance, the influence of prosocial orientations was amplified by education, and political differences became more pronounced when combined with online toxicity experience. Specific demographic groups (e.g., non-straight individuals, minorities) showed heightened sensitivity to specific harm types like Child Harm.

3. AI Model Performance

Personalization vs. Aggregation: Personalized alignment methods consistently outperformed aggregated methods. Aligning models to individual annotator ratings significantly reduced Mean Absolute Error (MAE) compared to aligning to mean human ratings.
Method Comparison: k-shot steering (providing prompt-rating pairs in-context) yielded the strongest performance, outperforming value profile summaries. This suggests that natural language value profiles do not fully capture the latent factors driving human judgments.
Model Capabilities: General-purpose models (e.g., GPT-4.1, Claude) often outperformed specialized safety models (e.g., WildGuard, SafetyAnalyst) in predicting nuanced harm ratings, though specialized models showed fewer refusal/completion failures.

Key Contributions

Framework: A scalable framework for generating calibrated prompts spanning an ordinal harm spectrum, specifically emphasizing disagreement-prone borderline cases.
Benchmark (PLURIHARMS): A dataset of 150 prompts with 15,000 human ratings, enriched with annotator traits (demographic/psychological) and interpretable prompt-level features (actions, effects, values).
Insights: Empirical evidence that prompt features tied to tangible harms amplify perceived harmfulness, while annotator traits and their interactions explain systematic disagreement, highlighting the role of pluralistic perspectives.
Evaluation: A benchmark of safety models demonstrating that personalized alignment significantly improves prediction accuracy over consensus-based approaches, while highlighting the limitations of current systems.

Significance and Claims

The paper argues that moving beyond "one-size-fits-all" safety toward pluralistically safe AI requires systems that model disagreement rather than collapsing it into consensus. PLURIHARMS provides a principled benchmark for this shift.

The authors claim that:

Safety judgments are not objective facts but are shaped jointly by content and pluralistic human perspectives.
Treating annotator disagreement as noise obscures the "ambiguous middle ground" where safety systems are most likely to fail.
Personalized alignment is a necessary step for capturing the nuance of human harm judgments, as aggregated models inevitably blur the systematic differences arising from diverse social identities and value orientations.

The work is modest about its scope, acknowledging limitations in scale, demographic coverage (U.S.-based, English-only), and the focus on prompts rather than full human-AI interactions. However, it establishes a foundational step for understanding and modeling the full spectrum of human harm judgments.