🤖 machine learning

Understanding Annotator Safety Policy with Interpretability

This paper introduces Annotator Policy Models (APMs), an interpretable framework that infers annotators' internal safety policies directly from their labeling behavior to distinguish between operational failures, policy ambiguity, and value pluralism, thereby enabling more targeted and inclusive safety policy design without additional annotation burden.

Original authors: Alex Oesterling, Donghao Ren, Yannick Assogba, Dominik Moritz, Sunnie S. Y. Kim, Leon Gatys, Fred Hohman

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alex Oesterling, Donghao Ren, Yannick Assogba, Dominik Moritz, Sunnie S. Y. Kim, Leon Gatys, Fred Hohman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the head chef of a massive restaurant (an AI system). Your goal is to serve food that is safe for everyone to eat. To do this, you hire a team of 100 different food critics (annotators) to taste your dishes and decide: "Is this safe to serve?" or "Is this poisonous?"

The problem? The critics disagree. Sometimes one critic says a spicy dish is "safe," while another says it's "toxic." Sometimes they disagree because they didn't read the menu correctly. Sometimes they disagree because the menu instructions were vague. And sometimes, they disagree because they genuinely have different cultural tastes.

Usually, the restaurant manager just takes a vote. If 51% say "safe," the dish goes out. But this is dangerous. It hides the fact that 49% of your critics think it's poison, and it doesn't tell you why they disagreed.

This paper introduces a new tool called Annotator Policy Models (APMs). Think of APMs as "Mind-Reading Glasses" that let you see exactly how each critic decides what is safe, just by looking at their past notes, without ever having to ask them to explain themselves.

Here is how the paper breaks it down:

1. The Problem: The "Black Box" of Disagreement

When critics (human or AI) label data, they often disagree.

Operational Failures: The critic didn't understand the job. (e.g., They were supposed to taste the chef's sauce, but they tasted the customer's spicy salsa instead).
Policy Ambiguity: The rulebook was confusing. (e.g., The rule says "No offensive language," but doesn't say if quoting a bad word for a lesson counts as a violation).
Value Pluralism: The critics just have different values. (e.g., One critic thinks a joke is funny; another thinks it's hurtful).

Usually, we just count the votes. But if we don't know why they voted the way they did, we can't fix the menu or the rules.

2. The Solution: The "Mind-Reading Glasses" (APMs)

Instead of asking the critics, "Why did you vote this way?" (which is slow, expensive, and people often lie or forget), the authors built a computer model that learns the critic's internal rulebook just by watching what they vote on.

How it works: The model looks at thousands of past votes. It finds patterns. For example, it might realize, "Ah, this specific critic always votes 'unsafe' whenever the word 'gun' appears, but they don't care about the word 'knife'."
The Magic: The model translates these patterns into simple, readable rules (like a flowchart). It doesn't just say "Unsafe"; it says "Unsafe because: Gun + No Context."
No Extra Work: The critics don't have to do anything new. The model just studies their existing work.

3. What the Glasses Revealed

The authors tested these glasses on two groups: AI critics (LLMs) and human critics.

A. Catching Mistakes (Operational Failures)
The glasses showed that some critics were looking at the wrong thing. They were judging the customer's rude question instead of the chef's polite answer. The model spotted this pattern instantly, allowing the manager to retrain those specific critics.

B. Finding Vague Rules (Policy Ambiguity)
The glasses showed that some critics were confused by the rules. For example, when a chef tried to change the subject to avoid a rude question, some critics marked it as "unsafe" because they wanted a direct refusal. The model highlighted this confusion, telling the managers: "Hey, our rulebook needs to be clearer about what counts as a 'good' refusal."

C. Hearing Different Voices (Value Pluralism)
This was the most interesting part. The glasses found that critics from different backgrounds (like different ages or education levels) had different "safety priorities."

Example: One group of critics thought a model politely changing the subject was safe. Another group thought it was dangerous because it felt like the model was hiding the truth.
The Insight: If you just take a majority vote, you silence the minority group. The APMs let you see that these different groups exist and what they care about, so you can design a system that respects more perspectives, rather than just the loudest one.

4. The "What-If" Test

To make sure the glasses were working, the authors played a trick. They took a text that a critic marked as "unsafe," and they used the model to figure out exactly what made it unsafe. Then, they asked an AI to change just that one thing (e.g., swap a "bomb" for a "cake").

The Result: When they showed the new text to the original critic, the critic changed their vote to "safe."
Why it matters: This proved the model wasn't just guessing; it actually understood the critic's logic. It knew exactly which ingredient caused the "poison" label.

Summary

The paper argues that we shouldn't just count votes to decide what AI is safe. We need to understand the individual logic behind the votes.

By using these Annotator Policy Models, we can:

Fix training mistakes.
Clarify confusing rules.
Ensure that diverse viewpoints aren't accidentally erased by a simple majority vote.

It's like moving from a restaurant where you just take a vote on the menu, to a restaurant where you can see exactly why every single critic liked or disliked a dish, so you can make the food better for everyone.

Technical Summary: Understanding Annotator Safety Policy with Interpretability

Problem Statement

Safety policies are fundamental to defining and mitigating harmful AI outputs, yet the annotation processes used to train and evaluate these systems suffer from pervasive disagreement. This disagreement stems from three distinct sources:

Operational Failures: Annotators misunderstand or misexecute the task (e.g., labeling the user prompt instead of the model response).
Policy Ambiguity: Safety guidelines contain wording that allows for multiple valid interpretations (e.g., whether deflection constitutes a safe refusal).
Value Pluralism: Annotators hold genuinely different perspectives on safety based on demographics, culture, and personal values.

Current practices, such as majority voting, conflate these sources, often failing to address genuine ambiguity or silencing minority perspectives. Furthermore, directly asking annotators to explain their reasoning is costly, increases annotation burden, and is often unreliable due to post-hoc rationalization or the unfaithfulness of chain-of-thought reasoning in LLMs. The paper addresses the challenge of distinguishing these sources of disagreement to enable targeted responses (e.g., better training, policy refinement, or inclusive design) without requiring additional human effort.

Methodology: Annotator Policy Models (APMs)

The authors introduce Annotator Policy Models (APMs), interpretable models that learn individual annotators' internal safety policies directly from their labeling behavior.

Core Architecture

APMs utilize Concept Bottleneck Models (CBMs) to decompose the prediction process into two stages:

Concept Encoding ( $g$ ): Input text is mapped to a binary feature space of $c$ safety-relevant concepts. To avoid the cost of human concept labeling, the authors employ a Label-Free CBM approach. They generate a candidate concept list from safety datasets (BeaverTails) using GPT-4o, cluster and summarize them, and then use embedding models (Gemini) to label the presence of these concepts in text via cosine similarity and the Sparsemax operator. Crucially, the concept space captures factual presence (e.g., "mention of a weapon") rather than interpretive judgments (e.g., "harmful mention").
Policy Learning ( $f$ ): For each annotator $k$ $k$ , a simple, interpretable function is trained to map the binary concept vector to a safe/unsafe label. The authors propose two model classes:
- Non-Negative Logistic Regression (NNLR): Weights are constrained to be non-negative, ensuring the model defaults to "safe" and only predicts "unsafe" when positive features accumulate sufficient evidence.
- Disjunctive Normal Form (DNF): Boolean rules in "ORs-of-ANDs" form, where any single rule firing triggers an "unsafe" prediction. This aligns with the intuition that a single policy violation renders text unsafe.

Design Principles

Positive Features Only: Features represent the presence of concepts, not their absence, to avoid spurious correlations (e.g., "unsafe because it lacks food").
Safe Until Proven Unsafe: Models default to a safe prediction unless specific unsafe features are present.

Validation and Results

The authors validate APMs using both LLM and human annotation data.

LLM Validation

Using datasets BeaverTails and WildGuardMix, APMs were trained on five different LLM annotators (GPT-4o, GPT-4o-mini, and various Qwen models).

Predictive Accuracy: APMs achieved 73–90% accuracy and 0.78–0.95 AUC (for NNLR) across LLM annotators, demonstrating that simple interpretable models can approximate the behavior of complex foundation models.
Generalization: Concepts mined from BeaverTails generalized well to WildGuardMix with minimal performance degradation.
Controlled Policy Recovery: In synthetic experiments where "Alice" (treats weapons as safe) and "Bob" (treats drugs as safe) were simulated, APMs successfully recovered these known policy differences. "Diffing" the models (comparing feature sets) automatically identified the specific concepts driving the divergence.
Counterfactual Faithfulness: When inputs were minimally modified to remove features identified by the APM as causing an "unsafe" label, the original annotator's prediction changed in the majority of cases, confirming that APMs capture causally relevant features rather than mere correlations.

Human Annotation Applications (DICES Dataset)

Applied to the DICES dataset (350 interactions rated by 123 human annotators), APMs surfaced three key insights:

Operational Failures: Identified annotators who systematically labeled the full interaction as unsafe rather than just the model response, indicating a misunderstanding of the task.
Policy Ambiguity: Revealed disagreements on edge cases, such as whether "deflection" or "follow-up questions" on harmful topics constitute safe responses, highlighting where policy wording is unclear.
Value Pluralism: By training APMs on demographic groups (e.g., Millennial vs. Gen X+, College-educated vs. High School), the authors found systematic differences in safety priorities. For instance, Millennial annotators were more likely to flag "kind redirection" of flat-earth arguments as unsafe compared to Gen X+ annotators.
- Unique Rule Contribution (URC): The study validated that unique rules learned by group-specific APMs preferentially activated on samples where that group disagreed with others, confirming these rules capture genuine behavioral differences.
- Majority Vote Suppression: The analysis showed that majority-vote aggregation obscures the perspectives of minority groups. APMs identified a significant proportion of samples deemed unsafe by specific demographic groups but safe by the majority, suggesting that standard aggregation erases these safety concerns.

Key Contributions

Interpretable Annotator Policy Models: The introduction of APMs to explicitly model individual annotator safety policies using ML interpretability techniques without requiring additional annotation effort.
Extensive Validation: Demonstration that APMs accurately model annotator behavior (>80% accuracy on LLMs), faithfully predict responses to counterfactual edits, and recover known policy differences in controlled settings.
Surfacing Disagreement Sources: Empirical evidence that APMs can distinguish operational failures, policy ambiguity, and value pluralism in human annotation pipelines.
Demographic Analysis: The revelation of systematic safety prioritization differences across demographic groups and the demonstration of how majority voting suppresses these diverse perspectives.

Significance and Claims

The paper claims that APMs provide a mechanism to make annotator reasoning visible and comparable, moving beyond opaque aggregation methods. By disentangling the sources of disagreement, developers can respond appropriately:

Operational failures can be addressed through improved training and quality control.
Policy ambiguity can be resolved through targeted policy refinement and clarification.
Value pluralism can be addressed through deliberation on how to incorporate diverse perspectives, potentially by concatenating unique rules from underrepresented groups to the majority-vote model to ensure inclusive safety coverage.

The authors emphasize that while the focus is on safety, the methodology is generalizable to any subjective annotation task. They note limitations, including the reliance on LLM-generated concepts and the fact that APMs reveal what annotators disagree on, but not necessarily the deep sociological why without further user studies. The work positions APMs as a diagnostic tool to improve the transparency, targeting, and inclusivity of AI safety policy design.