🤖 AI

Navigating the Sea of LLM Evaluation: Investigating Bias in Toxicity Benchmarks

This paper investigates inherent biases in current LLM toxicity benchmarks, revealing that factors such as task type, input domain, and model choice significantly compromise evaluation consistency and highlighting the urgent need for more robust safety assessment frameworks.

Original authors: Regina Gugg, Selina Niederländer, Andreas Stöckl, Martin Flechl

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Regina Gugg, Selina Niederländer, Andreas Stöckl, Martin Flechl

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a new employee to work in a customer service department. Before you hire them, you give them a series of tests to see if they are polite, safe, and won't say anything mean or dangerous.

This paper is like a group of researchers who decided to check if those "safety tests" are actually reliable. They found that the tests are surprisingly fragile, like a house of cards that falls over if you just change the wind direction slightly.

Here is a breakdown of their findings using simple analogies:

1. The "One-Size-Fits-All" Trap

Most current safety tests for AI are like a driving test where you only ever drive on a straight, empty highway. The AI passes easily. But in the real world, AI doesn't just drive on highways; it drives in heavy traffic, on winding mountain roads, and in the rain.

The researchers found that when they changed the "driving conditions" (the task) from a simple question-and-answer format to summarizing a long text, the AI's safety performance changed drastically.

The Analogy: Imagine a guard who is very good at stopping people at a front door (answering questions). But if you ask that same guard to sort through a pile of mail (summarizing), they might accidentally let dangerous letters slip through, or conversely, they might stop perfectly safe letters.
The Finding: When the AI was asked to summarize text instead of just answer it, the safety tests flagged much more content as "toxic" or "harmful." This suggests that if you only test an AI on simple questions, you might miss how it behaves when it's actually doing complex work like summarizing chat histories.

2. The "Accent" Problem

The researchers also tested if the AI's safety changed depending on the "topic" or "domain" of the conversation. They took the same harmful ideas and rewrote them to sound like they belonged in different worlds: Finance, Sports, Social Media, and Chemical Engineering.

The Analogy: Imagine a metal detector at an airport. It beeps loudly when you walk through with a knife. But what if you wrap that knife in a specific type of fabric or put it inside a box labeled "Chemistry Lab"? The researchers found that the safety detectors (the AI classifiers) often stopped beeping.
The Finding: When harmful content was dressed up in the language of specific industries (like using technical jargon in Chemistry or Finance), the safety tests were much less likely to catch it. It seems the "guards" are easily fooled by fancy vocabulary or specific contexts.

3. The "Referee" Disagreement

Finally, the researchers looked at the "referees" themselves—the automated tools used to grade the AI's safety. They used four different referees to grade the same AI responses.

The Analogy: Imagine a sports game where four different referees are watching the same play. You would expect them to agree on whether a foul was committed. Instead, the researchers found that these referees barely agreed with each other.
The Finding: The tools used to measure toxicity often gave completely different scores for the exact same sentence. One tool might say, "This is safe," while another says, "This is dangerous." They only agreed a tiny bit of the time. This means there is no single, universal "truth" about what is toxic; it depends entirely on which tool you use.

The Bottom Line

The paper concludes that we cannot blindly trust the current safety scores we see for AI models.

If you change the task (like asking the AI to summarize instead of chat), the safety score changes.
If you change the topic (like talking about sports instead of general chat), the safety score changes.
If you change the tool used to measure safety, the score changes.

The researchers are essentially saying: "We are trying to certify that AI is safe, but our measuring tapes are stretching and shrinking depending on how we use them. We need better, more consistent ways to test these models before we let them loose in the real world."

Technical Summary: Navigating the Sea of LLM Evaluation: Investigating Bias in Toxicity Benchmarks

Problem Statement

The rapid adoption of Large Language Models (LLMs) in research and industry has exposed a critical gap in the systematic evaluation of safety and toxicity benchmarks. While organizations increasingly rely on these benchmarks to certify models for customer-facing applications (e.g., summarization tools, customer support bots, and autonomous agentic workflows), current evaluation setups often lack robustness. The paper argues that unrecognized biases related to model choice, metrics, and task types can lead to the deployment of vulnerable or unsafe systems. Specifically, existing benchmarks often fail to generalize across different contexts or tasks, providing a false sense of security. Furthermore, the lack of consistent definitions for toxicity and the rapid proliferation of new benchmarks make it difficult to contextualize results or determine if widespread benchmark designs are limited by their creators' specific setups.

Methodology

To investigate the robustness of established benchmarking setups, the authors conducted a meta-study using four prominent safety benchmarks:

RealToxicityPrompts (RTP): Measures toxicity via text completion.
HarmBench: Assesses adversarial robustness and refusal behavior.
ToxiGen: Evaluates implicit hate speech.
DoNotAnswer (DNA): Tests refusal behavior to harmful instructions.

The study evaluated five LLMs (GPT-3.5-Turbo, Mistral-7B-Instruct, Llama2-7B-Chat, Qwen2-7B-Instruct, and DeepSeek-LLM-7B-Chat) across a baseline setup and two specific perturbation conditions:

1. Task Shifting

The authors replaced the original response tasks (e.g., direct Q&A, text completion) with summarization. This was chosen because summarization is underrepresented in current alignment benchmarks despite its prevalence in real-world agentic workflows. The goal was to determine if safety mechanisms hold when a model processes and condenses harmful inputs rather than responding directly to them.

2. Domain Shifting

Benchmark inputs were adapted into four target domains: Finance, Social Media, Chemical Engineering, and Sports. This was achieved using GPT-4o to rephrase inputs while preserving their original malicious intent (for harmful subsets) or neutrality (for benign subsets). The quality of these adaptations was validated using Claude 3 Haiku as an LLM judge across five criteria (Domain Transfer, Intent Preservation, Contextual Naturalness, Coherence, and Overall Quality), with mean scores exceeding 3.5/5.0.

3. Statistical Analysis

The study employed several statistical methods to analyze discrepancies:

McNemar's Test: Used to analyze task-shift effects on paired binary outcomes, reporting paired odds ratios ( $OR_p$ ).
Generalized Linear Models (GLMs): Used with binomial families and clustered standard errors to analyze domain-shift effects, reporting population-level odds ratios ( $OR_{pop}$ ).
Chi-Squared Tests & Cramér's V: Used to measure the effect size of domain sensitivity across specific model-benchmark combinations.
Cohen's Kappa ( $\kappa$ ): Used to measure pairwise agreement between the different classifiers used by the benchmarks (e.g., Perspective API, Longformer, ToxiGenRoBERTa).

Key Results

Task-Shift Discrepancies

Shifting the task from direct response to summarization significantly altered benchmark behaviors:

Increased Harmful Classifications: For DoNotAnswer (DNA) and ToxiGen, shifting to summarization consistently increased the rate of harmful classifications across all models. The paired odds ratios ( $OR_p$ ) ranged from 0.01 to 0.18, indicating a strong shift toward flagging content as harmful. RTP showed a similar pattern, though with wider confidence intervals.
Decreased Harmful Classifications: HarmBench behaved differently; summarization tended to reduce harmful outputs for specific models (GPT-3.5-Turbo and Mistral-7B-Instruct showed strong reductive effects with $OR_p$ of 18.0 and 39.0, respectively), while other models showed little change.
Conclusion: Changing the task framing significantly skews benchmark outcomes, highlighting a blind spot in standard direct-response evaluations.

Domain-Shift Sensitivity

Adapting inputs to specific domains generally decreased the detection of harmfulness, suggesting classifiers are easily "fooled" by domain-specific vocabulary:

RTP Sensitivity: The Perspective API classifier showed the strongest domain sensitivity. All four target domains significantly reduced toxic classifications for both toxic and non-toxic subsets, with technical domains (Chemical Engineering, Finance) producing the largest shifts.
Variable Effects: HarmBench and DNA showed more modest effects, with only specific domains (e.g., Social Media for DNA) reaching statistical significance. ToxiGen showed reduced toxicity detection across domains, though only the Finance domain was statistically significant for the hate subset.
Benign Inputs: Domain shifting also skewed results for benign inputs; for ToxiGen Neutral, Chemical Engineering and Finance domains significantly increased harmful classifications ( $OR_{pop} \approx 7.2$ ).
Model-Benchmark Interaction: Domain sensitivity was highly concentrated in specific benchmark-model combinations. HarmBench exhibited the most severe and unpredictable inconsistencies, with Mistral-7B-Instruct showing the largest effect size (Cramér's $V = 0.549$ ).

Cross-Benchmark Agreement

The study found remarkably low agreement among the evaluation classifiers:

Most classifier pairs exhibited only slight agreement ( $\kappa < 0.20$ ).
Only a minority of pairs showed moderate agreement, such as Longformer $\times$ Llama2-13B-cls on HarmBench ( $\kappa = 0.384$ ) and Perspective API $\times$ ToxiGenRoBERTa on RTP ( $\kappa = 0.331$ ). The authors note this likely reflects shared training data or architectures rather than a robust consensus on what constitutes harmful content.
Overall, the results highlight the subjective nature of current automated safety evaluations.

Significance and Claims

The paper claims that its findings cast significant doubt on the reliability of current benchmark suites as foundational decision-making tools for enterprise deployments. The authors emphasize that:

Instability is Systemic: Benchmark results are highly sensitive to simple changes in task framing (e.g., summarization) and context (domain shifting).
Real-World Risk: The inconsistency poses a substantial risk for applications requiring robust content moderation, particularly as summarization becomes ubiquitous in agentic workflows.
Need for Consolidation: There is an urgent need for more consolidated and systematic benchmarking efforts. The authors argue for aggregating results across benchmarks and rigorously studying stability to reduce duplicated effort and mitigate evaluation bias.
Future Directions: The field requires improved methodologies to contextualize benchmark outcomes, track true model progress, and account for systemic issues like dataset contamination. Developing frameworks to systematically adapt and cross-evaluate safety benchmarks for diverse, real-world use cases is identified as a critical direction for future work.

The authors maintain a modest tone, acknowledging that their study is limited by the specific set of models, benchmarks, and English-language focus used, and that no single study can provide a fully comprehensive evaluation without broader community efforts.