💬 NLP

LLM Judges Inconsistently Disagree Across Safety Criteria and Harm Categories

This paper reveals that Large Language Models used as automated judges exhibit significant inconsistency and unreliability, particularly when evaluating regulated domains like finance or varying by language and criteria, highlighting the need for careful implementation in safety assessments.

Original authors: Krishnapriya Vishnubhotla, Soumya Vajjala, Akriti Vij, Isar Nejadgholi

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Krishnapriya Vishnubhotla, Soumya Vajjala, Akriti Vij, Isar Nejadgholi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have hired a panel of six different art critics to judge a series of paintings. You want them to agree on which paintings are "safe" (good, appropriate) and which are "unsafe" (bad, harmful). You expect that if you show them the same painting, or even a slightly different version of it (like a photo of the painting in a different frame, or a translation of the artist's note into another language), they should all give roughly the same verdict.

This paper is about testing exactly that, but instead of art critics, the "judges" are Large Language Models (AI), and instead of paintings, they are judging AI-generated text for safety.

Here is the breakdown of what the researchers found, using simple analogies:

1. The "Obvious vs. Subtle" Problem

The researchers tested the AI judges on two very different types of "bad" content:

The "Screaming Fire" Test (Violence): They asked the AI to judge text about violence, hate speech, or illegal acts.
- The Result: The AI judges were pretty good at this. If a painting was clearly on fire, all six critics agreed, "This is unsafe."
The "Financial Advice" Test (Regulated Domains): They asked the AI to judge text where an AI gives advice on things like credit scores, visa applications, or medical advice.
- The Result: The AI judges were terrible at agreeing here. It's like asking six critics to judge a painting that is almost a masterpiece but has one tiny, subtle flaw. One critic says, "It's safe, just a little risky," while another says, "It's dangerous, throw it out!" They couldn't agree on what "safe" even meant in these complex, real-world scenarios.

2. The "Magic Mirror" Problem (Self-Consistency)

The researchers took a single AI response and showed it to the judges in different "mirrors":

Translation: They showed the text in English, then French, then Hindi, etc.
Paraphrasing: They rewrote the text to sound more formal, more casual, or to change the order of sentences, without changing the actual meaning.
The Result: The judges were inconsistent. If a judge said, "This English sentence is safe," they might look at the exact same sentence translated into Hindi and say, "Wait, this Hindi version is unsafe!" Or, they might look at a rephrased version and change their mind.
The Metaphor: Imagine a security guard who stops a person in a red shirt but lets the same person in a blue shirt pass, even though they are the exact same person. The AI judges are easily tricked by how the words are dressed up, not just what the words actually say.

3. The "Jury" Problem (Cross-Consistency)

The researchers also asked: "If we put all six AI judges in a room together, will they agree with each other?"

The Result: No. Even when looking at the exact same text in the exact same language, the judges often gave opposite answers.
The Metaphor: Imagine a jury where one person votes "Guilty," another votes "Not Guilty," and a third votes "Maybe." They are all looking at the same evidence, but they have completely different internal rulebooks for what counts as a crime. The paper found that for complex topics (like financial advice), the "jury" is often in total chaos.

4. The "Fake Agreement" Trap

The paper points out a tricky statistical illusion.

Sometimes, all the judges say "Safe" 99% of the time. If you just count the "Yes" votes, it looks like they are in perfect agreement (100% consistency).
The Catch: But if they are just saying "Safe" because they are lazy or biased, and not actually thinking about the specific details, that's not real agreement. The researchers used special math (like a "chance-corrected" score) to strip away this fake agreement.
The Metaphor: It's like a group of friends guessing the answer to a trivia question. If the answer is "Yes" 99% of the time, and they all guess "Yes," they look like geniuses. But if the question is actually hard and they are just guessing the most common answer, they aren't actually agreeing on the reasoning. The paper shows that when you look deeper, the AI judges are actually disagreeing a lot.

The Bottom Line

The paper concludes that while AI judges are okay at spotting obvious, loud dangers (like violence), they are unreliable and inconsistent when judging nuanced, real-world advice (like finance or law).

If you use an AI to act as a safety guard for complex topics, you can't trust it to be consistent. It might pass a dangerous answer just because the sentence structure changed slightly, or it might reject a safe answer just because it was translated into a different language. The "jury" of AI models is currently too divided to be trusted for high-stakes decisions without human oversight.

Technical Summary: LLM Judges Inconsistently Disagree Across Safety Criteria and Harm Categories

Problem Statement

The use of Large Language Models (LLMs) as automated judges ("LLM-as-a-Judge") has become a prevalent paradigm for evaluating model safety, often replacing or supplementing human annotation due to scalability. However, existing research predominantly focuses on the accuracy of these judges against human annotators or assumes safety as a uni-dimensional construct (safe vs. unsafe).

This paper addresses a critical gap: the consistency of LLM judges when evaluating safety across multi-dimensional criteria and semantically invariant transformations (e.g., translations, paraphrases) in a reference-free setting. The authors investigate whether LLMs provide reliable, stable judgments when the input content is altered linguistically but retains the same semantic meaning, and whether different judges agree with one another (cross-consistency) across diverse safety dimensions, particularly in nuanced, regulated domains versus overtly harmful content.

Methodology

The study employs a reference-free evaluation workflow designed to measure both self-consistency (a single judge's stability across variations) and cross-consistency (agreement among multiple judges).

1. Data Construction

Seed Queries: The authors utilized 89 seed queries from the AIRBench-2024 dataset, categorized into two distinct risk types:
- OPMISUSE (Operational Misuse): 50 queries focusing on "Automated Decision Making" and "Advice in Regulated Industries" (e.g., finance, hiring). These represent nuanced, culture-dependent scenarios.
- VIOLENCE: 39 queries covering overtly harmful content (violence, extremism, self-harm, criminal activities).
Variation Generation:
- Cultural/Geographical Variations: For OPMISUSE, 5 variations per query were generated by adapting names, locations, and currencies to different cultures (totaling 300 queries).
- Translations: All queries and responses were translated into five languages (French, Arabic, Farsi, Telugu, Hindi) in addition to English.
- Paraphrases: For OPMISUSE responses, 10 semantically consistent variations were generated per response, altering tone (formal vs. conversational), structure (verdict-first vs. verdict-last), and explicitness, while preserving factual content.
Response Generation: All responses were generated by a single strong generator model (GPT-4o-Mini) to isolate judge variability.

2. Evaluation Setup

Judges: Six state-of-the-art LLMs were used as judges: Gemini-2.5-Flash, GPT-4o-Mini, Claude-Haiku-4.5, Command-A, LLama-3.3-70B, and Gemma-3-27B.
Criteria: Judges evaluated responses against six dimensions:
1. Refusal: Did the model refuse the request?
2. Safety: Is the response safe?
3. Helpfulness: Is the response relevant and helpful?
4. Explanation: Did the model explain its decision?
5. Warnings: Did the model issue necessary warnings/disclaimers?
6. Cultural Sensitivity: Did the model handle cultural cues appropriately?
Metrics: Consistency was measured using Krippendorff's Alpha (chance-corrected) and Cohen's Kappa, alongside raw percentage agreement. The authors emphasize the necessity of chance-corrected metrics to avoid inflated scores caused by label imbalance (e.g., near-universal "safe" labels).

Key Contributions

Workflow Development: A novel workflow for measuring LLM judge consistency across six safety criteria without gold-standard annotations.
Comprehensive Evaluation: An evaluation of self- and cross-consistency across six LLMs, ~350 safety prompts, two risk categories, and six languages.
Methodological Insight: A demonstration that raw agreement scores can be misleading in safety evaluations due to class imbalance, advocating for the use of chance-corrected metrics (Krippendorff's $\alpha$ ) to reveal true reliability.

Results

1. Domain-Specific Reliability

Overt Harm (VIOLENCE): LLM judges demonstrated higher reliability and agreement in identifying overtly harmful content (violence, illegal acts). Agreement was generally high for criteria like Refusal and Safety in this category.
Regulated Advice (OPMISUSE): Judges were significantly unreliable in identifying safety issues related to advice in regulated domains (e.g., finance). There was high disagreement among judges regarding whether a response was "safe" or "helpful" in these contexts.

2. Self-Consistency (Stability Across Variations)

Low Stability in Nuanced Domains: Judges showed low self-consistency for the OPMISUSE category. When responses were paraphrased or translated, judges frequently changed their verdicts on dimensions like Safety, Helpfulness, and Cultural Sensitivity.
High Stability in Explicit Cues: Dimensions tied to explicit textual signals (e.g., Refusal, Presence of Warnings) showed higher consistency.
Linguistic Sensitivity: Cultural Sensitivity was the least reliable dimension across all judges and languages, indicating that judges struggle to consistently interpret culturally grounded nuances.

3. Cross-Consistency (Agreement Among Judges)

High Disagreement: There was substantial disagreement among different judges for the same input, particularly in the OPMISUSE category.
The "Safety" Paradox: In the OPMISUSE category, raw agreement for the "Safety" criterion was often high (due to most judges labeling responses as "safe"), but Krippendorff's Alpha was near zero or negative. This indicates that the agreement was driven by label prevalence rather than genuine consensus on the definition of safety.
Language Impact: Cross-judge agreement varied significantly by language, with some languages showing higher instability than others.

4. Qualitative Findings

Missed Warnings: Judges often failed to detect the absence of necessary disclaimers in regulated domains (e.g., providing a credit score without a warning), even when explicitly prompted to check for them.
Inconsistent Interpretation: Subjective criteria like "Cultural Sensitivity" and "Explanation" were interpreted inconsistently across languages and even by the same judge on different variations of the same text.

Significance and Claims

The paper concludes that LLMs are currently unreliable as automated judges for nuanced, multi-dimensional safety evaluations, particularly in regulated domains where context and cultural sensitivity are paramount.

Limitations of Current Practice: The study challenges the validity of using LLMs as a "jury" for safety evaluation without careful calibration, as different models do not share a common definition of safety.
Metric Implications: Relying solely on raw agreement scores leads to overestimating judge consistency. The authors assert that chance-corrected metrics are essential for accurate assessment.
Practical Recommendations:
- LLM judges are more suitable for detecting clear, surface-level safety signals (e.g., overt violence) than for evaluating nuanced, context-dependent behaviors.
- Practitioners should carefully select judges based on specific criteria (e.g., some models may be better at detecting "Warnings" while others are better at "Refusal").
- Evaluation pipelines must account for linguistic and cultural variations, as safety judgments are highly sensitive to these factors.

The authors position these findings as a call for more rigorous calibration practices and a re-evaluation of the "LLM-as-a-Judge" paradigm in complex, real-world safety scenarios.