Technical Summary: Safety is Contextual, LLM-Judges Are Not

Problem Statement

Large Language Models (LLMs) acting as judges are the de facto standard for evaluating safety at scale, as safety assessments often lack a single ground-truth answer. However, current evaluations of these judges rely heavily on human agreement metrics within static benchmarks. This approach fails to capture critical properties required for real-world deployment, where safety definitions vary across cultures, domains, and time. Specifically, it remains unclear whether LLM-judges can:

1. Utilize in-context information: Can they learn from demonstrations or adapt to new semantic knowledge (e.g., slang, recent events) provided in the prompt?
2. Adhere to specific safety policies: Can they be steered to follow custom or divergent safety definitions, or do they rigidly rely on internal priors established during pre-training and post-training?

The paper argues that without understanding these properties, practitioners may deploy judges that appear accurate in static benchmarks but fail to adapt to specific enterprise policies, cultural nuances, or temporal shifts in language and threats.

Methodology

The authors conducted a meta-evaluation of 13 diverse LLM-judges (spanning open/closed-source, general-purpose, and safety-specific models) across five languages (English, French, Japanese, Arabic, Korean). The study utilized three datasets:

• MultilingualPrompts: An internal dataset of 779 prompts requiring niche regional knowledge.
• NovelPrompts: An English dataset of 194 prompts containing concepts post-dating the models' training cutoffs (post-July 2025).
• Sorry-BENCH: A public benchmark of 1,000 samples used for robustness checks.

The evaluation focused on two core properties:

1. Susceptibility: The likelihood of a judge changing its prediction when provided with in-context information. This was tested via:
   • Demonstrations: Providing 2–4 examples (correct, incorrect, or none) to see if judges learn task format or label space.
   • Novel Context: Providing explanations for niche terms or post-cutoff concepts to see if judges incorporate new semantic knowledge.
   • Irrelevant Context: Providing shuffled or unrelated information to test robustness.
2. Steerability: The likelihood of a judge changing its prediction when given a modified safety policy. This was tested via:
   • Policy Variants: Modifying severity thresholds for specific categories (e.g., making self-harm safe and sexual content unsafe).
   • Absurd Policies: Defining ball sports as "unsafe" to test if judges can override safety priors for orthogonal tasks.
   • Task Reframing: Masking the safety task as an arbitrary classification task (Category A vs. B) to isolate the impact of the "safety" concept on rigidity.

Performance was measured using accuracy, F1 scores, and prediction flip rates (the proportion of samples where the judge's output changed between conditions).

Key Results

1. Susceptibility to In-Context Information

• Demonstrations are largely ineffective: Providing demonstrations (even misleading ones) had a marginal or inconsistent impact on performance. Judges rarely changed their predictions based on examples, suggesting they rely on internal priors rather than learning from the few-shot context.
• Novel context bridges knowledge gaps: Judges showed significant susceptibility to novel semantic information. When provided with context explaining post-cutoff concepts (in NovelPrompts), F1 scores improved by an average of 10%. This effect was most pronounced for samples with low corpus frequency (low prior knowledge).
• Prior knowledge dictates susceptibility: Judges are significantly more likely to change predictions on samples where they have weak priors (low word frequency in training data). Conversely, they remain rigid on samples they "know," even when presented with contradictory context.
• Inherent variability: Susceptibility is a model-specific property. Some models (e.g., Tiny-Aya) are highly susceptible to both helpful and distracting context, while others are robust.

2. Steerability to Safety Policies

• Judges struggle to adapt to new policies: When presented with safety definitions that contradict their internal priors (e.g., reclassifying self-harm as safe), judges failed to adapt. They changed only ~5% of predictions when >15% were required to align with the new policy, resulting in significant accuracy drops relative to the new ground truth.
• Absurd policies reveal rigidity: While judges could easily follow an absurd policy (ball sports = unsafe) when the task was orthogonal to safety, they struggled when the new policy conflicted with their learned safety boundaries.
• Task framing matters: Steerability increased significantly (by over 2x) when the safety evaluation task was reframed as an arbitrary classification task (A vs. B) without the word "safety." This suggests the rigidity stems from the model's internal safety boundary rather than an inability to understand instructions.
• Steerability is model-dependent: Like susceptibility, steerability varies by model but does not correlate with human agreement accuracy or susceptibility.

Significance and Claims

The paper claims that current evaluation practices, which focus solely on human agreement in static, aligned settings, provide a "false sense of security." The authors introduce susceptibility and steerability as essential, previously overlooked properties for evaluating safety judges.

Key claims include:

• Safety is contextual, judges are not: While safety policies vary by culture, domain, and time, LLM-judges generally do not adapt to these variations. They tend to apply the safety boundaries they were extensively post-trained on, regardless of the instructions provided in the prompt.
• Contextual learning is conditional: Judges can learn from new information, but only when their prior knowledge on the topic is weak. They are robust to misleading demonstrations and irrelevant context.
• Selection criteria must change: Practitioners cannot rely on a single "best" judge based on accuracy. Instead, the choice of judge should be driven by the deployment scenario:
  • For tasks requiring adaptation to novel information (e.g., real-time misinformation), a susceptible judge is preferred.
  • For tasks requiring adherence to specific enterprise or cultural policies, a steerable judge is required.
  • If the base model's internal policy is trusted, human agreement metrics suffice, but instructions and demonstrations may be ignored.

The authors conclude that the community must move beyond static benchmarks to comprehensively evaluate the combined properties of susceptibility, steerability, and accuracy to ensure safety systems are robust and adaptable in the real world.