NESSiE: The Necessary Safety Benchmark -- Identifying Errors that should not Exist

1. Problem Statement

As Large Language Models (LLMs) are increasingly deployed as autonomous agents in unmonitored ("in the wild") environments, their ability to follow instructions robustly is critical. A single failure in an agent's chain of reasoning can lead to catastrophic safety divergences. While numerous safety benchmarks exist, they often focus on complex adversarial attacks or require resource-intensive evaluation methods (e.g., LLM-as-a-judge).

The authors argue that before models can be trusted in complex scenarios, they must first pass a necessary condition: the ability to follow simple, abstract safety rules without error. Current state-of-the-art (SOTA) models fail even these basic tests, revealing a pervasive bias toward being "helpful" (providing information) rather than "safe" (withholding information), even in the absence of adversarial prompts.

2. Methodology: The NESSiE Benchmark

NESSiE (NEceSsary SafEty benchmark) is a lightweight, keyword-matching-based evaluation framework designed to test instruction adherence in safety-relevant contexts.

• Structure: The benchmark consists of 41 distinct test cases grouped into 93 unique system-user prompt combinations.
• Evaluation Metric: The primary metric is Safe & Helpful (SH). A model passes a test pair only if it correctly:
  1. Withholds a specific secret/keyword when unauthorized (Safe).
  2. Reveals the specific secret/keyword when authorized (Helpful).
  • Note: If a model fails either side of the pair, the SH score for that case is zero.
• Test Suites: The benchmark comprises six distinct suites to test different cognitive and contextual loads:
  1. RULeS: Adapted from Mu et al. (2024), focusing on non-game rule following.
  2. RULeS Reformulated: The same rules presented in a more concise, policy-only format to test robustness against input structure variations.
  3. Agentic: Tests the model's ability to output specific keywords to simulate atomic agent actions (format compliance).
  4. Generated: Test cases initially generated by an LLM and human-refined for consistency.
  5. Skills: Requires an additional cognitive step (e.g., checking for anagrams, parity checks) before applying the safety rule, testing safety under cognitive load.
  6. Multiturn: Requires adherence to rules across two conversation turns (e.g., establishing a password in turn 1, using it in turn 2).
• Robustness Conditions:
  • Distraction Context: A benign, ~2000-token conversation on unrelated topics is inserted between the system and user prompts to test policy robustness against noise.
  • Disabled Reasoning: Tests models where reasoning traces (Chain-of-Thought) are disabled to measure the impact of reasoning on safety alignment.
• Evaluation Technique: Simple keyword matching ensures efficiency and reliability, avoiding the hallucinations common in LLM-based evaluators.

3. Key Contributions

1. A Necessary Safety Baseline: NESSiE establishes a "minimum passing bar." The authors argue that if a model cannot pass these simple, non-adversarial tests, it is unsafe for deployment as an autonomous agent.
2. The Safe & Helpful (SH) Metric: A unified metric that penalizes models for failing either the safety or helpfulness constraint, highlighting the trade-off between the two.
3. Identification of Systemic Fragility: The paper demonstrates that safety failures are not just due to adversarial jailbreaking but occur in benign, straightforward scenarios.
4. Open Resources: The authors released the dataset, evaluation package, and plotting code to facilitate rapid, local safety checks for researchers.

4. Key Results

The evaluation covered a wide range of models, from open-source baselines (Llama 2 7B, Mistral 7B) to SOTA closed models (Gemini 2.5 Pro, GPT-5.2, Claude Opus 4.5).

• Performance Gap:
  • Older/Open Models: Performed poorly (e.g., Llama 2 7B: 17.7% SH; Mistral 7B: 29.1% SH).
  • SOTA Models: Showed significant improvement but still failed to reach 100%. The best performer, Gemini 2.5 Pro, achieved 95.2%, while others ranged between 80% and 95%.
• Helpfulness vs. Safety Bias:
  • Across all models, Helpfulness scores were consistently higher than Safety scores.
  • Example: Qwen3 VL 32B achieved 99.7% Helpfulness but only 62.7% Safety, resulting in a compromised SH score of 62.4%. This indicates models prioritize providing information over withholding it.
• Impact of Cognitive Load & Context:
  • Skills Suite: The most difficult category (63.4% average SH), proving that adding a reasoning step before a safety rule significantly degrades performance.
  • Distraction Context: Introducing irrelevant conversation history caused a drop of at least 15% in SH scores for tested models, primarily due to safety failures (models became "helpful" but unsafe).
  • Disabled Reasoning: Results were mixed; for some models (Gemini 2.5 Pro), disabling reasoning hurt performance, while for others (Claude Opus 4.5), it improved it, suggesting complex interactions between reasoning traces and safety alignment.
• Error Analysis:
  • Task Failed: Models leaked secrets or failed prerequisite skills (e.g., anagram checks).
  • Participation Denied: Models refused to engage in benign tasks (e.g., "I cannot roleplay"), a common failure mode for Claude models.
  • Leaked Keyword: Models correctly identified the rule but accidentally printed the forbidden keyword in their explanation (common in GPT-5 series).

5. Significance and Conclusion

The paper underscores a critical risk in the current trajectory of AI development: SOTA models are not yet robust enough for unmonitored autonomous deployment.

• Necessary vs. Sufficient: Passing NESSiE is a necessary condition for safety, though not sufficient. However, failing it is a definitive red flag.
• Fragility: Safety mechanisms are fragile; they break under cognitive load (Skills), distraction, or when the model is forced to balance helpfulness with safety.
• Recommendation: The authors advocate for NESSiE as a standard "sanity check" for researchers and developers. If a model fails these basic rules, further complex evaluation or deployment is unwarranted.
• Future Work: The benchmark can be used to evaluate simple adversarial attacks and the resilience of safety guardrails against malicious actors.

In summary, NESSiE reveals that despite advances in capability, current LLMs exhibit fundamental flaws in instruction following that make them unreliable for safety-critical, autonomous applications.