Technical Summary: SafePyramid

Problem Statement

Current Large Language Model (LLM) guardrails predominantly rely on predefined risk taxonomies (e.g., fixed categories like "Privacy," "Fraud," or "Sexual Content") to classify user-model interactions. While effective for general safety, this paradigm lacks the flexibility required for real-world applications where safety standards vary significantly across organizations, jurisdictions, and specific use cases (e.g., a short-video platform's youth safety policy vs. a gaming forum's anti-cheating rules).

The emerging setting of in-context policy guardrailing addresses this by allowing guardrails to execute safety policies provided explicitly in the context at inference time, rather than relying on static, pretrained risk priors. However, evaluating this capability remains an unexplored challenge. Existing benchmarks either focus on general instruction following, fixed-taxonomy safety, or lack the granularity to diagnose specific failures in policy execution. There is no systematic framework to evaluate whether guardrails can:

1. Reliably execute arbitrary safety policies provided in context.
2. Resolve complex dependencies between rules (e.g., exceptions and conditions).
3. Adapt to novel policy frameworks that introduce new concepts and terminology.

Methodology: SafePyramid

To address these gaps, the authors introduce SafePyramid, a hierarchical benchmark designed to systematically evaluate in-context policy guardrailing. The benchmark is constructed using a rigorous multi-stage pipeline involving frontier LLMs (Grok-4.1, GPT-5.4, Gemini-3.1-Pro, Claude-Opus-4.6) for generation, followed by cross-model validation and human expert review.

Dataset Composition

SafePyramid comprises:

• 1,000 multi-turn conversations across 10 safety domains (Academic Integrity, Content Moderation, Critical Infrastructure, Defamation, Discrimination, Fraud, Intellectual Property, Privacy, Sexual Content, Specialized Advice).
• 3,000 application-specific policies, totaling 61,699 distinct natural-language rules.
• Three Difficulty Levels (Hierarchical Design):
  • L0 (Understanding Individual Rules): Evaluates the ability to determine if a single rule is violated. It includes Decisive Rules (clearly triggered or not) and Distractor Rules (topically related but not triggered).
  • L1 (Resolving Rule Dependencies): Extends L0 by introducing Exception Rules (which waive base rules under specific conditions) and Conditional Rules (which tighten base rules or create new violations under specific conditions). This level tests the model's ability to handle interdependent logic.
  • L2 (Adapting to Novel Policy Frameworks): Extends L1 by rewriting the underlying policy structure using fully fictional concepts and terminology (inspired by CL-Bench). This forces models to learn and apply the policy framework from scratch without relying on pretrained safety priors or real-world legal knowledge.

Evaluation Protocols

The benchmark supports two evaluation modes:

1. Per-Policy Evaluation: The model reads the full policy and conversation to predict the complete set of violated rules. This is the primary setting for frontier LLMs.
2. Per-Rule Evaluation: The model evaluates one target rule at a time, aggregating binary decisions. This is designed for fine-tuned guard models with fixed output interfaces.

Metrics:

• Rule Matching Rate (RMR): Measures the percentage of cases where the predicted violated-rule set matches the ground truth within a tolerance threshold ($\tau$). The main metric is the average RMR over $\tau \in \{0.7, 0.8, 0.9, 1.0\}$.
• Rule Disagreement Rate (RDR): A micro-averaged metric quantifying the total number of rule-level errors (false positives and false negatives) relative to the total number of rules involved.

Key Results

The authors evaluated 10 frontier LLMs and 5 policy-configurable guard models on SafePyramid. The results highlight significant limitations in current systems:

• Overall Performance: Even the best-performing model, GPT-5.5, achieves an exact match (RMR@1.0) on only 54.0% of L0 cases, 35.3% of L1 cases, and 12.9% of L2 cases.
• Difficulty Progression: There is a monotonic performance drop from L0 to L2 across all models. The drop from L0 to L1 indicates that handling rule dependencies (exceptions/conditions) is a major bottleneck. The further drop to L2 suggests that adapting to novel frameworks is an even more severe challenge.
• Guard Model Limitations: Policy-configurable guard models (e.g., GPT-OSS-Safeguard) perform competitively on L0 but their performance collapses on L1 and L2, suggesting they are not yet designed for full-policy execution with complex dependencies.
• Error Analysis:
  • L0 Errors: Dominated by Decisive Rules, often due to "surface-cue matching" where models fail to distinguish between a chatbot endorsing a behavior versus attributing a view to a third party.
  • L1 Errors: Shift to Exception Rules, where models frequently over-trigger exceptions based on partial keyword matches without verifying all activation conditions.
  • L2 Errors: Increasingly involve Conditional Rules. Models struggle to distinguish between the conditional rule itself (a modifier) and the base rule it modifies, often treating the conditional as a violation in itself when novel frameworks are introduced.
• Mitigation Strategies:
  • Per-Rule Evaluation: Decomposing the task into per-rule classification significantly improves performance for smaller guard models (e.g., GPT-OSS-Safeguard-120B improved from 23.6% to 52.4% average RMR), indicating that full-policy composition is a bottleneck.
  • Agentic Harnesses: Using agentic workflows (e.g., Codex, Claude Code) to decompose the task and verify intermediate steps improves performance, particularly on L1 and L2, suggesting that explicit reasoning and verification help with dependency resolution.
  • Reasoning Effort: Increasing reasoning effort (e.g., "xhigh" effort in GPT-5.5) yields marginal gains on L0 but substantial improvements on L1 and L2, confirming that complex dependency resolution requires more computational budget.

Significance and Contributions

The paper makes four primary contributions:

1. Problem Formulation: It formally defines in-context policy guardrailing as a task where guardrails must identify safety violations using policy specifications provided in context, moving beyond fixed risk taxonomies.
2. Capability Identification: It identifies and isolates three critical capabilities required for this setting: individual rule understanding, rule dependency resolution, and novel policy framework adaptation.
3. Benchmark Introduction: It introduces SafePyramid, a comprehensive, three-level hierarchical benchmark containing 1,000 conversations, 3,000 policies, and 61,699 rules, constructed with rigorous quality control.
4. Empirical Evidence: It provides extensive evaluation showing that current state-of-the-art models are far from reliable in-context policy guardrails, particularly when faced with rule dependencies and novel frameworks.

The authors conclude that the field must move beyond simple risk classification toward explicit, auditable, and adaptable policy execution. The findings suggest that stronger guardrails will likely require better policy decomposition, context organization, and rule-level verification mechanisms, potentially leveraging agentic workflows. SafePyramid serves as a diagnostic tool to track progress in these areas.