What Benchmarks Don't Measure: The Case for Evaluating Abstention Competence in Autonomous Agents

Technical Summary: What Benchmarks Don't Measure: The Case for Evaluating Abstention Competence in Autonomous Agents

Problem Statement
Current evaluation paradigms for autonomous agents are systematically blind to a critical failure mode: proceeding when an agent should have paused. Dominant benchmarks measure task success (completion rates, step counts, answer accuracy) but lack scenarios where the correct ground-truth behavior is to abstain. Consequently, agents trained under human-feedback objectives develop a structural "compliance bias"—a tendency to proceed even when lacking necessary inputs, evidence, or authorization. This bias is entrenched because benchmark scoring regimes treat proceeding as the default correct action, often penalizing agents for pausing (e.g., via fixed step budgets) or failing to distinguish a principled pause from a silent failure. The paper argues that this evaluation gap prevents the development of agents capable of "informed abstention," leading to irreversible, hazardous state changes in enterprise environments.

Methodology
The authors propose a framework to evaluate and enforce "informed abstention," defined as a structured pause triggered by specific unmet conditions rather than abstract confidence.

1. Taxonomy of Abstention-Warranted Scenarios: The paper introduces a three-gap taxonomy to categorize when an agent should pause:

   • Specification Gap: Required information is absent (defined by tool schema, not model confidence). The appropriate response is targeted clarification.
   • Verification Gap: The agent cannot confirm a necessary world state before acting. The appropriate response is a bounded verification step.
   • Authority Gap: The action requires explicit authorization for a binding commitment or high-impact change. The appropriate response is a commitment pause requesting approval.

2. Evaluation Protocols: To measure abstention competence, the authors propose three composite metrics:

   • Safety Rate (SR): The fraction of hazardous scenarios where the agent produces a blocking response.
   • Usability Rate (UR): The fraction of safe scenarios where the agent completes the task without unnecessary friction.
   • Informed Refusal Rate (IRR): The fraction of blocking responses that explicitly name the unmet precondition, distinguishing calibrated safety from vague over-refusal.

3. Experimental Setup:

   • Dataset: 144 enterprise agent scenarios derived from 24 human-authored seeds, augmented to include 120 hazardous scenarios (categorized by the three gaps) and 24 matched safe controls.
   • Models: Seven model families (GPT-4o, GPT-5.4-mini, Llama 3.1 8B, Claude Sonnet 4.6, Claude Opus 4.6, Gemini 2.5 Pro, Gemini 2.5 Flash) using a ReAct architecture.
   • Conditions: Three evaluation conditions were tested:
     • Baseline: No abstention policy.
     • Prompt-Only: Safety instructions via system prompt.
     • Checkpoint: Runtime enforcement via a pre-execution hook (InformedAbstentionToolWrapper) that performs schema checks, state polling, and authorization verification using a separate guard model.

Key Results
The preliminary results across 144 scenarios and five model families reveal significant findings regarding the safety-usability tradeoff:

• Universal Compliance Bias: All models exhibited compliance bias at baseline, but failure modes varied. Some models (e.g., Llama 3.1 8B) defaulted to "proceed" (high Usability, low Safety), while others (e.g., Claude Opus 4.6) defaulted to "refuse" (high Safety, near-zero Usability).
• The Prompt-Only Cliff: Relying solely on system prompts to enforce safety proved unreliable. For heavily safety-aligned models like Claude Opus 4.6, prompt-only policies caused a "usability cliff," dropping Usability Rate to near 0% as models interpreted safety instructions as absolute overrides of user intent. Conversely, some models showed improved usability with prompts, highlighting that the effect is model-dependent and unpredictable.
• Runtime Enforcement Efficacy: The Checkpoint condition (runtime enforcement) achieved the highest performance across all models.
  • Safety: All models converged to a Safety Rate between 88% and 91%, suggesting a structural ceiling where agents refuse natively without invoking tools.
  • Usability: Runtime enforcement dramatically recovered usability for models that previously over-refused (e.g., +87.5 percentage points for GPT-4o and +70.9 for Claude Opus 4.6 compared to prompt-only).
  • Informed Refusal: The Checkpoint architecture guaranteed 100% IRR by design, as the wrapper always emitted a structured explanation for blocks, whereas baseline and prompt-only models relied on estimated LLM judges and showed lower IRR.
• Verification Gap Difficulty: The Verification Gap proved the most challenging scenario type across all models at baseline, with the largest gains observed under the Checkpoint condition, supporting the hypothesis that self-knowledge degrades under distribution shift.

Significance and Claims
The paper positions these results as preliminary work intended to spark further conversation within the benchmark community. Its primary contributions are:

1. Theoretical Account: Identifying that the inability to measure abstention competence is a direct result of reward hacking in human-feedback pipelines and the design of current benchmarks.
2. Taxonomy and Metrics: Providing a principled vocabulary (the three-gap taxonomy) and a composite metric suite (SR, UR, IRR) to make abstention visible and evaluable.
3. Feasibility Demonstration: Showing that the safety-usability tradeoff is not inherent but tunable, and that runtime enforcement can resolve the "over-refusal" and "under-refusal" failure modes simultaneously.

The authors conclude that benchmarks must evolve to include scenarios where pausing is the correct ground truth. Without such changes, training incentives will continue to reinforce compliance bias, preventing the deployment of agents that can reliably stop when they should. The proposed tools are offered as a starting point for stress-testing against wider agent architectures and deployment domains.