Technical Summary: DRIP-R

Problem Statement

While Large Language Model (LLM) agents are increasingly deployed in high-stakes domains like retail, healthcare, and finance, existing benchmarks fail to evaluate their performance under real-world policy ambiguity. Current agent evaluations typically rely on "cleanly specified," narrow, and unambiguous policies designed specifically for the benchmark. However, real-world domain policies (e.g., Amazon's return policy) are often incomplete, contain implicit assumptions, and admit multiple valid interpretations.

This gap creates a critical evaluation blind spot: when legitimate values conflict (e.g., customer satisfaction vs. regulatory compliance), no single "correct" resolution exists. Human organizations navigate this through institutional safeguards (discretion, escalation, precedent), but LLM agents may lack comparable mechanisms, potentially exploiting loopholes or producing unintended outcomes while appearing policy-compliant. The paper posits that ambiguity poses a systematic challenge to LLM decision-making that current benchmarks do not capture.

Methodology

Benchmark Construction (DRIP-R)

The authors introduce DRIP-R (Decision-making and Reasoning In ambiguous Policy for Retail), a benchmark grounded in Amazon's publicly available return policy. The construction process involves:

1. Ambiguity Extraction: Using LLMs (GPT-4.1) and human experts to identify and verify policy ambiguities based on a taxonomy including vagueness, semantic ambiguity, referential ambiguity, incompleteness, and coordination ambiguity.
2. Task Generation: Creating 40 unique "Order Return" tasks where resolution depends on at least one ambiguous policy clause. Each task involves specific product details, timelines, and customer circumstances.
3. Persona Integration:
   • User Personas: 10 distinct personas derived from the Persona-Hub dataset, augmented with Big Five personality traits, communication styles (direct/indirect, rapport/report), and language styles.
   • Agent Personas: 5 distinct agent personas (Direct, Fair, Agreeable, Helpful, Very Helpful) based on the Big Five framework to test how agent disposition influences outcomes.
4. Simulation Environment: A full-duplex conversational simulation modeled as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP). The agent interacts with a simulated user, utilizing tool-calling capabilities (e.g., get_order_details) to gather facts before making decisions. The conversation ends upon resolution, user abort, or reaching a turn limit (20 turns).

Evaluation Framework

DRIP-R replaces binary success metrics with a multi-dimensional evaluation framework using an "LLM-as-a-Judge" approach. Five parallel judges (using GPT-5) evaluate conversations across four primary dimensions, each with sub-metrics:

1. Policy Adherence: Measures if the agent's messages and final resolution are grounded in the provided policy text (justifiable, not necessarily "correct"). Includes per-turn justification and resolution outcome support.
2. Dialogue Quality: Assesses conversational competence, including consistency, logical progression toward resolution, and verbosity.
3. Behavioral Alignment: Evaluates if agents and users act consistently with their assigned personas and implicit values (based on Schwartz value theory: Conformity, Benevolence, Self-Direction, Security, Universalism, Power).
4. Task-Resolution & Interest Alignment:
   • Task-Resolution: Does the outcome address the specific user request and constraints?
   • Interest Alignment: Measures the balance between customer goal satisfaction and company interest protection.

Calibration: The judges are calibrated against human annotations (3 experts) using Distribution Preserving Few-Shot sampling (DPFS) and G-Eval to minimize bias and maximize adjacent agreement (agreement within one point on a 5-point scale).

Key Contributions

1. DRIP-R Benchmark: The first open-source benchmark specifically designed to evaluate LLM agents in realistic, policy-ambiguous retail return scenarios. It utilizes real-world, multi-document policies with high linguistic complexity (Flesch-Kincaid grade level ~11.89), contrasting with prior benchmarks that use simplified, single-document rules.
2. Multidimensional Evaluation Infrastructure: A novel simulation and evaluation pipeline that assesses agents not just on task completion, but on policy adherence, behavioral alignment with personas, and the trade-offs between customer and company interests.
3. Empirical Analysis of Ambiguity: A systematic study revealing how different LLMs reason, interact, and make decisions under ambiguity, highlighting that ambiguity is a genuine, systematic challenge.

Results

The authors evaluated four models (two open-source: gpt-oss-120b, Qwen3-35B-A3B; two API-only: gpt-5, gpt-4.1) across 400 scenarios (40 tasks × 10 personas).

• Fundamental Disagreement: Frontier models fundamentally disagree on identical ambiguous scenarios. Only 4.2% of scenarios resulted in full agreement (all models converging on the same resolution), while 13.3% resulted in full disagreement (every model producing a distinct outcome).
• Policy Adherence Trajectories: Policy adherence is generally high early in conversations but diverges as the conversation progresses. Models like Qwen3-35B and gpt-oss-120b showed significant divergence between their stated reasoning and final resolutions, particularly in refund-granting outcomes. GPT-5 maintained uniformly high adherence regardless of the outcome.
• Impact of Personas:
  • User Personas: Highly extraverted and neurotic user personas were associated with outcomes favoring customer goals over company interests. More agreeable personas correlated with higher company interest alignment.
  • Agent Personas: Agent personas reliably anchored outcomes. DIRECT agents consistently yielded the highest denial rates, while VERY_HELPFUL agents yielded the lowest. However, the specific alternative to denial (e.g., full refund vs. escalation) varied by model.
• Ambiguity Types: Different ambiguity types (e.g., Coordination Ambiguity) systematically steer agents toward different strategies. Coordination Ambiguity was found to be particularly difficult for agents to resolve consistently with their stated reasoning.

Significance and Claims

The paper claims that DRIP-R establishes a foundation for building accountable AI systems that operate under the inherent ambiguity of real-world policies. By demonstrating that current frontier models lack consensus on ambiguous tasks, the authors argue that ambiguity is not a minor edge case but a central difficulty in agentic deployment.

The work highlights that:

• Current benchmarks abstract away the "central difficulty" of real-world deployment.
• Ambiguity leads to systematic challenges in decision-making, where agents may produce technically valid but unintended outcomes.
• Evaluating agents requires looking beyond binary success to understand how they interpret ambiguity, justify decisions, and balance competing stakeholder interests.

The authors modestly note limitations, including the current focus on the retail domain (though the framework is generalizable) and the reliance on a primary policy document. They identify expanding the benchmark to other high-stakes domains (airline, healthcare, legal) and testing on a broader set of models as direct avenues for future work.