Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations

Technical Summary: Lost in Simulation

Problem Statement
Agentic benchmarks are increasingly relying on Large Language Model (LLM)-simulated users to evaluate AI agent performance at scale, replacing costly human evaluations. However, the robustness, validity, and fairness of this approach remain unexamined. Current practices often rely on a single user simulation model, treating users as a homogeneous group while overlooking linguistic and cultural diversity. This raises critical questions: Do simulated users reliably predict real-world agent performance? Do they introduce systematic biases that misrepresent agent capabilities across diverse populations?

Methodology
The authors conducted a comprehensive user study to evaluate LLM-simulated users as proxies for real humans using the $\tau$-Bench retail task suite.

• Participants: The study recruited $\sim$40 participants per group from the United States, India, Kenya, and Nigeria. US participants were stratified by dialect (Standard American English [SAE] vs. African American Vernacular English [AAVE]) and age (18–34, 35–54, 55+). Non-US participants were recruited from the 18–34 age group.
• Task Design: From the 115 retail tasks in $\tau$-Bench, the authors selected a difficulty-balanced subset of 18 tasks (3 tasks per difficulty level, defined by a model-based success rate metric).
• Experimental Setup: Participants interacted with a fixed agent model (GPT-4o) via a chat interface to complete tasks. These interactions were compared against interactions where the "user" was simulated by various LLMs (GPT-4o, Sonnet 3.7, Sonnet 4.5, Kimi-K2-Thinking).
• Evaluation Metrics:
  • Success Rate: The percentage of tasks where the agent correctly executed required actions and conveyed necessary information.
  • Expected Calibration Error (ECE): Adapted from probabilistic classifiers to measure the weighted absolute deviation between agent success rates with simulated users versus real human users across difficulty levels.
  • Interaction Analysis: Quantitative analysis of conversational structure (turns, questions, politeness markers) and qualitative error attribution (argument errors, missing actions, output errors).

Key Contributions and Results

1. Lack of Robustness:
   Agent success rates vary significantly depending on the choice of the user simulation model. While some models cluster around 67–71% success, switching from Sonnet 3.7 to Sonnet 4.5 as the user model resulted in a 9 percentage point difference in success rates. This indicates that single-model simulations are unreliable for establishing robust performance baselines.

2. Systematic Miscalibration (Validity):
   Simulated users do not serve as reliable proxies for real humans. Even in the US context (where LLMs are often assumed to be best calibrated), the Expected Calibration Error (ECE) was 15.1.

   • Task Difficulty Bias: Simulated users systematically underestimate agent performance on the most challenging tasks (0% difficulty bin) and overestimate performance on moderately difficult tasks (60% difficulty bin).
   • Conversational Artifacts: Simulated users exhibit distinct behavioral patterns compared to humans, including significantly higher rates of question-asking (18.8% vs. 9.8% for humans) and politeness markers (39.2% vs. 19.9%).

3. Demographic Disparities (Fairness):
   The reliability of user simulation varies drastically across demographic groups, revealing significant fairness issues:

   • Dialect Bias: Agents achieved a 50.6% success rate with SAE speakers but only 39.4% with AAVE speakers. The calibration error (ECE) was nearly double for AAVE speakers (20.3) compared to SAE speakers (11.7).
   • Age Interaction: Disparities compound with age. The performance gap between SAE and AAVE speakers widened from ~12 percentage points in the 35–54 age group to 19 percentage points in the 55+ group.
   • Geographic Bias: Simulated users were least effective proxies for AAVE and Indian English speakers (ECE $\approx$ 18.9), while performing best for SAE speakers (ECE = 13.0).

4. Divergent Failure Modes:
   The nature of errors differs between simulated and human interactions.

   • Error Attribution: In human conversations, users are the primary source of failure (62.2%), often due to ambiguity or partial compliance. In simulated conversations, the agent is responsible for the majority of failures (48.9%).
   • Error Types: Agents interacting with simulated users make more frequent output errors (omitting or including incorrect outputs) but fewer missing or extra action errors. This suggests simulated users may follow instructions too precisely, failing to surface the ambiguity and misunderstanding that real users introduce.

Significance and Claims
The paper argues that current evaluation practices relying on LLM-simulated users risk misrepresenting agent capabilities across diverse user populations. By optimizing for simulated users, developers may inadvertently create systems that appear robust in benchmarks but fail disproportionately for real-world users, particularly those speaking AAVE or from non-Western backgrounds. The authors conclude that user simulation cannot yet serve as a standalone evaluation paradigm; instead, agentic benchmarks must validate simulated outcomes against demographically diverse human data and report results across multiple simulation models to ensure robustness and fairness.