🤖 AI

Decision-aware User Simulation Agent for Evaluating Conversational Recommender Systems

This paper proposes "Hesitator," a theory-grounded user simulation framework that explicitly models human decision-making under choice overload by separating item selection from commitment decisions, thereby correcting the unrealistic high acceptance rates and lack of hesitation often found in existing LLM-based conversational recommender system evaluators.

Original authors: Yuan-Chi Li, Li-Chi Chen, Sung-Yi Wu, Yu-Che Tsai, Shou-De Lin

Published 2026-05-08

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuan-Chi Li, Li-Chi Chen, Sung-Yi Wu, Yu-Che Tsai, Shou-De Lin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to buy a new pair of running shoes. You tell a salesperson, "I need high-quality shoes under $150." The salesperson (a computer program) instantly pulls up a list of 10 perfect options.

The Problem with Current "Fake" Shoppers
In the world of testing these sales programs, researchers use "User Simulators"—AI agents that pretend to be customers to see if the sales program works.

The paper argues that current AI shoppers are too perfect. They are like supercomputers with infinite brainpower. When shown 10 pairs of shoes, they instantly calculate the pros and cons of every single one, pick the best one, and say, "I'll take it!" immediately.

In the real world, humans aren't like that. If you see 10 pairs of shoes, your brain gets tired. You feel overwhelmed. You might think, "Wow, there are too many choices. I'm not sure which one is right. Let me think about this later." This is called choice overload. Because current AI simulators don't get tired or overwhelmed, they say "Yes" way too often, making the sales program look better than it actually is.

The Solution: "Hesitator"
The authors created a new AI shopper named Hesitator. Think of Hesitator not as a supercomputer, but as a realistic human with a busy brain.

Hesitator works in two distinct steps, mimicking how our brains actually handle decisions:

The Filter (The "No" Button):
First, Hesitator doesn't look at all 10 shoes at once. It uses a quick filter. "Do I have $150? Is it a good brand?" If a shoe fails this basic check, it's tossed out immediately. This is like a bouncer at a club; you don't let everyone in to dance. This saves mental energy.
The Hesitation (The "Wait" Button):
Even if the shoes pass the filter, Hesitator checks its own "mental battery." It asks: "How many options are left? How complicated are the details? Am I unsure about what I want?"
- If the list is short and simple, it says, "Okay, I'll buy this."
- If the list is still huge or confusing, Hesitator says, "I'm overwhelmed. I need to pause and think about this later."

Why This Matters
The paper tested Hesitator against other AI shoppers in two scenarios: buying electronics and buying video games.

The Old AI Shoppers: When the researchers gave them more options (making the choice harder), these AI shoppers actually got better at saying "Yes." They didn't care that the task was getting harder. This is unrealistic.
Hesitator: When given more options, Hesitator started saying "No" or "Wait" more often. Its behavior matched real human psychology: more choices = more confusion = fewer immediate purchases.

The Result
By adding this "Hesitation" step, the researchers found that the sales programs they were testing were actually less effective than previously thought. The old simulators were lying by being too eager. Hesitator tells the truth by getting overwhelmed, just like a real person would.

In Summary
The paper introduces a new way to simulate human shoppers that admits, "Sometimes, too many choices make me freeze." By building this hesitation into the AI, researchers can now test sales programs more accurately, ensuring they work for real humans who get tired of making decisions, not just for super-computers that never get overwhelmed.

Technical Summary: Decision-aware User Simulation Agent for Evaluating Conversational Recommender Systems

Problem Statement

Conversational Recommender Systems (CRS) increasingly rely on Large Language Model (LLM) based user simulators for automated evaluation. However, a critical limitation exists in current simulators: they often exhibit unrealistically strong information-processing capabilities inherited from their underlying LLMs. Unlike real human consumers, these agents rarely exhibit hesitation or decision deferral when faced with excessive options (choice overload). Consequently, existing simulators produce overly optimistic acceptance rates, obscuring the "intention–behavior frictions" that reduce real-world conversion rates. Current frameworks fail to model the cognitive constraints and decision deferral processes documented in psychological economics.

Methodology: The Hesitator Framework

The authors propose Hesitator, a theory-grounded user simulation framework designed to explicitly model human decision-making under choice overload. The framework is built around a modular Decision Module that separates utility-based item selection from overload-aware commitment decisions.

1. Core Architecture

The framework operates as a turn-based dialogue between a Sales Agent and a User Agent, governed by a global state $G = (P, S)$ representing user persona and shopping scenario. The Decision Module consists of two cognitively grounded components:

Selection Module (Utility Evaluation):
- Mechanism: Implements a two-stage heuristic-to-compensatory process based on Bettman et al. (1998).
- Stage 1 (Non-compensatory Filtering): Uses Elimination by Aspects (EBA) to prune candidates that violate hard constraints (e.g., budget).
- Stage 2 (Compensatory Evaluation): Applies a Weighted Additive (WADD) function to the remaining candidates to estimate utility.
- Thresholding: Compares the highest utility against a pickiness-dependent threshold. If the utility is insufficient, the recommendation is rejected immediately.
Hesitation Module (Commitment Decision):
- Mechanism: Models the probability of deferral based on perceived cognitive load, even if a satisfactory option exists.
- Overload Vector: Quantifies choice overload via a four-dimensional vector ( $v$ ) corresponding to: Assortment Size, Choice Set Complexity, Task Difficulty, and Preference Uncertainty.
- Meta-Analytic Calibration: Instead of relying on LLM priors, the module maps discrete LLM-perceived overload levels to continuous effect sizes using regression coefficients from a meta-analysis by Chernev et al. (2015).
- Probabilistic Output: Converts the aggregated overload effect size ( $d_{total}$ ) into an acceptance probability ( $P_{accept}$ ) using an arcsine-based proportion transformation. A purchase is triggered only if a random sample falls below this probability; otherwise, the agent defers.
Response Module: Converts the internal decision (accept, reject, or defer) into natural language responses via an LLM.

2. Experimental Setup

Domains: Electronics and Video Games (from Amazon Reviews 2023).
Baselines: Compared against three state-of-the-art simulators: PersonaLens, UserSimCRS, and RecUserSim.
Conditions: Evaluated under Low, Medium, and Severe cognitive overload conditions.
Metrics: Success Rate (SR), Subjective Dialogue Quality, and Believability (Hallucination Ratio).

Key Results

1. Mitigation of Unrealistic Behaviors

Under increasing cognitive overload (from Low to Severe), baseline simulators exhibited flat or even increasing success rates, contradicting human behavior.

Correction: Integrating the Decision Module (+DM) into these baselines caused success rates to decline significantly under severe overload, aligning with human cognitive patterns.
Example: In the Electronics domain, RecUserSim+DM dropped from 67.5% to 22.5% success rate under severe overload, whereas the vanilla version remained high.

2. Reproduction of Psychological Economics Phenomena

Hesitator successfully reproduced established behavioral patterns:

Inverted-U Curve: Success rates initially increased with information load but declined once cognitive load became excessive (Jacoby et al., 1974).
Attribute vs. Assortment Effects: Increasing attribute information improved performance (providing diagnostic cues), while increasing assortment size reduced performance (inducing decision conflict), consistent with Tversky and Shafir (1992).
Preference Uncertainty: Agents with lower preference uncertainty (higher expertise) showed greater resilience to information overload, exhibiting flatter post-peak slopes.

3. Conversational Quality and Believability

Quality: Hesitator outperformed baselines in subjective evaluations across dimensions like naturalness, clarity, and realism.
Believability: The addition of the Decision Module did not increase hallucination ratios; in fact, it slightly reduced them for most simulators (e.g., PersonaLens dropped from 0.1290 to 0.1088).

4. Ablation Study

Removing the structured Selection Module (replacing it with a standard rating prompt) led to unstable performance as attribute counts increased, failing to reproduce the expected positive effect of additional attributes. This confirms the necessity of the structured two-stage decision strategy.

Significance and Contributions

The paper claims the following contributions:

Identification of a Critical Gap: The authors identify that existing LLM-based simulators lack cognitive constraints, leading to inflated performance estimates for CRS due to unrealistically high acceptance rates under choice overload.
Plug-and-Play Decision Module: They propose a modular component that models intention–behavior friction, allowing simulated users to adapt commitment decisions based on environmental cognitive load.
Hesitator Framework: A complete simulation framework that integrates this module, successfully reproducing established behavioral patterns from decision-making research (e.g., choice overload, decision conflict).
Reliable Evaluation Benchmark: The framework provides a more realistic and reliable benchmark for evaluating CRS by ensuring that simulated user behavior reflects the hesitation and deferral observed in real-world consumer psychology.

The authors conclude that by grounding the agent's deferral behavior in large-scale empirical evidence (meta-analysis) rather than model priors, Hesitator offers a more faithful representation of human decision dynamics under cognitive constraints.