Technical Summary: Simulated Customers Never Walk Away

Problem Statement

The field of conversational AI has increasingly relied on LLM-as-user-simulation as core infrastructure for benchmarking (e.g., $\tau$-bench), training pipelines, and fidelity studies. Existing validation frameworks primarily measure communicative fidelity: whether a simulator sounds like a human in terms of style, persona consistency, and emotional reaction. These frameworks typically validate against ground truth generated by paid participants role-playing assigned goals (e.g., "book this flight").

The paper identifies a structural blind spot in this paradigm: when a user's goal is assigned, their willingness to act is exogenous. Consequently, existing benchmarks cannot test whether simulators reproduce decision fidelity—the endogenous, latent, and decaying willingness of a real person deciding whether to commit to a consequential choice. Real customers may lose interest, stall, or disengage, but a role-player told to buy a flight never genuinely decides not to. The paper argues that for persuasion agents (sales, fundraising, negotiation), decision fidelity is the critical property determining whether simulator-based conclusions transfer to reality, yet it remains unmeasured because prior benchmarks lack real outcomes.

Methodology

The authors introduce a teacher-forced probe protocol to measure decision fidelity against real, consequential outcomes, utilizing a unique testbed: ZhenaiSales.

1. Testbed: ZhenaiSales

• Data: 2,790 production conversations between a deployed LLM sales agent and real parents of a Chinese relationship-matchmaking service.
• Ground Truth: 793 conversations resulted in verified payments (buyers); 1,997 did not (non-buyers).
• Key Feature: Unlike role-play corpora, these users have real, endogenous willingness and their decisions involve real financial cost. Conversations are truncated at the first payment timestamp to prevent outcome leakage.

2. The Protocol

To isolate the user model from the agent and control for confounds, the authors employ a teacher-forced approach:

• Fixed Context: At specific probe points (30%, 60%, 90% of conversation length), the simulator is fed the identical real conversation prefix ($h_t$) and the user's demographic profile.
• Fixed Instrument: An LLM perceiver ($\Phi$) maps both the real user's next turn and the simulator's generated turn to a structured decision-state schema (engagement stage, emotion, blocker).
• Measurement: The study compares the distribution of decision states between real and simulated turns, specifically conditioning on the user's eventual real outcome (buyer vs. non-buyer).

3. Metric: Outcome-Conditioned Depth Bias

The authors define a scalar endpoint, $\Delta$, to measure the "disengagement deficit":

• Depth ($\delta$): An ordinal scale from resisting (0) to deciding (4).
• Bias ($D$): The difference between simulated depth and real depth for a given conversation.
• Contrast ($\Delta$): The difference in mean bias between the non-buyer stratum and the buyer stratum ($\Delta = E_{y=0}[\bar{D}] - E_{y=1}[\bar{D}]$).
• Hypothesis: Perfect decision fidelity implies $\Delta = 0$. A positive $\Delta$ indicates the simulator inflates engagement depth for non-buyers more than for buyers.

Key Contributions

1. Definition of Decision Fidelity: The paper articulates decision fidelity as a distinct, measurable property of user simulators, arguing it is the necessary condition for transferring evaluation results to reality, unlike communicative fidelity.
2. New Testbed and Protocol: Introduction of ZhenaiSales, the first user-simulation testbed grounded in verified purchase decisions, and a teacher-forced, instrument-controlled protocol for measuring decision fidelity.
3. Discovery of the Disengagement Deficit: Documentation of a systematic, outcome-correlated failure where LLM simulators reproduce eventual buyers accurately but systematically inflate eventual non-buyers toward the purchase frame.

Results

The study reveals a significant disengagement deficit across multiple model families and experimental conditions.

1. Primary Finding: The Disengagement Deficit

• Buyers: Simulators reproduce eventual buyers almost exactly (mean depth bias $\approx +0.09$).
• Non-Buyers: Simulators systematically inflate eventual non-buyers toward the purchase frame (mean depth bias $\approx +0.40$).
• Statistical Significance: The outcome-conditioned contrast is significant ($\Delta = +0.304$, Cohen's $d = 0.38$, $p < 0.001$).
• Mechanism: The simulator halves the resistance expressed by real non-buyers (25.1% $\to$ 13.5%) and nearly doubles their deliberation (21.9% $\to$ 40.1%), while the rate of "deciding" (imminent purchase) remains unchanged. Simulators do not fabricate fake sales; they manufacture "interested deliberators" out of people who would otherwise disengage.

2. Generality and Robustness

• Model Families: The deficit replicates when swapping the simulator backbone to DeepSeek-V4-Flash ($\Delta = +0.330$, $d = 0.41$, $p = 0.002$).
• Instrument Swap: Re-scoring with a different model family (DeepSeek instrument) strengthens the deficit ($\Delta = +0.351$, $d = 0.42$), ruling out judge artifacts.
• Explicit Instructions: Explicitly instructing the simulator that it may be disinterested, perfunctory, or impatient reduces marginal bias five-fold but fails to fix the conditional contrast ($\Delta$ remains $\approx +0.285$). The instruction causes the simulator to over-resist buyers (relocating the error) rather than correctly disengaging non-buyers.

3. Consequences for Agent Training

• Mispricing Tactics: The deficit is concentrated on pressure tactics. After a sales "pitch," real users deliberate 45.3% of the time, whereas simulated users deliberate 75.2%. After a "push-close," simulated resistance is suppressed by 8.2 percentage points.
• Reward Landscape: Agents trained or evaluated against such simulators receive strong positive signals for pressure tactics that, in reality, elicit resistance or silence. This creates a directional bias that rewards over-pursuit.

Significance and Claims

The paper claims that decision fidelity is the property that determines whether simulator-based conclusions transfer to reality, yet it is currently unmeasurable by standard benchmarks.

• The "Marginal Realism Trap": The authors argue that optimizing for marginal realism (population-level alignment) is insufficient and potentially misleading. A simulator can match aggregate statistics while being systematically wrong for the specific sub-population (non-buyers) that determines revenue.
• Root Cause: The deficit is attributed to instruction tuning. Alignment data rewards models for being helpful, responsive, and forward-moving interlocutors. It lacks sufficient examples of real-world disengagement (terse deflection, stalling, silence). Consequently, simulators default to cooperative continuation, rendering resistance only as its in-distribution neighbor: deliberation.
• Practical Warning: Any pipeline evaluating or reinforcement-training persuasion agents against current LLM simulators is grading them on an "easy" population. These pipelines systematically overstate funnel progress, particularly for the customers who walk away.
• Future Direction: The paper posits that decision fidelity cannot be prompted into existence; it requires learning from real disengagement trajectories (like those in ZhenaiSales) or enforcing an external willingness model.

The paper concludes that for agents whose purpose is to change a decision, how users decide must be faithful, not just how they talk. Simulated customers, in current paradigms, never walk away.