Technical Summary: Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors

Problem Formulation

As Large Language Models (LLMs) are increasingly deployed as user simulators for the interactive training and evaluation of AI assistants, a critical challenge remains: whether these simulators faithfully capture the broad, heterogeneous distribution of real user behaviors. Existing works often focus on generating human-like responses, yet they may fail to represent the diversity of real-world interactions. A simulator can fail in two primary ways, analogous to precision and recall in generative modeling:

1. Low Precision: The simulator demonstrates behaviors that real users rarely exhibit.
2. Low Recall: The simulator fails to demonstrate behaviors that real users frequently exhibit.

These failure modes introduce bias into training and evaluation pipelines, potentially yielding assistants that struggle to generalize to the diverse behaviors of actual users. The paper addresses the lack of systematic methods to measure this distributional gap between real user behavior ($P$) and simulated user behavior ($Q$).

Methodology

The authors propose a three-stage method to quantify the distributional gap between real and simulated conversations, moving beyond surface-level lexical similarity to capture behavioral patterns.

1. Generating User Behavior Representations

For each conversation (both real and simulated), the method extracts a semantic representation of the user's behavior. Instead of using raw text, an LLM is prompted to generate a description of the user's behavior across six specific facets grounded in established frameworks:

• Requests: Explicitness, goal decomposition, and relevance to the primary goal.
• Responses: Engagement levels, evaluation of assistant outputs, and feedback types.
• Context: Type of background information provided, its relevance, and delivery timing (front-loaded vs. incremental).
• Communication Style: Register, emotional tone, verbosity, formatting, and social conventions.
• DAMSL Dialog Acts: Per-utterance analysis regarding information level, forward-looking function, and backward-looking function.
• SGD Dialog Acts: Classification into discrete labels (e.g., inform, request, affirm) adapted from the Schema-Guided Dialogue dataset.

These descriptions are concatenated and embedded into a shared semantic space using a text embedding model, creating a set of continuous, high-dimensional vectors for both real ($D_{real}$) and simulated ($D_{sim}$) conversations.

2. Quantizing into Behavioral Distributions

To enable reliable divergence estimation, the continuous embeddings are quantized into discrete distributions. The authors employ $k$-means clustering (with $k=500$) on the combined set of real and simulated embeddings. Each cluster represents a distinct "mode" of user behavior. This process yields two probability distributions:

• $\hat{P}$: The fraction of real user representations falling into each cluster.
• $\hat{Q}$: The fraction of simulated user representations falling into each cluster.

3. Measuring the Distributional Gap

The gap between $\hat{P}$ and $\hat{Q}$ is measured using divergence-based metrics that correspond to the failure modes:

• Forward KL Divergence ($KL(\hat{P} \parallel \hat{Q})$): Measures low recall (simulator misses behaviors common in real users).
• Backward KL Divergence ($KL(\hat{Q} \parallel \hat{P})$): Measures low precision (simulator exhibits behaviors rare in real users).
• Jensen–Shannon (JS) Divergence: A symmetric metric capturing both precision and recall failures.

Key Contributions

1. Measurement Framework: Introduction of a validated method to measure the distributional gap between real and simulated user behaviors, utilizing behavioral descriptions and clustering to abstract away surface-level features.
2. Systematic Evaluation: The first large-scale evaluation of 24 LLM-based user simulators (7 closed-source, 15 open-source, and 2 trained simulators) across coding and writing tasks.
3. Combinatorial Mitigation: Demonstration that combining behaviorally complementary simulators can reduce the distributional gap compared to using a single simulator.
4. Interpretable Analysis: A TF-IDF analysis of behavioral clusters that categorizes behaviors into those that are well-captured, missed, or hallucinated by simulators.

Experimental Results

The study evaluated 24 simulators on 5,000 real-world conversations from the WildChat dataset, focusing on coding and writing tasks.

• Distributional Gap: A significant distributional gap exists between all simulators and real users. The gap is generally smaller for coding tasks than for writing tasks.
• Model Performance:
  • Closed-Source Models: Generally outperformed open-source models. GPT-5.4 and Gemini 3.1 Pro achieved the smallest gaps.
  • Open-Source Models: The best open-source models (e.g., gemma-4-31B-it, gpt-oss-120b) were competitive with, and in some cases outperformed, several closed-source models.
  • Trained Simulators: Specialized simulators like humanlm-opinion and UserLM-8b performed on par with the best closed-source models, though UserLM-8b's results were noted with caution due to training data overlap with the evaluation set.
• Scale vs. Family: Model scale alone was insufficient to close the gap. While the Gemma and gpt-oss families showed consistent improvement with scale, other families (e.g., Qwen, Llama) showed inconsistent trends where smaller models sometimes outperformed larger ones.
• Facet Analysis: Simulators approximated "Requests" and "Context" facets relatively well but diverged significantly on "Communication Style," "DAMSL," and "SGD" dialog acts. Trained simulators showed a wider margin of improvement over general-purpose LLMs on these harder facets.
• Pairwise Comparisons: Most simulators behaved similarly to one another, but a few (e.g., UserLM-8b) stood apart.
• Mitigation via Combination: Combining behaviorally distinct simulators (e.g., Gemini 3.1 Pro + GPT-5.4) resulted in a distribution closer to real users than either simulator individually. Conversely, combining behaviorally similar simulators yielded no improvement.

Interpretability and Validation

• Human Study: An "odd-one-out" study confirmed that the clustering method captures meaningful behavioral similarity, with annotators correctly identifying outliers 86.7% of the time.
• Robustness: Ablation studies demonstrated that the simulator rankings are robust across different embedding models (e.g., BGE, E5) and clustering algorithms (GMM, Agglomerative).
• Behavioral Patterns: TF-IDF analysis of clusters for Gemini 3.1 Pro on coding revealed:
  • Well-Captured: Iterative problem-solving, rejecting proposals, and proactive error correction.
  • Missed: Terse communication, transactional exchanges, and authoritative directives.
  • Hallucinated: Excessive positivity, gratitude, and formal pleasantries not typical of real users.

Significance and Claims

The paper claims to advance the field toward user simulators that faithfully represent the diversity of real users. By introducing a method to measure the distributional gap, the authors provide a diagnostic tool to identify where simulators fail (precision vs. recall). The work suggests that while current LLM-based simulators exhibit large gaps, these can be mitigated through the strategic combination of behaviorally complementary models. The authors emphasize that closing this gap is crucial for building AI systems that generalize well to diverse real-world populations, while also acknowledging limitations regarding dataset bias (WildChat's skew toward IT/developer communities) and the potential for simulators to be misused.