💬 NLP

ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders

This paper introduces ConvApparel, a novel dataset and comprehensive validation framework designed to bridge the "realism gap" in LLM-based user simulators for conversational recommenders by leveraging a dual-agent data collection protocol and counterfactual validation to demonstrate that data-driven simulators offer more robust generalization than prompted baselines.

Original authors: Ofer Meshi, Krisztian Balog, Sally Goldman, Avi Caciularu, Guy Tennenholtz, Jihwan Jeong, Amir Globerson, Craig Boutilier

Published 2026-02-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ofer Meshi, Krisztian Balog, Sally Goldman, Avi Caciularu, Guy Tennenholtz, Jihwan Jeong, Amir Globerson, Craig Boutilier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Uncanny Valley" of Shopping Bots

Imagine you are trying to teach a robot how to be a helpful shopping assistant. You can't just ask real humans to shop with the robot 24/7 because it's expensive and takes forever. So, you build a simulator—a fake human powered by AI—that pretends to be a shopper so you can train your robot.

The problem? These fake shoppers are often too perfect. They are polite, patient, and logical. Real humans, however, get frustrated, change their minds, make typos, and sometimes act irrationally.

If you train your shopping robot on these "perfect" fake humans, the robot will become great at talking to robots, but it will fail miserably when a real, grumpy human walks in. This gap between the fake world and the real world is called the "Realism Gap."

The Solution: ConvApparel (The "Stress Test" for Shopping Bots)

The authors created a new dataset called ConvApparel to fix this. Think of it as a gym for AI shoppers, but with a twist.

1. The "Good Cop, Bad Cop" Protocol

Most datasets only show AI how to talk to a perfect, helpful salesperson. ConvApparel is different. It records real humans shopping with two types of AI assistants:

The "Good" Agent: A helpful, smart salesperson who understands you perfectly.
The "Bad" Agent: A confusing, unhelpful, and slightly annoying salesperson who misunderstands you, gives irrelevant facts, and circles back to questions you already answered.

Why this matters: It's like training a driver not just on a sunny day on an empty highway, but also in a rainstorm with a broken GPS. If your AI shopper can handle the "Bad Agent" realistically (by getting annoyed and asking for a manager), it proves it understands human emotions, not just the script.

2. The "Inner Monologue" Annotations

Usually, we only see what a shopper says. But in ConvApparel, after every turn of the conversation, the real human shopper had to fill out a quick survey about their inner feelings.

Did I feel frustrated?
Was I confused?
Did I want to quit?

This is like giving the researchers a X-ray of the shopper's brain. It allows them to see if the AI simulator is actually capturing those hidden feelings or just pretending to be happy.

The Three-Part "Lie Detector" Test

The paper doesn't just give you data; it gives you a Validation Framework (a way to check if a simulator is lying). They use three methods to see if an AI simulator is "real":

The Statistical Check (The "Census"):
They count things: How many words do people use? How many questions do they ask? If the simulator's numbers match the real humans' numbers, it passes the first test.
- Analogy: It's like checking if a fake crowd at a concert has the same average height and age as a real crowd.
The "Human-Likeness" Score (The "Turing Test"):
They trained a special AI (a "Discriminator") to read conversations and guess: "Is this a human or a robot?"
- Analogy: Imagine a detective trying to spot a spy in a room. If the simulator can fool the detective, it gets a high score. The paper found that even the best simulators still got caught easily—they just sounded too perfect.
The "Counterfactual" Test (The "What If?" Scenario):
This is the most important part. They trained a simulator on conversations with the "Good" agent, then threw it into a conversation with the "Bad" agent.
- The Test: Did the simulator get frustrated? Did it stop buying things?
- The Result: Simple "prompted" bots (robots just following instructions) didn't change their behavior; they kept being polite even when the salesperson was terrible. But the data-driven simulators (robots trained on real data) actually got annoyed and acted like real humans would. This proved they learned a deeper model of human behavior.

The Main Takeaway

The paper concludes that we are still not there yet. Even the smartest AI simulators have a "Realism Gap." They aren't quite human enough to perfectly mimic our quirks and frustrations.

However, the paper shows that training on real data (learning from actual human interactions) is much better than just giving the AI a set of rules (prompting). The data-driven simulators are the closest we've gotten to a "digital twin" of a human shopper, capable of reacting realistically even when things go wrong.

In short: We built a better training ground (ConvApparel) and a better way to grade the students (the Validation Framework). We found that while our AI students are smart, they still need more practice to truly understand the messy, emotional, and sometimes irrational nature of real human shoppers.

1. Problem Statement

The paper addresses a critical bottleneck in the development of Conversational Recommender Systems (CRSs): the "realism gap" in Large Language Model (LLM) based user simulators.

The Issue: While LLMs have revolutionized conversational AI, current user simulators often fail to generalize to real-world interactions. They tend to exhibit systematic deviations from human behavior, such as excessive verbosity, inconsistent personas, unrealistic patience, and a lack of coherent preference expression.
The Consequence: Systems trained or evaluated on these unrealistic simulators may be optimized for "simulated" interactions but fail when deployed with actual human users.
The Gap in Evaluation: Existing evaluation methods often rely on simple statistical alignment or static metrics, which are insufficient to detect whether a simulator has truly learned a robust model of user behavior or is merely mimicking surface-level patterns from its training data.

2. Methodology

The authors propose a two-pronged approach: the creation of a novel dataset and a comprehensive validation framework.

A. The ConvApparel Dataset

The authors introduce ConvApparel, a dataset of 4,146 human-AI shopping conversations involving 897 participants.

Domain: Apparel shopping (footwear, outerwear, etc.) using a multi-modal interface (text + item carousels).
Dual-Agent Protocol (Key Innovation): To enable rigorous testing, the dataset was collected using two distinct recommender agents:
- "Good" Agent: A helpful assistant using robust semantic retrieval.
- "Bad" Agent: An intentionally unhelpful, confusing, and tangential agent designed to test the simulator's ability to react to poor system performance.
First-Person Annotations: Unlike standard datasets, participants provided turn-by-turn, self-reported annotations of their internal states (satisfaction, frustration, confusion, purchase likelihood) immediately after each turn. This provides ground-truth data for latent user states, avoiding the bias of third-party inference.

B. The Validation Framework

The paper proposes a three-pillar framework to evaluate simulator fidelity:

Population-Level Statistical Alignment (PLSA):
- Compares the distribution of simulator behaviors against human populations.
- Metrics include basic conversation statistics (turns, word count), behavioral dialog acts (accept/reject, ask clarification), and user experience metrics (satisfaction, frustration).
- Uses statistical tests (Mann-Whitney U, Kolmogorov-Smirnov) to quantify distributional differences.
Human-Likeness Score (HLS):
- A discriminator-based metric inspired by the Turing Test.
- A binary classifier (LLM-based) is trained to distinguish between human and simulated conversations.
- The score represents the probability a conversation is human; a low score indicates the simulator is easily detectable as synthetic, revealing a "holistic" realism gap even if statistical metrics align.
Counterfactual Validation (Primary Contribution):
- Tests generalizability by training a simulator on data from one agent type (e.g., "Good") and evaluating its behavior on an unseen agent type (e.g., "Bad").
- Hypothesis: A truly robust simulator should learn an underlying model of user psychology and react plausibly to new conditions (e.g., showing increased frustration when interacting with a "Bad" agent), rather than just overfitting to the training distribution.

3. Key Contributions

ConvApparel Dataset: The first dataset designed specifically for counterfactual validation of user simulators, featuring controlled variations in system quality and fine-grained first-person user annotations.
Comprehensive Validation Framework: A multi-faceted approach combining statistical alignment, a discriminator-based "human-likeness" score, and counterfactual testing to move beyond simple performance metrics.
Empirical Benchmarking: Evaluation of three representative LLM-based simulator types:
- Prompted: Zero-shot prompting with behavioral instructions.
- In-Context Learning (ICL): Retrieval-augmented generation using similar human conversations.
- Supervised Fine-Tuning (SFT): Fine-tuning a base LLM on the ConvApparel dataset.

4. Results and Analysis

The experiments reveal significant insights regarding the "realism gap":

Existence of a Realism Gap: All tested simulators (Prompted, ICL, SFT) exhibit a significant realism gap. Even the most advanced models fail the Human-Likeness Score (HLS), with the discriminator assigning near-zero probabilities of humanity to all simulated conversations (average HLS $\approx$ 0.004).
Statistical Alignment vs. Generalization:
- In-Distribution: Data-driven simulators (ICL and SFT) show better statistical alignment with human behavior than the Prompted baseline, particularly in the "Good" agent scenario.
- Counterfactual Robustness: The most critical finding is in the counterfactual validation. When trained on "Good" agent data and tested on the "Bad" agent:
  - Prompted Simulators: Fail to adapt realistically; they often maintain a polite or static persona regardless of the agent's poor performance.
  - Data-Driven Simulators (ICL/SFT): Successfully adapt. They exhibit increased frustration, ask more clarification questions, and reject recommendations more frequently, mirroring the behavioral shifts of real humans.
Discriminator Insights: The discriminator can easily distinguish between simulators. SFT-trained models are harder to distinguish from humans than Prompted models, suggesting they capture subtler behavioral artifacts, though they still fail to be indistinguishable.

5. Significance and Future Directions

Paradigm Shift: The paper argues that the field must move beyond "closing the realism gap" (building better simulators) to first "measuring the gap" with rigorous tools. The proposed framework provides these tools.
Validation of Data-Driven Approaches: The results suggest that while current LLM simulators are not perfect, data-driven methods (ICL, SFT) embody more robust user models than simple prompting, particularly in their ability to generalize to unseen, suboptimal system behaviors.
Implications for CRS Training: Using simulators that fail counterfactual validation risks training CRS agents that are brittle and fail in real-world deployment. The framework allows researchers to select simulators that are robust enough for training.
Limitations & Future Work: The authors acknowledge limitations, including the focus on a single domain (apparel), the uni-modal nature of user input (text-only, no clicks), and the reliance on LLM-as-a-judge for some metrics. Future work should explore downstream impacts on agent training and balancing realism with controllability (e.g., steerable personas).

In conclusion, ConvApparel provides the necessary infrastructure to rigorously test whether user simulators are merely mimicking patterns or truly modeling human behavior, marking a significant step toward reliable, next-generation conversational AI.