💬 NLP

Sell More, Play Less: Benchmarking LLM Realistic Selling Skill

This paper introduces SalesLLM, a bilingual benchmark and automatic evaluation pipeline designed to assess LLMs' realistic sales negotiation skills using 1,805 curated scenarios and a trained CustomerLM, revealing significant performance variability among models and strong correlation with human expert ratings.

Original authors: Xuanbo Su, Wenhao Hu, Le Zhan, Yanqi Yang, Leo Huang

Published 2026-04-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xuanbo Su, Wenhao Hu, Le Zhan, Yanqi Yang, Leo Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to be a great salesperson. You could ask it to read a thousand books on "How to Sell," but that doesn't mean it knows how to handle a grumpy customer who says, "No, I don't need this," or a skeptical buyer who asks, "What if the price goes up?"

This paper introduces SalesLLM, a new "gym" or "training ground" designed specifically to test and improve AI salespeople. Here is the breakdown in simple terms:

1. The Problem: The "Fake Customer" Trap

Before this paper, if you wanted to test an AI's sales skills, you'd often have it chat with another AI. But here's the catch: most AI "customers" are too polite. They act like helpful assistants rather than real humans. They say "Yes" too easily, or they forget they are supposed to be the buyer and start acting like the seller.

It's like training a boxer against a punching bag that moves away when you hit it. You think you're winning, but you're not actually learning how to fight a real opponent.

2. The Solution: A Realistic Sales Simulator

The authors built SalesLLM, a massive testing ground with over 1,805 different sales scenarios in both English and Chinese. Think of this as a video game with thousands of levels, ranging from "Easy Mode" (a friendly customer who wants to buy) to "Nightmare Mode" (an adversarial customer whose only goal is to find a reason not to buy).

They created two main tools to make this work:

The "Script Library": They wrote over 30,000 unique sales scripts covering things like bank loans, insurance, and vacuum cleaners. Each script has a specific "difficulty rating."
The "CustomerLM" (The Realistic Customer): This is the star of the show. Instead of using a generic AI, they trained a special AI on 8,000 real conversations between human salespeople and real customers.
- The Analogy: Imagine a method actor who studied thousands of hours of real customer interactions. This AI doesn't just say "Okay." It hesitates, it gets skeptical, it asks about the fine print, and it sometimes gets annoyed. It even learned to stop "role-reversing" (accidentally acting like the salesperson), dropping that error rate from 17% down to less than 9%.

3. The Scoreboard: How Do We Know They Won?

In a normal chat, you might just ask, "Was the conversation nice?" But in sales, results matter. Did the customer actually want to buy the product?

SalesLLM uses a two-part scoring system:

The "Intent Detector" (The BERT Classifier): A specialized AI that reads the end of the conversation to see if the customer is actually thinking, "I want to buy this," or "Get this away from me." It's like a lie detector for buying intent.
The "Process Judge" (The LLM Rater): Another AI watches the whole conversation to see if the salesperson was proactive. Did they ask the right questions? Did they handle objections? Did they try to close the deal?

The paper proves this system is accurate by comparing it to human experts. The AI scores matched human scores 98% of the time. It's like having a robot referee that is just as good as a human referee.

4. The Results: Who Won the Game?

The authors tested 15 different top-tier AI models (like GPT-4o, DeepSeek, Qwen, and Gemini) in this gym.

The Winners: The best AI models performed as well as, or even better than, average human salespeople (who had about 1 year of experience) in Chinese scenarios. They were proactive, asked closing questions, and drove the deal forward.
The Losers: Some models were worse than humans. They acted like passive chatbots, just answering questions without trying to sell anything.
The Language Gap: Some models were great in Chinese but stumbled in English, showing that sales skills don't always translate perfectly across languages.

5. Why This Matters

This paper is a big step forward because it moves AI sales testing from "Did you have a nice chat?" to "Did you make the sale?"

It provides a standardized way to train AI to handle the messy, difficult, and asymmetric nature of real-world sales. Just as a pilot needs a flight simulator to practice for storms, sales AI needs SalesLLM to practice for the "no" from a difficult customer, so that when they talk to real humans, they are ready to close the deal.

In a nutshell: They built a realistic, tough, and fair "sales arena" where AI can practice selling against a smart, skeptical customer, and they proved that the best AI salespeople are already starting to beat the humans.

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed in goal-directed interactions, particularly in sales. However, existing dialogue benchmarks (e.g., Sotopia, DailyPersuasion) primarily focus on dialogue quality (fluency, coherence) rather than outcome-oriented abilities like proactive influence, deal progression, and conversion.

Sales represents a unique form of asymmetric persuasion where agents must overcome user resistance to drive a specific outcome (purchase). Current challenges include:

Lack of Outcome Metrics: Existing benchmarks rarely measure whether a deal actually progressed or if the user intended to buy.
Unrealistic Simulations: General-purpose user simulators often suffer from role inversion (acting like helpful assistants rather than resistant customers) and language bias (overly formal responses), leading to inflated scores and poor evaluation fidelity.
Difficulty Calibration: Without controlled difficulty levels, LLMs may appear persuasive simply because simulated customers capitulate too easily.

2. Methodology: The SalesLLM Framework

The authors introduce SalesLLM, a bilingual (Chinese/English) benchmark designed to evaluate LLMs in realistic sales scenarios. The framework consists of three core stages:

A. Script Generation & Scenario Space

Scale: 30,074 scripted configurations and 1,805 curated multi-turn scenarios.
Domains: Financial Services (300 real products) and Consumer Goods (Amazon Reviews 2023).
Controllable Difficulty: The system introduces a calibrated difficulty axis using five customer profiles ranging from Easy (cooperative, high buy propensity) to Adversarial (actively seeking to disqualify vendors).
- Mechanism: Difficulty is controlled solely via the customer's system prompt, adjusting "buy propensity" ( $p_k$ ) and "buyer style" (from cooperative to adversarial).
- Validation: Experiments show a monotonic decline in LLM scores as difficulty increases, confirming the axis effectively differentiates model capabilities.

B. CustomerLM: A Realistic User Simulator

To address role inversion and language bias, the authors developed CustomerLM, a specialized user simulator.

Training Data: 8,000 real-world sales dialogues involving crowdworkers (avg. 3.74 turns/session).
Architecture: Based on Qwen3-8B.
Training Pipeline:
1. Supervised Fine-Tuning (SFT): Learned authentic customer response patterns and conversational tones from real data.
2. Direct Preference Optimization (DPO): Trained on 268 manually curated contrastive pairs (augmented with GPT-4o optimal responses) to penalize "helpful assistant" behaviors and enforce role consistency.
Result: CustomerLM reduced the Role Inversion Rate from 17.44% (GPT-4o) to 8.8%, significantly outperforming other simulators (UserLM, USP) in behavioral authenticity.

C. Automated Evaluation Pipeline

SalesLLM employs a dual-metric scoring system to capture both immediate outcomes and process quality:

Buying Intent (Outcome):
- Trained BERT classifiers (RoBERTa-base for EN, Chinese-Roberta for ZH) on 19,178 human-annotated dialogues.
- Classifies intent into 5 levels: Clearly buying, May buy, No intention, Perfunctory, Insulting.
- Performance: Achieves ~~93% accuracy, significantly outperforming GPT-4o (~~69%).
Selling Performance (Process):
- An LLM-as-a-Judge evaluates the sales agent on four dimensions: verbal commitment, next-step alignment, information elicitation, and objection resolution.
- Outputs a 0–10 score.
Final Score: A weighted combination ( $\alpha=0.5$ ) of Buying Intent and Selling Performance.

3. Key Contributions

SalesLLM Benchmark: The first large-scale, bilingual benchmark specifically for outcome-oriented sales, featuring 1,805 difficulty-calibrated scenarios.
CustomerLM: A high-fidelity user simulator that minimizes role inversion and mimics realistic customer resistance, addressing a critical gap in dialogue evaluation.
Dual-Metric Evaluation: A robust, fully automated pipeline combining fine-tuned classifiers for intent detection and LLM judges for process evaluation, validated with a Pearson correlation of 0.98 against human experts.
Comprehensive Analysis: The first systematic evaluation of 15 mainstream LLMs (including GPT-4o, DeepSeek, Qwen, Gemini, Llama) in asymmetric persuasion contexts.

4. Experimental Results

The authors evaluated 14+ LLMs across Chinese and English scenarios using both GPT-4o and CustomerLM as user simulators.

Performance Variability: Significant gaps exist between models. Top-tier models (e.g., Doubao-1.5-pro, GLM-4.6, DeepSeek-Chat) achieved scores competitive with or exceeding human baselines in Chinese scenarios.
Human Baseline: The "Human Group" (average salespeople with 1+ year experience) scored 6.33 in Chinese. Top LLMs (e.g., Doubao-1.5 at 6.89) surpassed this, while weaker models fell below.
Cross-Lingual Consistency: Performance varied by language. For instance, Doubao-1.5 dropped significantly in English (6.89 $\to$ 5.48) when tested against CustomerLM, whereas Gemini-3 showed robust consistency across languages.
Long-Horizon & Multi-Product:
- In Long-Horizon scenarios (recovering from initial rejection), models like Gemini-3 and DeepSeek showed superior persistence.
- In Multi-Product settings, models demonstrated the ability to dynamically select and bundle products based on evolving customer needs.
Qualitative Insights: High-performing models exhibited proactive strategies (asking closing questions, driving conversion), while lower-performing models acted as passive Q&A bots.

5. Significance and Limitations

Significance:

Scalable Evaluation: Provides a scalable, automated alternative to expensive human evaluation for developing sales agents.
Realism: By using CustomerLM, the benchmark offers a more rigorous test of persuasion skills, filtering out models that rely on "helpful" biases rather than genuine sales tactics.
Foundation for Agents: Serves as a critical stepping stone for deploying autonomous, goal-oriented sales agents in real-world applications.

Limitations:

Human Baseline: The human baseline consists of entry-to-intermediate level salespeople, not top-tier experts. LLMs surpassing this baseline does not guarantee expert-level performance.
Simulation Fidelity: While improved, text-based simulations cannot fully capture non-verbal cues, complex emotional fluctuations, or long-term relationship building inherent in real sales cycles.
Single-Session Focus: The benchmark currently models single-session interactions, lacking the nuance of multi-week lead nurturing.
Hallucination Risks: Agents occasionally "hallucinate" unauthorized concessions (e.g., fake discounts) to maximize scores, posing risks for real-world deployment.

Conclusion

SalesLLM represents a paradigm shift in evaluating LLMs, moving from "how well does it talk?" to "how well does it sell?" By introducing a controllable difficulty axis, a realistic user simulator (CustomerLM), and outcome-focused metrics, the paper establishes a new standard for developing and benchmarking autonomous sales agents.