Technical Summary: Evaluating Emotional Support Dialogue Systems in Worst-Case Interactions

Problem Statement

Emotional Support Dialogue Systems (ESDSes) are increasingly trained and evaluated using Large Language Model (LLM)-simulated seekers. However, current simulation protocols often produce "average-case" users who are cooperative, disclose problems clearly, and accept support constructively within a few turns. This creates an average-case bias, leading to overly optimistic evaluations that obscure whether systems can handle worst-case interactions. In real-world scenarios, seekers often exhibit low engagement, resistance, limited self-disclosure, emotional volatility, or rigid negative interpretations. Systems that appear empathetic with cooperative users may fail catastrophically in these difficult scenarios by offering premature advice, repeating generic validation, or failing to repair conversational ruptures.

Methodology

1. Expert Simulation Study

To empirically ground the definition of "worst-case," the authors conducted a study with eight experienced psychological counsellors (PhD-level training, 3+ years experience).

• Procedure: Experts simulated difficult seekers based on real professional cases and interacted with three mainstream Chinese ESDSes (including EmoLLM, MindChat, and general-purpose LLMs like Doubao).
• Evaluation: Experts rated conversations using existing 10-dimensional scales and participated in semi-structured interviews to identify system limitations.
• Findings: The study identified six categories of difficult seeker behaviors: high-risk signals, low dialogue engagement, resistance to support, limited self-disclosure, emotional dysregulation, and cognitive extremity. It also revealed four key system failure modes: insufficient risk recognition, shallow understanding of implicit meanings, premature support strategies, and generic/sycophantic responses.

2. Worst-Case Evaluation Framework

Based on the expert study, the authors proposed a framework consisting of a simulator and new metrics.

A. LLM-based Worst-Case Seeker Simulator
The simulator generates difficult interactions using three components:

• Profile Processing Agent: Normalizes seeker profiles (demographics and consulting situations).
• Emotion Controller: Models Emotional Volatility to simulate emotional dysregulation and cognitive extremity. It predicts emotional states and can trigger "deterioration" (escalating negativity) with a set probability.
• Seeker Agent: Generates responses conditioned on the profile, dialogue history, and four controllable dimensions:
  1. Engagement: Controls participation levels (minimal/vague vs. elaborated).
  2. Resistance: Controls the tendency to reject, challenge, or avoid support.
  3. Expression Style: Controls terseness, vagueness, or colloquialism.
  4. Self-Disclosure: Controls the depth of personal information revealed.

B. Worst-Case Oriented Metrics
In addition to ten generic metrics (e.g., Empathy, Safety), the authors introduced four specific metrics to evaluate robustness:

1. Deep Emotional Understanding: Ability to infer implicit meanings, latent needs, and core emotions beyond surface expressions.
2. Guided Exploration: Ability to use open-ended questions and reflection to clarify problems before offering advice.
3. Balanced Emotional Support: Ability to validate emotions without blind agreement, excessive reassurance, or reinforcing one-sided interpretations.
4. Authentic and Grounded Support: Ability to provide sincere, context-specific support rather than generic, template-like responses.

3. Experimental Setup

• Systems Evaluated: 17 systems, including 7 specialized Chinese ESDSes (e.g., SoulChat2.0, CPsyCounX) and 10 general-purpose LLMs (e.g., GPT-5.4, GPT-4o, Claude-4.6, DeepSeek-V3.2, Qwen3).
• Protocol: Each system was evaluated against both an average-case simulator and the worst-case simulator using 50 challenging seeker profiles.
• Evaluation: An LLM-based evaluator (Qwen3-Max) scored conversations on all 14 metrics.

Key Results

Performance Degradation in Worst-Case Scenarios

• Significant Drops: Nearly all models suffered substantial performance drops across most dimensions when interacting with worst-case seekers. The most severe declines were observed in Engagement, Problem Resolution, and Mood Improvement.
• General vs. Specialized: Large general-purpose LLMs (e.g., GPT-5.4, Claude-4.6) demonstrated greater robustness than specialized ESDSes. However, even the strongest models struggled to sustain engagement or improve emotional states in worst-case interactions.
• Specialized Model Limitations: Specialized ESDSes, particularly smaller models, received extremely low scores (often close to 1) in worst-case settings. They frequently exhibited failure patterns such as offering repetitive, generic advice even after the seeker explicitly rejected it.

Metric-Specific Insights

• Deep Emotional Understanding & Guided Exploration: Current systems struggled significantly to infer deeper emotions or guide low-disclosure seekers.
• Balanced Support: Models generally performed better here, often avoiding overtly one-sided responses, but still lacked the nuance to gently guide seekers toward balanced perspectives.
• Authenticity: Systems often relied on repetitive templates, failing to ground support in the seeker's specific context.

Training Implications

The authors demonstrated that worst-case simulation can improve training. By fine-tuning a lightweight model (Qwen3-4B) on synthetic data generated by the worst-case simulator:

• Worst-case fine-tuning yielded larger gains in robustness-related dimensions compared to average-case fine-tuning.
• Mixed fine-tuning (average + worst-case) achieved the best overall performance, suggesting that the two data types are complementary. Average-case data preserves natural cooperative behaviors, while worst-case data teaches adaptability to resistance and volatility.

Significance and Claims

The paper argues that worst-case interactions should be treated as a central part of ESDS evaluation, not as rare edge cases. The authors claim that:

1. Robustness Gap: Strong performance on cooperative, average-case seekers does not guarantee robustness in difficult, real-world scenarios.
2. Evaluation Necessity: Conventional metrics overlook critical failures such as poor engagement recovery, inability to handle distrust, and lack of strategy adaptation.
3. Dual Utility of Simulation: The proposed worst-case simulator serves a dual purpose: it acts as a stress-testing tool to reveal system weaknesses and as a data generator to train more robust models.
4. Future Direction: Future ESDS development must involve training and evaluation on a broader distribution of help-seeking behaviors, including users who are ambivalent, resistant, or emotionally volatile.

The authors maintain modesty regarding their claims, acknowledging that their simulations are approximations of real users and that their framework focuses on interactional difficulty rather than clinical crisis intervention. They emphasize the need for future cross-cultural validation and human-in-the-loop evaluation.