Technical Summary: EmoS – A Theory-Grounded Framework for Evaluating and Aligning Emotional Intelligence in Spoken Language Models

1. Problem Statement

Current Spoken Language Models (SLMs) have achieved significant milestones in auditory comprehension, context awareness, and instruction-following. However, their evaluation of Emotional Intelligence (EI) remains rudimentary, largely confined to basic paralinguistic perception (e.g., detecting tone or volume) rather than deep cognitive reasoning. Existing benchmarks often treat speech merely as an auxiliary input for text processing or rely on coarse-grained emotion classification.

The paper identifies a critical gap: genuine EI, as defined by the psychological four-branch model (Perceiving, Understanding, Using, and Managing Emotion), involves complex cognitive abilities to reason about emotions and utilize them for problem-solving. Current evaluations fail to capture this hierarchy. Furthermore, relying on proprietary models (e.g., GPT-4o-Audio) as evaluators creates an artificially low performance ceiling, masking the intrinsic deficiencies of current systems. There is a lack of fine-grained, theoretically aligned datasets necessary to train specialized evaluators capable of valid EI reasoning.

2. Methodology

The authors propose a comprehensive framework comprising three core components: a new benchmark (EmoSBench), a specialized dataset (EmoDialogue), and a dedicated evaluation model (EmoS).

2.1 EmoSBench: The Evaluation Benchmark

Built upon the Mayer and Salovey four-branch model, EmoSBench evaluates SLMs across four hierarchical dimensions and ten specific sub-tasks:

• Perceiving Emotion:
  • Basic Acoustic Information Perception (BAP): Identifying explicit emotional and paralinguistic features.
  • Implicit Attitude Analysis (IAA): Capturing subtle cues from faint acoustic hints or textual implications.
• Understanding Emotion:
  • Emotional State Tracking (EST): Identifying emotional transitions across dialogue turns.
  • Emotional Causation Analysis (ECA): Connecting current emotional states to historical triggers.
• Using Emotion:
  • Emotion-Cognition Matching (ECM): Understanding how emotional states facilitate cognitive tasks.
  • Emotion-Driven Plan Adjustment (EPA): Detecting acoustic-semantic conflicts (e.g., hesitant tone vs. committed text) to guide decision-making.
• Managing Emotion:
  • Social Strategy Execution (SSE): Coordinating text and tone for challenging social objectives.
  • Proactive Mitigation and Emotional Buffering (PMB): Softening negative information.
  • Conflict Resolution and De-escalation (CRD): Empathy-first strategies in high-pressure scenarios.
  • Value Alignment and Safety Response (VAR): Maintaining dialogue while upholding safety principles against harmful inputs.

2.2 EmoDialogue: The Training Dataset

To address the scarcity of fine-grained EI data, the authors curated EmoDialogue, a bilingual (Chinese/English) dataset of over 70,000 pairs.

• Synthesis: Generated using the DeepSeek-R1 LLM to simulate realistic user-model interactions.
• Grading: Each user input is paired with four candidate responses, rigorously graded from 1 to 4 based on EI quality (from mechanical/incoherent to high-EI with proactive guidance).
• Synthesis: High-fidelity audio was generated using the Doubao TTS API with explicit control over emotion, speed, and volume, augmented with non-verbal cues (sighs, coughs).
• Quality Control: Automated filtering via Whisper-Large-V3 (WER/CER < 5%) followed by manual review for naturalness and consistency.

2.3 EmoS: The Specialized Evaluator Model

EmoS is a reward model optimized via a two-stage training paradigm:

1. Supervised Fine-Tuning (SFT): Initialized on EmoDialogue to establish a foundational distribution of EI knowledge.
2. Reinforcement Learning (RL): Utilizing Group Relative Policy Optimization (GRPO) to align the model's scoring and reasoning with theoretical guidelines.

Reward Engineering:
To enforce precise ordinal scoring and valid reasoning, the authors designed a composite reward function:

• Steep Exponential Accuracy Reward (SEAR): $R_a(\Delta) = \alpha \cdot e^{-\lambda\Delta}$. This penalizes prediction errors exponentially, ensuring that even minor deviations (e.g., predicting 3 instead of 4) receive significantly lower rewards than correct predictions, combating optimization laziness.
• Rationale Fidelity Reward (RFR): A mechanism using an automated critic (Qwen3-8B) to evaluate the reasoning process. An adaptive weighting coefficient ensures that while correct answers receive full reinforcement, the model still receives residual supervision on its reasoning logic even during incorrect predictions.

3. Key Results

3.1 Benchmark Performance (EmoSBench)

Experiments on EmoSBench reveal a substantial gap between current models and human performance:

• Human Baseline: 86.0% average accuracy.
• Leading Proprietary Models: GPT-4o-Audio achieved 52.6%, and Gemini 2.5 Pro achieved 54.0%.
• Leading Open-Source Models: Most open-source SLMs (e.g., Qwen2-Audio, Freeze-Omni) scored below 30%, performing barely better than random guessing. Even Qwen3-Omni (30B parameters) only reached 50.1%.
• EmoS Performance: The specialized EmoS model achieved 83.8% average accuracy, approaching human-level performance and significantly outperforming all baselines.

3.2 Real-World Generalization

Evaluated on an authentic, unconstrained dataset derived from YouTube videos (531 samples):

• Performance degraded across all models due to background noise and spontaneous disfluencies.
• EmoS maintained an accuracy of 62.9%, outperforming the strongest baseline (Gemini 2.5 Pro at 43.3%) by a margin of 19.6%. This suggests the model learned transferable emotional reasoning rather than merely overfitting to synthetic data.

3.3 Ablation Studies

• SFT alone improved the baseline from 37.2% to 57.0%.
• GRPO alone (without SFT) reached 63.0%, indicating the value of RL exploration.
• SEAR and RFR individually boosted performance, with the combination of SFT + GRPO + SEAR + RFR yielding the peak 83.8% accuracy.

4. Significance and Claims

The paper claims to establish a foundational framework for advancing emotionally intelligent dialogue systems. Its primary contributions are:

1. Theoretical Grounding: It moves EI evaluation beyond simple acoustic perception to a hierarchical cognitive framework (Perceiving, Understanding, Using, Managing), offering a more realistic reflection of high-EI interactions.
2. Data Innovation: It introduces the first dataset with theoretically defined scoring gradients, emphasizing the quality of emotional interaction over simple classification.
3. Specialized Evaluation: It demonstrates that relying on general-purpose SOTA models as judges sets an artificially low ceiling. By developing a specialized evaluator (EmoS) trained on theory-aligned data, the authors show that precise, human-level EI evaluation is achievable.

The authors conclude that while current SLMs possess strong general instruction-following capabilities, they lack the nuanced cognitive reasoning essential for complex EI scenarios. EmoS bridges this gap, providing a robust tool for evaluating and aligning future spoken language models toward genuine emotional intelligence.