Technical Summary: SYMPATHEIA – Emotionally Adaptive Voice Assistant with Continuous Affect Conditioning

Problem Statement

Empathetic spoken dialogue systems require the ability to infer a user's emotional state to generate appropriate responses. However, everyday speech often contains weak, neutral, or ambiguous affective cues that are difficult to infer reliably from lexical or acoustic signals alone. Current systems face two primary limitations:

1. Ambiguity in Speech: Many methods are optimized for high-intensity emotional cues, failing to handle the subtle or mixed affective states common in natural conversation. They lack mechanisms to combine ambiguous speech cues with external emotional signals (e.g., facial expressions or biosignals).
2. Discrete Representation: Emotion is frequently modeled using fixed, discrete categories (e.g., "happy," "sad"). This restricts natural interaction, as real-world emotional expression varies across speakers and cultures. Continuous representations, such as valence–arousal (VA), are better suited to capture this nuance but are underutilized in speech-to-speech generation.

Methodology

The authors introduce SYMPATHEIA, a speech-to-speech dialogue framework designed to generate semantically appropriate and emotionally aligned responses. The system integrates implicit affect inference from user speech with optional explicit affect control via a continuous VA interface.

System Architecture

• Backbone: The model is built upon GLM-4-Voice-9B, an end-to-end speech-language model.
• Tokenization: Input audio is encoded into discrete speech tokens using a WhisperVQ tokenizer (12.5 Hz).
• Conditioning: A continuous valence–arousal pair ($z = (v, a)$) is inserted into the system prompt as User emotion (valence=v, arousal=a).
  • VA Interface: The VA coordinates ($v, a \in [-1, 1]$) serve as a shared interface for external emotion sensing modules (facial expression, biosignals like EEG/ECG, or textual descriptions).
  • Modality Agnostic: External sensors map their native outputs to the VA plane using a probability-weighted mapping to an expected coordinate, allowing the dialogue model to remain agnostic to the specific sensing modality.
• Training Strategy: The model is fine-tuned using Low-Rank Adaptation (LoRA) on the SYMPATHEIA-18k dataset.
  • VA Anchors: Training utilizes 12 named emotion anchors (e.g., happy, sad, angry) mapped to specific VA coordinates. This encourages the model to learn a continuous affect space rather than isolated class labels.
  • Dropout: To ensure robustness, the VA condition is randomly dropped for one-third of non-neutral training samples, forcing the model to infer affect from speech cues when external signals are absent.

Dataset: SYMPATHEIA-18k

To address the lack of emotion-conditioned speech-to-speech data, the authors constructed SYMPATHEIA-18k, a synthetic corpus of 18,000 query–response pairs:

• Emotional Split (~12k): Pairs affect-rich user queries with emotion-appropriate responses to teach semantic and acoustic alignment.
• Neutral Split (6k): Pairs 500 emotionally neutral queries with 12 different responses (one for each target emotion). This isolates explicit affect control, training the model to rely on the VA condition when speech provides no emotional evidence.
• Generation: Text was generated using Qwen3-32B with specific empathetic strategies, and audio was synthesized using Qwen3-TTS to ensure emotion-consistent prosody and pacing.

Key Contributions

1. SYMPATHEIA Framework: A voice-native model that combines implicit speech-based affect inference with explicit, continuous VA conditioning from external sensors or user interfaces.
2. SYMPATHEIA-18k Dataset: A large-scale synthetic dataset featuring both expressive and neutral interactions, specifically designed to train models to distinguish between affective states and utilize external affect signals.
3. Modular Multimodal Interface: A system design that allows diverse external affect sources (face, biosignals, text) to guide response generation through a unified VA interface without requiring architectural changes to the dialogue backbone.

Results

The paper evaluates SYMPATHEIA against strong spoken-dialogue baselines (including GLM-4-Voice, Qwen3-Omni, and OpenS2S) across several metrics:

• Empathetic Response Quality: SYMPATHEIA achieved the highest scores in automated empathy evaluations and human Emotion Mean Opinion Score (MOS) studies. It significantly outperformed baselines, particularly in settings requiring explicit affect control (Neutral queries with VA prompts).
• Response Variability: Unlike baselines that often produced identical responses regardless of the target emotion, SYMPATHEIA demonstrated lower semantic and lexical similarity across different emotion conditions, indicating it successfully adapts content and tone based on the affect signal.
• Prosodic Alignment: Analysis of prosodic features (F0, energy, speaking rate) showed that SYMPATHEIA's generated speech correlates significantly with the target VA coordinates. Higher arousal targets resulted in higher pitch variability and faster speaking rates, while valence influenced energy and spectral content.
• Multimodal Integration: Integrating external cues (face, EEG, ECG, text) improved empathy scores over the "no-cue" condition. Facial expressions and textual descriptions yielded the largest gains, while physiological signals provided improvements despite being noisier.
• Ablation Studies:
  • Fine-tuning did not degrade the model's general conversational capabilities (measured by UTMOS, BERTScore, and ASR-WER).
  • The model remained effective when VA coordinates were perturbed with Gaussian noise, demonstrating tolerance to upstream sensing errors and the ability to handle off-anchor values.

Significance and Claims

The paper posits that continuous affect conditioning is a practical and effective mechanism for building emotionally adaptive voice assistants. By decoupling sensing from generation via a shared VA interface, SYMPATHEIA can integrate diverse emotion recognition systems while maintaining a robust dialogue backbone.

The authors claim that their approach addresses the critical gap of handling ambiguous or neutral speech by allowing external signals to guide the emotional tone of the response. They emphasize that while the current evaluation relies heavily on automated judges and synthetic data, the results suggest that explicit VA control helps organize both semantic and acoustic response strategies, leading to more empathetic interactions.

Limitations Acknowledged:

• The training data is synthetic and may not fully capture the disfluency and long-horizon dynamics of spontaneous real-world conversation.
• The VA plane is a compact representation; a single point cannot fully capture blended emotions or rapid intra-turn shifts.
• There is no universal mapping from discrete emotion labels to VA coordinates across cultures or individuals.
• Current evaluations are primarily automated; future work requires human assessment on spontaneous, long-duration conversations to validate perceived empathy and trust in real deployments.