Modeling and Benchmarking Spoken Dialogue Rewards with Modality and Colloquialness

1. Problem Statement

The rapid advancement of end-to-end spoken dialogue systems (SDS) has outpaced the development of reliable evaluation and optimization methods. Current approaches face two critical gaps that hinder the alignment of spoken dialogue with human preferences:

• The Modality Gap: Speech contains paralinguistic information (prosody, emotion, channel conditions) that text-based evaluators cannot perceive. General-purpose Audio Large Language Models (LLMs) often suffer from "modality blindness," failing to distinguish between natural human speech and synthesized artifacts when semantic content is identical.
• The Colloquialness Gap: Text-optimized systems often produce responses that are grammatically correct but sound overly scripted or formal when spoken. Natural human conversation relies on brevity, fragmentation, discourse markers, and interactional cues, which standard evaluators struggle to quantify.

Existing reward models either rely on rigid, handcrafted acoustic rules (lacking flexibility) or focus on single-turn Text-to-Speech (TTS) quality, failing to capture the long-range coherence and spontaneity required for multi-turn spoken dialogue.

2. Methodology

A. Dataset Construction: SDiaReward-Dataset

The authors constructed a large-scale, episode-level preference dataset containing 11,630 pairwise samples (approx. 200 hours of paired speech). The dataset is designed to explicitly target the two identified gaps through two distinct subsets:

1. Modality-Aware Subset: Juxtaposes authentic human speech (from "Wild" YouTube recordings, semi-wild MELD, and scripted DailyTalk) against high-fidelity synthesized counterparts generated by SoulX-Podcast. This forces the model to learn acoustic naturalness and prosodic fidelity independent of semantic content.
2. Colloquialness Subset: Contrasts formal, written-style dialogues with their spoken-style rewrites (incorporating fillers, fragmentation, and informal markers) while keeping the acoustic conditions identical (using the same TTS engine). This isolates stylistic naturalness as the preference signal.

The dataset undergoes rigorous quality control, including LLM-based filtering for content adequacy and coherence, and is structured into multi-turn episodes (up to 16 turns).

B. Benchmark: ESDR-Bench

To ensure robust evaluation, the authors established ESDR-Bench, a stratified benchmark derived from the validation split. Unlike random splits, ESDR-Bench uses stratified sampling across source types (Wild, Semi-wild, Scripted) and metadata (emotion, sentiment, dialogue acts) to prevent high-frequency data types from dominating the evaluation and to test model generalization across diverse acoustic domains.

C. Reward Modeling Framework (SDiaReward)

The proposed SDiaReward is an end-to-end multi-turn reward model trained on the constructed dataset.

• Architecture: Built upon a multimodal LLM backbone (Qwen2.5-Omni), it processes interleaved speech-text sequences. It uses a Mean Pooling strategy over the hidden representations to aggregate holistic episode-level context, followed by a task-specific MLP head to output a scalar reward.
• Multi-Criteria Decomposition: The model is trained to handle two distinct evaluation criteria via system prompts:
  1. Modality-Awareness: Evaluating acoustic coherence and prosodic naturalness.
  2. Colloquialness: Evaluating conversational spontaneity and the avoidance of robotic formalisms.
• Loss Function: The model is optimized using the Bradley-Terry preference framework. Crucially, the authors introduce a Center Loss regularization term ($L_{center}$) to the standard pairwise loss. This anchors the reward distribution around zero, preventing "score drift" where models might reward cleaner audio channels over better dialogue quality, thereby ensuring the model learns relative preferences within domains rather than absolute biases.

3. Key Contributions

1. SDiaReward-Dataset: The first large-scale corpus specifically curated for episode-level spoken dialogue reward modeling, featuring 11k preference pairs that explicitly address modality and colloquialness gaps.
2. SDiaReward Framework: An end-to-end reward model that jointly assesses paralinguistic fidelity and interactional spontaneity without relying on handcrafted acoustic features.
3. ESDR-Bench: A stratified benchmark with multi-dimensional annotations that provides a rigorous standard for evaluating generalization across different recording conditions and dialogue styles.

4. Experimental Results

Experiments on ESDR-Bench demonstrate that SDiaReward significantly outperforms both proprietary (e.g., GPT-4o-audio, Gemini 2.5) and open-source general-purpose Audio LLMs.

• Performance: SDiaReward-7B achieved 96.61% Micro Accuracy on the Modality benchmark and 97.20% on the Colloquialness benchmark. In contrast, the best baseline (Gemini 2.5 Pro) scored only 72.63% on Modality, highlighting the inability of zero-shot models to discern subtle prosodic naturalness.
• Generalization: The model maintained high Macro accuracy (94.91%), indicating robustness across heterogeneous data splits (Wild, Semi-wild, Scripted). Smaller models (3B) showed significant performance drops in "Semi-wild" scenarios, suggesting that resolving complex, semi-scripted interactions requires sufficient model scale.
• Human Alignment: A blinded human study on 75 stratified pairs showed an 83.5% agreement rate between the model's predictions and human ground-truth labels.
• Ablation Studies:
  • Pooling: Mean Pooling outperformed Last-Token and Attention Pooling, suggesting reward-relevant information is distributed across the entire context.
  • Center Loss: The inclusion of Center Loss stabilized training and improved accuracy (from 95.37% to 96.70%) by preventing logit saturation and score drift.
  • Domain Bias: Analysis revealed that the model implicitly learns relative ranking functions calibrated to specific domain difficulties (e.g., assigning lower absolute scores to "Scripted" data while maintaining high classification accuracy), rather than a globally absolute metric.

5. Significance

This work addresses a fundamental bottleneck in the deployment of end-to-end spoken dialogue systems. By shifting from rule-based acoustic metrics to data-driven preference learning, the authors demonstrate that it is possible to teach models to value "conversational expressiveness" and "acoustic authenticity."

The findings suggest that while LLMs can naturally bridge the colloquialness gap through linguistic priors, the modality gap requires specialized, data-driven reward modeling. SDiaReward provides a scalable solution for aligning spoken dialogue systems, enabling Reinforcement Learning from Human Feedback (RLHF) that prioritizes not just what is said, but how it is said, paving the way for more natural and emotionally resonant human-AI interactions.