Technical Summary: From Awareness to Adherence in Spoken Dialogue Systems

1. Problem Statement

Despite the success of end-to-end (E2E) spoken dialogue systems (SDS) in single-turn interactions, maintaining strict context adherence in multi-round conversations remains a significant challenge. Current state-of-the-art models often generate responses that are incoherent with established dialogue history or fail to respect user constraints, leading to a degraded user experience.

While prior research frequently attributes these failures to models "forgetting" dialogue history and proposes external retrieval mechanisms, this work identifies a more nuanced bottleneck: a gap between latent context awareness and active adherence. The authors posit that models often possess the internal ability to recognize relevant past information (latent awareness) but fail to manifest this context in the final output due to strong parametric priors that overshadow contextual signals during the decoding stage. Thus, the core issue is not merely a memory deficit but a generation flaw requiring a decoding-centric intervention.

2. Methodology: Audio-Adapted Context-Aware Decoding (CAD)

To bridge the gap between latent awareness and active adherence, the authors propose an audio-adapted Context-Aware Decoding (CAD) approach. This method amplifies contextual influence during inference without requiring additional training or external retrieval modules. The framework operates through three primary stages:

A. Problem Formalization

The system treats multi-round dialogue as a sequence of rounds, where each round consists of a user turn ($U$) and an assistant turn ($A$). The goal is to generate the assistant's response ($A_n$) conditioned on the conversation history ($H_{n-1}$) and the current query ($U_n$), ensuring strict faithfulness to the history.

B. Latent Context Awareness for Key Context Extraction

Instead of treating the entire history as uniformly relevant, the method leverages the model's internal attention mechanisms to dynamically identify and isolate key historical rounds ($C_{key}$). The pipeline involves:

1. Layer Selection: Analyzing attention maps from specific layers (e.g., the last four layers) to capture high-level semantics.
2. Token-to-Turn Aggregation: Aggregating token-level attention scores into turn-level scores using the mean, max, or sum.
3. Turn-to-Round Aggregation: Combining user and assistant turn scores into a round score. Crucially, the user turn (often composed of many speech tokens/codecs) is down-weighted by a ratio $\beta$ to prevent long audio sequences from dominating the score.
4. Context Scope: Selecting the top-$K$ scoring rounds as the key context $C_{key}$.

C. Context-Aware Decoding Mechanism

Once $C_{key}$ is identified, the decoding process contrasts the output distributions with and without this specific context:

1. Conditional Logits: Compute logits based on the full input context $C_n$.
2. Unconditional Logits: Compute logits based on an unconditional context $C_{unc}$, defined as the full input with the key context $C_{key}$ masked out. This represents the model's parametric prior when it cannot "see" the key context.
3. Logit Adjustment: The final logit $\tilde{l}_j$ for a token is calculated by adding a penalty weight $\alpha$ to the difference between the conditional and unconditional logits:
   $$\tilde{l}_j = \log P(A_j | C_{unc}) + \alpha [\log P(A_j | C_n) - \log P(A_j | C_{unc})]$$
   This formulation amplifies the shift caused by the key context, pushing the generation toward context-favored tokens and away from generic priors.

3. Key Contributions

The paper outlines three primary contributions:

1. Formalization of the Awareness-Adherence Gap: It is the first work to explicitly frame multi-round spoken dialogue failures not just as forgetting, but as a critical disconnect between a model's internal recognition of context and its active adherence to that context during generation.
2. Audio-Adapted CAD Method: The proposal of a decoding strategy that amplifies contextual signals by contrasting output distributions with and without dynamically isolated key context. This approach is modality-agnostic and extends to Speech-to-Speech generation.
3. Empirical Validation: Extensive evaluations on the Audio MultiChallenge benchmark demonstrate that the approach significantly improves strict context adherence, specifically in Semantic Memory and Self Coherence tasks, effectively reducing hallucinations in extended dialogues.

4. Experimental Results

The method was evaluated on the Audio MultiChallenge benchmark using three state-of-the-art models: MiMo-Audio-7B, Qwen3-Omni-30B, and Kimi-Audio-7B.

• Performance Gains: Integrating the proposed CAD consistently improved the Average Pass Ratio (APR) across all models.
  • Qwen3-Omni: Showed the most dramatic improvement, with an absolute increase of 13.30% in APR (Semantic Memory rose from 22.67% to 39.33%).
  • MiMo-Audio: Achieved an absolute average gain of 8.10%.
  • Kimi-Audio: Achieved an absolute average gain of 6.70%.
• Ablation Studies:
  • Context Selection: Using the "Whole History" as context resulted in a performance drop (21.04% vs. 26.01% baseline), confirming that naive context selection introduces noise. Precise isolation of key context is essential.
  • Hyperparameters: The optimal configuration involved averaging the last four attention layers, using mean aggregation for turn scoring, and a turn-to-round ratio ($\beta$) of 0.5 to balance audio token length.
  • Penalty Weight: Increasing the penalty weight ($\alpha$) improved performance up to a saturation point, while expanding the context scope ($K$) beyond 1 round generally degraded performance, suggesting that irrelevant turns can be detrimental if amplified.

5. Significance and Claims

The authors claim that their work provides a robust, model-agnostic strategy for enhancing the faithfulness of spoken dialogue systems. By shifting the focus from external retrieval to internal decoding interventions, the method successfully compels systems to transition from mere internal recognition of context to strict, context-faithful adherence. The results validate that addressing generation flaws through decoding-centric approaches is a viable path to resolving context failures in multi-turn spoken dialogues, offering a solution that does not require retraining or additional external modules.