Technical Summary: EchoDistill

Problem Statement

Audio Large Language Models (ALLMs) are increasingly deployed in real-time applications such as in-car assistants and online meetings. However, these models are highly vulnerable to real-world noise (e.g., device artifacts and environmental interference), which distorts waveforms and degrades generation quality. Under severe noise conditions, models often misinterpret acoustic evidence, leading to semantic drift and hallucinations where responses drift toward language priors rather than remaining grounded in the audio input.

Existing robustness methods primarily rely on:

1. Waveform-level acoustic enhancement (e.g., STFT, Wavelet transforms).
2. Answer-level supervision.
3. Internal suppression of noise representations.

The authors argue that these runtime strategies fail to intrinsically enhance model robustness through the training process. Furthermore, some approaches incur a severe "alignment tax," degrading general capabilities and downstream task performance. The paper identifies a critical gap: correct reasoning trajectories often exhibit sparse audio grounding, where only a few critical acoustic segments drive the decision, making sequence-level rewards too coarse to identify token-level predictions grounded in reliable signals.

Methodology: EchoDistill

The authors propose EchoDistill, an alignment-based noisy-to-clean self-distillation framework. The core idea leverages an information gap: a "student" model processes noisy audio during inference, while a "teacher" model (the same architecture, but frozen) processes the corresponding clean audio during training.

Key Components:

1. Noisy Student Rollouts:
   The student samples a group of candidate responses from noisy audio inputs ($a^n_i$). These trajectories are evaluated using task rewards ($r^{(k)}_i$), exposing the model to linguistically plausible but acoustically unsupported errors.

2. Noisy-to-Clean Evidence Alignment:
   To address the coarseness of sequence-level rewards, EchoDistill aligns the noisy student's token-level distribution with the clean teacher's semantic preferences.

   • A guidance response ($\tilde{y}_i$) is selected (generated by the clean teacher or selected from student candidates).
   • The loss ($L_{distill}$) computes the masked KL divergence between the clean teacher's next-token distribution ($q_{\phi,t}$) and the noisy student's distribution ($p_{\theta,t}$) on the same continuation.
   • This encourages the noisy student to preserve the clean-audio semantic preference without forcing it to copy a single sentence.

3. Audio-Aware Reward Shaping:
   The framework differentiates between responses that are merely correct and those grounded in audio evidence.

   • An audio-aware similarity score ($s_i$) is derived from the distillation loss.
   • This score acts as a reward bonus for candidates that receive a positive task reward, effectively shaping the reward signal to favor trajectories that align with clean semantic evidence.
   • Incorrect candidates do not receive this bonus, even if their distribution is close to the teacher.

4. Optimization via GRPO:
   The framework utilizes Group Relative Policy Optimization (GRPO). The clean teacher serves as a detached reference scorer for the sampled responses, allowing the calculation of group-relative advantages without an additional value model. The final loss combines the policy loss and the distillation loss.

5. Inference:
   At inference time, the teacher branch, clean audio, and reward computation are removed. The model operates as a single student ($\pi_\theta$) processing noisy audio, preserving the original inference cost.

Key Contributions

1. Analysis of Audio Grounding: The authors demonstrate that correct reasoning trajectories exhibit inherent sparsity in acoustic evidence. Fine-grained token-level semantic correction is identified as fundamentally necessary for reliable audio reasoning, as correctness alone does not guarantee strong audio grounding.
2. EchoDistill Framework: A novel self-distillation method that aligns noisy-audio student inference trajectories with privileged clean-audio teacher semantics at the token level. This enables the model to remain grounded in reliable acoustic evidence rather than drifting toward language priors.
3. Empirical Validation: Extensive experiments across multiple ALLMs (Qwen-Omni, MiniCPM-o, StepAudio) and domains (Music, Sound, Speech) demonstrate consistent improvements in robustness, particularly in the challenging Sound and Speech domains.

Experimental Results

Experiments were conducted on datasets including MMAR (training) and MMAU (evaluation) under strong noise (SNR = -10 dB).

• Overall Performance: Compared to the strongest baseline, EchoDistill achieved an average improvement of 4.18% in Generation Success Rate (GSR) under strong noise.
• Domain Specificity:
  • Sound & Speech: The method showed the most significant gains in these domains, where noise obscures fine-grained phonetic and event details. For instance, on Qwen-Omni in the Sound domain, GSR improved from 68.25% to 75.33%.
  • Music: While signal-processing baselines were competitive in accuracy, EchoDistill demonstrated superior generative stability (e.g., 86.00% GSR on Step-Audio2 vs. 79.00% for STFT).
• Ablation Studies:
  • Distill-only vs. GRPO-only: The noisy-to-clean distillation component provided the core semantic anchor, improving GSR by 2.58% over GRPO-only.
  • Synergy: The full EchoDistill framework outperformed both variants, showing that policy optimization and distillation are complementary. The full model improved GSR by 4.53% over the GRPO-only variant on average.
• Semantic Drift Mitigation: Training dynamics showed that EchoDistill progressively aligns noisy-input generation with clean-audio semantic behavior, effectively suppressing semantic drift without simply removing waveform noise.

Significance and Claims

The paper claims that EchoDistill significantly improves the semantic reliability and task performance of Audio LLMs under complex noise without introducing additional inference costs.

• Mechanism: The method addresses the root cause of hallucination in noisy conditions by training the model to align its noisy-input output distribution with the clean-audio semantic distribution.
• Scalability: The framework is compatible with external front-end denoising methods (e.g., SEEN), as it operates at the semantic generation level. The combination of front-end denoising and noisy-to-clean alignment offers a scalable path for robustness.
• Limitations: The authors acknowledge that the framework depends on the reliability of the clean-audio teacher and requires extra computation during training (rollouts, scoring, alignment). Currently, it focuses on single-audio understanding.

In summary, EchoDistill proposes a paradigm shift from purely acoustic enhancement to semantic alignment, ensuring that ALLMs can recover coherent semantic representations even when the acoustic signal is severely corrupted.