Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs

1. Problem Statement

The rapid evolution of Large Language Models (LLMs) has led to the emergence of SpeechLLMs, which integrate speech as a native modality to process spoken language directly. These models aim to bypass traditional cascaded architectures (Automatic Speech Recognition $\rightarrow$ Machine Translation) and direct end-to-end models.

However, it remains an open question whether integrating speech directly into LLMs actually improves Speech-to-Text (ST) translation quality compared to established cascaded systems (which combine powerful Speech Foundation Models like Whisper with high-performing LLMs) or traditional direct models. Existing literature lacks a systematic comparison of these paradigms across diverse, real-world conditions such as noise, disfluencies, accents, and long-form speech.

2. Methodology: The Hearing-to-Translate Suite

The authors introduce Hearing to Translate, the first comprehensive test suite designed to rigorously benchmark Speech modality integration.

• Models Evaluated: The study compares 22 systems across three paradigms:
  1. Speech Foundation Models (SFMs): Standalone models (e.g., Whisper, SeamlessM4T, Canary, OWSM).
  2. Cascaded Systems: SFMs paired with LLMs (e.g., Whisper + Aya, Seamless + Tower+).
  3. SpeechLLMs: Models with native speech integration (e.g., Qwen3-Omni, Voxtral, Phi-4-Multimodal, DeSTA2).
• Benchmarks & Phenomena: The evaluation spans 16 benchmarks covering 13 language pairs and 9 challenging conditions:
  • Generic: Clean, standard speech.
  • Gender Bias: Evaluation of pronoun usage and stereotypical associations.
  • Accents/Dialects: Robustness to regional varieties (e.g., CommonAccent, ManDi).
  • Code-Switching: Handling mixed-language inputs.
  • Disfluencies: Handling hesitations, repetitions, and self-corrections.
  • Named Entities & Terminology: Accuracy in preserving proper nouns and domain terms.
  • Noise: Robustness to background and babble noise.
  • Emotion: Handling prosodic and affective cues.
  • Long-Form: Contextual consistency over multi-minute segments.
• Metrics:
  • Quality Estimation (QE): Due to the scarcity of reference translations in speech benchmarks, the study primarily uses xCOMETQE and METRICXQE (strict versions penalizing off-target outputs).
  • Gap Analysis: A unified gap formulation ($\Delta$) measures performance degradation across specific conditions (e.g., $\Delta_{noise}$, $\Delta_{disfluency}$).
  • Human Evaluation: A small-scale study on CoVoST2 using native speakers to validate automatic metrics and analyze error types (mistranslation, omission, over-translation).

3. Key Results

Overall Performance

• Cascades Remain King: Cascaded systems (SFM + LLM) remain the most reliable overall, delivering the strongest and most consistent translation quality across languages and conditions.
• SpeechLLM Potential: The best SpeechLLMs (specifically Voxtral and Qwen3-Omni) can match or even outperform the best cascades in specific settings, particularly when speech and language components are tightly integrated.
• Standalone SFMs Lag: Standalone SFMs generally perform worse than both cascades and SpeechLLMs, highlighting that the linguistic capabilities of LLMs are crucial for high-quality translation.

Phenomenon-Specific Findings

• Noise: SpeechLLMs are the most robust to noise. In cascaded systems, the ASR component often hallucinates under noise, and the LLM (lacking access to the original audio) propagates these errors. SpeechLLMs, having direct audio access, handle this better.
• Long-Form: Cascades (especially those with OWSM, Canary, or Voxtral) show near-zero degradation on long-form inputs. Many SpeechLLMs and SFMs suffer from extreme length degradation ($\Delta_{length} \approx 90\%$), though Voxtral is an exception due to its architectural design of re-concatenating chunk representations before LLM processing.
• Disfluencies: Voxtral, DeSTA2, and Whisper cascades are most robust. Interestingly, robustness is not solely driven by the encoder; how the encoder is integrated into the LLM matters (e.g., DeSTA2 and Voxtral outperform Qwen2-Audio despite sharing the Whisper encoder).
• Code-Switching: Cascaded Whisper and Voxtral achieve top performance. Standalone SFMs lag behind, indicating that both the encoder and the decoder (LLM) are critical for handling mixed languages.
• Gender Bias: Bias is primarily driven by the LLM decoder, not the speech module. Pairing ASR with specialized translation LLMs (like Tower+) significantly mitigates bias compared to general-purpose LLMs (like Gemma3).
• Accents: Accent robustness is primarily driven by the speech encoder. While some SFMs (like Seamless) excel, many SpeechLLMs struggle to generalize across dialects, showing high variance in performance.

Human Evaluation & Error Analysis

• Human scores closely mirror automatic QE metrics at the system level.
• Error Patterns:
  • Omissions are twice as frequent in SpeechLLMs compared to cascades.
  • Over-translation is significantly more prevalent in LLM-based systems.
  • Terminology errors remain a dominant failure mode across all paradigms.

4. Key Contributions

1. First Comprehensive Benchmark: Introduction of the Hearing-to-Translate Suite, providing a standardized, multi-dimensional evaluation framework for SpeechLLMs against 16 strong baselines.
2. Paradigm Comparison: A definitive analysis showing that while SpeechLLMs are rapidly evolving and can surpass cascades in specific robustness scenarios (noise, code-switching), cascaded architectures remain the state-of-the-art for general reliability, particularly for long-form and accent-robust translation.
3. Architectural Insights:
   • Demonstrated that LLM integration (either in a pipeline or native) is essential for high-quality ST; standalone SFMs are insufficient.
   • Identified that bias stems from the text-generation module, while accent robustness stems from the speech encoder.
   • Showed that noise robustness is a unique strength of native SpeechLLMs due to direct audio access.
4. Resource Release: The authors released the data, model outputs, and codebase to foster reproducibility.

5. Significance

This paper provides a critical reality check for the "SpeechLLM" hype. It establishes that while native integration offers distinct advantages in robustness (noise, code-switching) and potential efficiency, the modular cascade approach (SFM + Specialized LLM) currently offers the most robust and reliable solution for general-purpose speech translation.

The findings guide future research directions:

• For SpeechLLMs: Focus on improving long-context handling, reducing omission errors, and enhancing accent/dialect generalization.
• For Cascades: Leverage specialized LLMs (like Tower+) to mitigate gender bias and improve terminology handling.
• For Evaluation: Validates the use of strict Quality Estimation metrics as reliable proxies for human judgment in speech translation, enabling scalable evaluation where references are scarce.

Ultimately, the paper argues that the future of high-quality speech translation likely lies in hybrid approaches that leverage the acoustic robustness of advanced encoders and the linguistic flexibility of powerful LLMs, whether integrated natively or via cascades.