Technical Summary: Streaming Speech-to-Text Translation with a SpeechLLM

Problem Statement

Current systems for translating speech into text in another language typically rely on a cascaded architecture: a speech recognizer produces a transcript, which is then fed into a text-to-text translation system. While recent advancements have enabled "SpeechLLMs" (Large Language Models conditioned on speech audio) to perform end-to-end translation, existing implementations fail to operate in a true real streaming fashion.

Most existing SpeechLLMs either wait for a complete utterance before generating output or rely on fixed policies (such as "wait-k") that emit tokens at fixed intervals based on audio chunks. These fixed policies are ill-suited for real-world scenarios where speech speed, pauses, and background noise vary. Specifically:

• Fixed Policies: If a microphone opens before speech begins, or if a speaker hesitates, fixed policies force the model to hallucinate output or fall behind, leading to degraded translation quality or missed phrases.
• Energy Efficiency: Existing streaming models that output "wait" tokens to request more audio often incur high energy costs on edge devices because they require frequent, full evaluations of the large language model (LLM).

Methodology

1. The "Intermixed" Architecture

The paper proposes a novel architecture where the LLM itself learns a wait policy. Instead of a separate module deciding when to wait, the LLM is trained to output two types of tokens:

• Text Tokens: The actual translation output.
• Wait Tokens ($W$): Signals indicating the model needs more audio before proceeding.

In this "intermixed" model, the input sequence alternates between encoded speech vectors (from a speech encoder) and text tokens, mirroring the output sequence which interleaves text tokens and wait tokens. The model learns to decide, based on the partial audio seen so far, whether it has sufficient information to emit a translation token or must wait for the next audio chunk.

2. Phrase-Level Alignment for Training

A critical challenge in training streaming systems is determining when to emit specific words. Standard word-level alignment methods (e.g., SimAlign) often fail for linguistically distant language pairs (e.g., English and Korean) because concepts do not map one-to-one.

• Proposed Solution: The authors generate phrase-level alignments using an LLM with few-shot prompting.
• Mechanism: The source and target texts are aligned phrase-by-phrase. All words in a target phrase are aligned to the final word of the corresponding source phrase. This ensures the model only emits a target word after the necessary source audio has been heard, preventing premature generation.
• Training Objective: The model is trained to maximize the likelihood of a single, optimal step sequence (a mix of wait and emit decisions) derived from these alignments.

3. Early-Exit Wait Policy for Energy Efficiency

To address the energy cost of frequent LLM calls caused by wait tokens, the authors introduce an early-exit wait policy.

• Mechanism: A lightweight head (derived from the first few layers of the LLM) makes a preliminary decision: Wait ($W$) or Emit ($E$).
• Operation:
  • If the policy decides to Wait, the full LLM is not evaluated, saving significant computation.
  • If the policy decides to Emit, the full LLM is evaluated to generate zero or more text tokens followed by a wait token.
• Trade-off: This allows for a tunable balance between latency and energy consumption without compromising translation quality, as the full LLM retains the final authority to override an "emit" decision if it determines more audio is needed.

4. Evaluation Metrics

The paper critiques standard latency metrics like "average lagging" for their inconsistencies with streaming speech (e.g., handling post-audio emissions and short utterances). Instead, it proposes Average Logical Latency, which measures the average time a token is emitted relative to a theoretical baseline where tokens are generated evenly across the audio duration.

Key Results

Experiments were conducted on English-to-French, English-to-Korean, and English-to-German translation tasks using datasets like CoLiMu and Fleurs.

• Latency vs. Quality: The intermixed SpeechLLM achieves a latency of 1–2 seconds while maintaining translation quality (COMET score) close to non-streaming offline baselines. It significantly outperforms existing fixed-policy systems (Bestow with wait-k) and dynamic policies (AlignAtt) in the latency-quality trade-off.
• Robustness: Unlike fixed-policy models, which suffer catastrophic failure (hallucinations) when input contains silence or noise (tested on the "SilFleurs" dataset), the intermixed model adapts dynamically, maintaining consistent quality.
• Energy Efficiency: The early-exit policy reduces the number of LLM calls. For English-to-Korean, adjusting the wait penalty reduced energy use from 1.55 to 1.16 tokens per output token while keeping latency stable at ~2.3 seconds.
• Alignment Strategy: The proposed phrase-level alignment significantly outperforms word-level alignment (SimAlign). Word-level supervision led to premature generation and degraded quality, whereas phrase-level supervision provided reliable timing labels.

Significance and Claims

The paper claims to address the gap in real-time, adaptive streaming for SpeechLLMs. Its primary contributions are:

1. A Learned Wait Policy: Moving away from rigid, fixed policies to a model that learns when to wait based on the audio content, enabling true adaptability to real-world speech variations.
2. Phrase-Level Supervision: A method for generating training data that handles the complexities of cross-lingual phrase structures, which is essential for high-quality streaming translation between diverse languages.
3. Energy-Aware Streaming: The introduction of the early-exit policy demonstrates that streaming systems can be optimized for edge devices, reducing energy consumption without sacrificing the low-latency benefits of streaming.

The authors conclude that their system offers a superior balance of low latency, high translation quality, and robustness compared to existing fixed or dynamic policies, making it a viable candidate for real-world deployment in applications like mobile translation.