Technical Summary: DOA - Training-Free Decoder-Only Attention Policy for Long-Form Simultaneous Translation

Problem Statement

Simultaneous speech-to-text translation (SimulST) requires a streaming policy to decide when to read source audio and when to write target tokens, balancing translation quality against latency. While state-of-the-art systems typically utilize attention-based encoder-decoder (AED) architectures where cross-attention provides explicit alignment signals to guide these decisions, recent advances have shifted toward Speech Large Language Models (SpeechLLMs). These models employ decoder-only architectures that rely solely on self-attention, lacking explicit cross-attention scores.

This architectural shift raises a critical question: Does the self-attention mechanism in decoder-only SpeechLLMs contain sufficiently stable alignment signals to guide streaming decisions? Furthermore, existing approaches for adapting SpeechLLMs to SimulST often rely on training-based fine-tuning or fixed heuristic policies (e.g., wait-k) and have not been rigorously validated in long-form settings where models must process continuously growing acoustic and textual contexts.

Methodology: Decoder-Only Attention (DOA)

To address these gaps, the authors propose Decoder-Only Attention (DOA), a training-free policy that enables long-form simultaneous translation using off-the-shelf SpeechLLMs without retraining.

1. Deriving Proxy Alignment

Since decoder-only models process speech and text within a single autoregressive sequence, the authors derive a proxy cross-attention matrix directly from the decoder's self-attention weights.

• During autoregressive generation, self-attention weights $A^{(l,h)}$ are extracted for decoder layer $l$ and head $h$, covering a sequence of $S$ audio tokens and $T$ generated text tokens.
• Because audio tokens occupy the sequence prefix, the authors isolate the attention directed toward the acoustic context: $\tilde{A}^{(l,h)} = A^{(l,h)}[:, : S]$.
• This results in a matrix $\tilde{A} \in \mathbb{R}^{T \times S}$, analogous to the cross-attention matrix in AED models.
• The final proxy alignment $\bar{A}$ is obtained by averaging across selected layers and heads.

2. Streaming Decision Policy

The policy uses the derived alignment to determine when to emit tokens:

• For each generated token $t$, the aligned audio position $a_t$ is identified as the position receiving the highest attention score ($a_t = \arg\max_s \bar{A}_{t,s}$).
• A cutoff hyperparameter $f$ defines the number of most recently received audio frames considered "acoustically unstable."
• Tokens aligned to positions within the last $f$ frames are withheld, as their supporting acoustic evidence may evolve with future input.
• Only tokens satisfying $a_t < S - f$ are committed to the output.

3. Long-Form Adaptation

To handle unbounded audio streams, DOA implements history selection mechanisms to prevent memory overflow:

• Textual History: Two strategies are evaluated: (i) Fixed Words (retaining the last $N$ tokens) and (ii) Punctuation (retaining text after the most recent strong punctuation mark).
• Acoustic History: The proxy alignment is used to prune audio frames. Frames aligned exclusively with discarded textual history are removed, while frames aligned with the preserved prefix and new hypotheses are retained.
• Edge Case Handling: If attention-based pruning fails to discard frames and the audio history exceeds a predefined duration (e.g., 90s or 120s), the oldest frames are truncated to maintain bounded memory.

Key Contributions

1. First Training-Free Policy for Decoder-Only SimulST: DOA is the first policy to enable long-form simultaneous translation with off-the-shelf decoder-only SpeechLLMs by leveraging self-attention as a proxy alignment signal.
2. Validation of Self-Attention Stability: The work empirically investigates and confirms that decoder self-attention provides sufficiently stable alignment information to drive streaming decisions, challenging the assumption that explicit cross-attention is necessary.
3. Long-Form Capability: Unlike prior work focused on pre-segmented utterances, DOA is designed for continuous, long-form audio streams with dynamic context pruning.
4. Model Agnosticism: The framework requires only access to decoder self-attention weights, making it applicable to any SpeechLLM without task-specific adaptation.

Experimental Results

The authors evaluated DOA on Phi4-Multimodal and Qwen3-Omni for English-to-German and English-to-Italian translation, comparing against the StreamAtt baseline applied to the AED model SeamlessM4T.

• History Selection: The Punctuation strategy consistently outperformed Fixed Words strategies in both quality (COMET) and stability across latency regimes. This contrasts with findings in AED systems, suggesting decoder-only models benefit more from sentence-level textual continuity.
• Layer and Head Analysis: Averaging the proxy alignment across all layers and heads yielded the most robust performance. Selecting specific layers led to instability or failure, while specific heads offered only minimal gains.
• Performance:
  • DOA-equipped SpeechLLMs achieved translation quality (COMET scores up to 0.828 for Qwen3-Omni en-it) close to offline decoding benchmarks without retraining.
  • The models demonstrated a favorable latency-quality trade-off, consistently outperforming the StreamAtt AED baseline.
  • Qwen3-Omni covered a broad operating range (400ms to 3.8s LongYAAL) with high quality, while Phi4-Multimodal offered easier latency management.

Significance and Claims

The paper claims that DOA successfully repurposes offline SpeechLLMs for simultaneous translation without retraining, demonstrating that decoder self-attention can serve as an effective alignment signal for streaming decisions. The authors emphasize that this approach generalizes across different model families (dense vs. MoE) and architectures, providing a simple yet effective mechanism for long-form SimulST.

The work highlights that attention-based alignment is not exclusive to encoder-decoder models and can be adapted to decoder-only architectures. However, the authors maintain modesty regarding their scope, noting limitations such as evaluation restricted to English source speech, the use of Latin-script target languages, and the lack of computationally aware latency comparisons due to heterogeneous GPU environments. They explicitly state that generalization to logographic or morphologically rich languages remains an open question.