💬 NLP

AlignAtt4LLM: Fast AlignAtt for Decoder-Only LLMs at IWSLT 2026 Simultaneous Speech Translation Task

This paper introduces AlignAtt4LLM, a novel simultaneous speech translation system that adapts the AlignAtt policy for decoder-only LLMs by employing forced alignment, selective attention head selection, and query/key capture, achieving state-of-the-art results for English-to-German and English-to-Italian translation on the IWSLT 2026 development set.

Original authors: Quentin Fuxa, Dominik Macháček

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Quentin Fuxa, Dominik Macháček

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A Real-Time Translation Relay Race

Imagine a relay race where two runners are passing a baton back and forth to translate a live speech from English into German, Italian, or Chinese.

Runner 1 (The Listener): This is an AI that listens to the audio and writes down what it hears in real-time.
Runner 2 (The Translator): This is a powerful AI that reads what the listener wrote and translates it into the target language.

The challenge? The listener is still hearing the speech while the translator is already writing. If the translator starts writing too early, they might guess wrong and have to erase their work (which looks messy to the audience). If they wait too long, the translation lags behind the speaker.

This paper introduces a new system, AlignAtt4LLM, that acts as a smart referee between these two runners. It tells the translator exactly when it is safe to commit to a word and when they should wait for more information.

The Problem: The "Black Box" Translator

In the past, translation systems were built like a factory with two distinct rooms: one for understanding (Encoder) and one for speaking (Decoder). The "referee" could easily peek into the factory to see how the two rooms were talking to each other to decide when to speak.

However, modern AI models (called Decoder-Only LLMs) are like a single, giant black box. They don't have a separate "understanding" room. They just read a prompt and write an answer. Because the "referee" can't peek inside the black box to see the connection between the English words and the German words, it usually has to guess. This often leads to hesitation or mistakes.

The Solution: The "Smart Window" and the "Flashlight"

The authors solved this by giving the referee two special tools to work with this black box:

1. The "Explicit Window" (Prompt Layout)

Instead of letting the AI guess where the English words are, the system forces the AI to write the English transcript inside a specific, clearly marked "window" in its instructions.

Analogy: Imagine the translator is reading a book where the original English text is highlighted in yellow. The referee doesn't need to guess where the English is; it just looks at the yellow highlight.

2. The "Flashlight" (Selective Attention Replay)

Inside the AI's brain, there are thousands of tiny "attention heads" (think of them as little flashlights) that decide which words are important. Most of these flashlights are looking at the wrong things (like the instructions or the words the translator already wrote).

The Innovation: The authors figured out which specific flashlights (only 8 out of hundreds) are actually looking at the English-to-German connection.
The Trick: They built a system that captures the exact beam of those specific flashlights after the AI has done its work, but before the result is finalized. They then use a "fast replay" to reconstruct just that tiny slice of information to check if the translator is ready to speak.
Analogy: It's like a security guard who doesn't need to see the whole building to know if a door is open. They just check the specific peephole that looks out the front door.

How the System Works (Step-by-Step)

Listening: The system hears a chunk of audio and converts it to text with timestamps (e.g., "cat" ends at 0.7 seconds).
Drafting: The translator AI quickly writes a few words of translation (a "draft").
The Check: The referee looks at the "Flashlight" data. It asks: "Is the translator focusing on words that have already been spoken, or are they guessing about words that haven't been said yet?"
The Decision:
- Safe: If the focus is on words already spoken, the referee says, "Go ahead, publish those words!"
- Unsafe: If the focus is on future words, the referee says, "Stop! Wait for more audio."
Result: The translator publishes a steady stream of text that never has to be erased (no "flickering").

The Results: Fast and Accurate

The team tested this on the IWSLT 2026 competition (a simulated real-time translation contest).

Speed: The system is very fast. It can start translating within about 2 seconds of hearing the speech.
Quality:
- For German and Italian, it beat the existing "baseline" systems (the standard competitors) in both speed and accuracy.
- For Chinese, the results were mixed. It was competitive in some ways, but the baseline was still slightly better. The authors suggest this is because the specific AI model they used (Gemma-4) isn't the best at Chinese yet, but the method they invented works fine and could be swapped with a better Chinese model later.

Why This Matters

This paper proves that you don't need to build a special, slow translation AI from scratch to do real-time work. You can take a powerful, general-purpose AI (like a smart chatbot), give it a specific "window" to look at, and use a clever "flashlight" trick to make it work in real-time without losing its smarts.

In short: They figured out how to make a "black box" translator play by the rules of a live relay race, ensuring it never trips over its own feet.

Technical Summary: AlignAtt4LLM

Problem Statement
The paper addresses the challenge of adapting offline, decoder-only Large Language Models (LLMs) for simultaneous speech translation (SST). Traditional simultaneous policies like AlignAtt (Papi et al., 2023, 2024) rely on encoder-decoder cross-attention to determine when a source prefix is sufficient for generating a target token. However, decoder-only LLMs lack this explicit cross-attention mechanism, as they process source and target text within a single causal self-attention stream. Consequently, applying standard AlignAtt to decoder-only models is not directly feasible. Furthermore, existing simultaneous policies for decoder-only models, such as LocalAgreement, often introduce higher latency and do not leverage internal model attention signals. The authors aim to bridge this gap by enabling a high-quality, offline decoder-only LLM to operate in a low-latency, simultaneous mode without retraining the model for specific language directions.

Methodology
The proposed system, AlignAtt4LLM, is a synchronous cascade consisting of two main components:

ASR Component: Uses Qwen3-ASR (1.7B) for transcription and Qwen3-ForcedAligner (0.6B) to assign word-level start and end times. The system operates in a chunk-based manner, re-transcribing the live audio tail at every chunk boundary to stabilize hypotheses.
MT Component: Uses Gemma-4 E4B-it served via vLLM. The core innovation lies in adapting the AlignAtt policy to this decoder-only architecture through four specific technical mechanisms:
- Explicit Source Span in Prompt: Since decoder-only models lack cross-attention, the authors structure the causal prompt to explicitly separate the source transcript from the target history. The prompt concatenates the system prompt, the live source transcript ( $s^{(k)}$ ), the translation instruction, the accepted target prefix ( $y_{1:m_k}$ ), and the current draft ( $\hat{y}^{(k)}$ ). This allows the policy to identify the source span within the self-attention matrix.
- Offline Selection of Translation-Specific Heads: Not all self-attention heads contribute to translation alignment. Following Liu et al. (2026), the authors perform an offline calibration using held-out parallel text and GPT-5-mini alignments to score heads via Translation Score (TS). They retain the top 8 heads per language pair, which are concentrated in the later layers of the backbone.
- Selective $qk$-fast Replay: The deployed vLLM runtime fuses attention kernels, hiding the full $n \times n$ attention matrix. To access the necessary alignment signals, the authors implement a runtime observer that captures the exact query ( $Q$ ) and key ( $K$ ) tensors consumed by the attention module after normalization and rotary position embedding (RoPE). Instead of reconstructing the full matrix, the system performs a "selective $qk$-fast" replay, recomputing only the specific block of attention scores between the draft tokens and the source span ( $\hat{A}_{D, S^{(k)}}$ ). This ensures bit-identical outputs to the unpatched model while avoiding the computational cost of full attention reconstruction.
- Runtime Query/Key Capture: To ensure the observer survives graph optimization (e.g., dead-code elimination), a zero-valued sentinel dependency is added to the attention output. This keeps the capture path in the live fan-in of the graph without altering numerical outputs.

Acceptance Policy
The policy scans drafted tokens from left to right, accepting a prefix as long as the alignment signal remains within the "accessible source frontier." The decision is based on:

Source-Frontier Check: The argmax of the reconstructed attention row must not exceed the number of accessible source words plus a margin ( $b=1$ ).
Provenance Mass: The sum of attention mass directed toward accessible source tokens ( $\pi^{acc}_t$ ) is calculated.
Stabilization: The system applies online prefix statistics (Welford) and a median filter to stabilize the alignment peak before making a decision.

Key Contributions

First Application of AlignAtt to Decoder-Only LLMs: The paper demonstrates that AlignAtt can be successfully adapted to decoder-only architectures by recovering the alignment signal from self-attention rather than cross-attention.
Runtime-Compatible Implementation: The authors propose a method to capture and replay attention signals in a production-grade, fused-kernel environment (vLLM) without modifying the model weights or requiring full attention matrix materialization.
Efficient Head Selection: The system shows that a sparse set of offline-calibrated attention heads is sufficient to drive the simultaneous policy, significantly reducing the computational overhead of the replay process compared to using all heads.

Results
Evaluated on the IWSLT 2026 Simultaneous Speech Translation task (English to German, Italian, and Chinese):

Performance: AlignAtt4LLM outperforms the provided no-context baselines in both low-latency (~2s CU-LongYAAL) and high-latency (<4s) regimes for English-to-German and English-to-Italian.
Chinese Translation: Results for English-to-Chinese are mixed; while chrF scores are competitive, BLEU and XCOMET-XL scores slightly favor the baseline. The authors attribute this partly to the Gemma-4 backbone's weaker performance on Chinese rather than the policy itself.
Latency vs. Quality: The system achieves a stable low-latency operating point while offering a high-latency regime with significantly improved translation quality.
Efficiency: The selective replay using only 8 heads incurs minimal overhead (median 25.4 ms/token) compared to eager implementations (63.7 ms/token), confirming the practicality of the approach.

Significance and Claims
The paper claims that AlignAtt4LLM is the first system to successfully apply the AlignAtt policy to a decoder-only LLM. It argues that the method is not tied to the specific Gemma-4 model; because the policy relies only on a deterministic prompt layout, calibrated attention heads, and query/key capture, it can be ported to stronger, translation-focused decoder-only backbones (e.g., HY-MT-1.5, MiLMMT-46) to potentially improve performance on non-European languages. The work demonstrates that high-quality offline models can be effectively repurposed for simultaneous translation without dedicated training, provided the alignment signal can be correctly extracted from the self-attention substrate.