A Pocket Offline Model for Simultaneous Speech Translation as CUNI Submission to IWSLT 2026

This paper presents the CUNI submission to the IWSLT 2026 Simultaneous Speech Translation Shared Task, which adapts the 1B-parameter offline Canary model with the AlignAtt policy to achieve high-quality, multilingual simultaneous translation across Czech, English, German, and Italian with low computational requirements.

The Gist

Imagine you have a very smart, multilingual translator named Canary. This translator is incredibly talented at listening to a whole speech and then writing down the translation perfectly. However, there's a catch: Canary is used to waiting until the speaker has finished their entire sentence before it starts working. It's like a waiter who refuses to take your order until you've finished reading the entire menu, even if you just want to order a coffee.

The team at Charles University (CUNI) wanted to teach Canary how to be a simultaneous translator—someone who can listen and translate at the same time, word-by-word, just like a human interpreter.

Here is how they did it, using some simple analogies:

1. The "Stop-and-Go" Strategy (AlignAtt)

To make Canary work in real-time, the team didn't retrain the whole translator from scratch. Instead, they gave Canary a new set of rules called AlignAtt.

Think of Canary's brain as having a pair of "attention glasses." When the translator hears a new chunk of audio, it looks at the sentence it's building. The AlignAtt rule says: "As soon as you hear a sound that matches a specific point in the audio you just heard, stop writing the rest of the sentence."

It's like a game of "telephone" where you only pass on the words you are 100% sure of. If the translator starts to guess a word based on audio that hasn't fully arrived yet, the system cuts that guess off. This prevents the translator from "hallucinating" (making things up) or repeating itself.

2. The "Pocket-Sized" Powerhouse

One of the coolest things about this project is the size of the translator. Most high-quality translators are like massive supercomputers that need a whole data center to run. Canary, however, is a 1-billion-parameter model.

The authors call this a "Pocket Offline Model." Imagine fitting a translator the size of a smartphone app into your pocket. It's small enough to run on a regular phone without needing to send data to a giant server in the cloud. This means you could use it even if you have no internet connection, and it would be much faster because it doesn't have to wait for a signal to travel back and forth.

3. The "Noise-Canceling" Headphones

Real-world audio is messy. People cough, cars honk, and rooms echo. To handle this, the team added a "noise filter" (called Silero VAD). Think of this as a pair of high-tech noise-canceling headphones for the translator. It ignores the silence and the background noise, only paying attention when a human voice is actually speaking. This keeps the translator from getting confused or trying to translate the sound of a door slamming.

4. The Race Against Time (The Results)

The team tested their new system in a simulated race against other top translators (the "baselines") for three language pairs: Czech to English, English to German, and English to Italian.

• The Setup: They tested two scenarios: a "High Latency" race (where the translator can wait a bit longer for more accuracy) and a "Low Latency" race (where speed is everything).
• The Outcome:
  • Speed vs. Quality: Their system was a champion in both categories. In the "High Latency" race, it translated significantly better than the organizers' best existing systems (improving scores by 4 to 8 points).
  • Beating the Competition: Even in the "Low Latency" race (where it had to be super fast), it held its own, often beating the previous best systems.
  • The Sliding Window: They also compared their method to another way of making offline models work in real-time (called "Sliding Window," which is like constantly re-reading a paragraph to fix mistakes). Their "Stop-and-Go" (AlignAtt) method was much better, producing higher quality translations with less delay.

5. What They Learned (and What They Didn't)

The team concluded that taking a powerful, offline translator and giving it a "real-time rulebook" is a winning strategy. It's simple, effective, and doesn't require expensive training.

However, they did hit a small snag. They tried to give the translator "context clues" (like telling it, "This is a political meeting, so use formal words"), but when they tried to combine these clues with their new "Stop-and-Go" rules, the translator got confused and stopped working. They suspect this is because the translator wasn't trained on enough examples of this specific combination.

The Bottom Line

The CUNI team successfully turned a "wait-for-the-end" translator into a "listen-and-translate-as-you-go" translator that fits in your pocket. They proved that you don't need a giant supercomputer to get high-quality, simultaneous translation; a small, smart model with the right rules can do the job just as well, if not better, than the big systems.

Technical Summary

Technical Summary: A Pocket Offline Model for Simultaneous Speech Translation

Problem Statement
The paper addresses the challenge of deploying simultaneous speech translation (SST) on resource-constrained, "pocket" devices. While offline speech-to-text translation models often offer superior quality, multilingual support, and robustness, they are typically not designed for real-time, incremental processing. Conversely, dedicated simultaneous models often require computationally expensive training or specific architectures that limit their scalability. The authors aim to bridge this gap by repurposing a high-quality, lightweight offline model (Canary-1B-v2) for simultaneous translation using state-of-the-art policies, thereby creating a system that is both computationally efficient (1B parameters) and capable of supporting 25 source and 25 target languages.

Methodology
The proposed system adapts the Canary-1B-v2 model, a multitask speech transcription and translation model, for simultaneous operation using the AlignAtt policy. The implementation involves several key technical components:

1. Model Adaptation in Nemo: Since the NeMo framework did not natively support the "decoder forced prefix injection" required for incremental output generation, the authors contributed modifications to the Nemo speech framework. These changes allow an optional initial prompt (containing language info and forced prefixes) to be passed to the decoder. Additionally, they fixed a bug in the cross-attention outputs for the beam decoder strategy to ensure deterministic dimension outputs and correct token mapping.
2. Simultaneous Policy (AlignAtt): The system utilizes AlignAtt, which leverages the decoder's cross-attention mechanism. It processes audio incrementally, omitting the suffix of the hypothesis once the first token attends to a predefined frame threshold. This prevents the model from generating text based on future audio chunks.
3. Processing Pipeline:
   • VAD: Silero Voice Activity Detection (VAD) is employed to filter non-voice segments, reducing noise and hallucinations.
   • Buffering: The system maintains a source audio buffer (accumulating the latest 30 seconds of raw audio) and a forced decoding target buffer (containing the stable part of the hypothesis).
   • Incremental Decoding: As new audio chunks arrive, the model decodes the target text. If the most attended source frame is near the end of the current chunk (but the chunk is not final), decoding continues. If a word falls within the AlignAtt frame threshold, it is removed from the output to allow for future refinement.
   • Finalization: When the final audio chunk is processed, AlignAtt is disabled, and the full generated sequence is output.
4. Evaluation Frameworks: The system was evaluated using SimulStreaming (primary) and Simulstream (for computationally aware requirements). Hyperparameters (MinChunkSize and Frames) were tuned via grid search to meet specific latency thresholds (LongYAAL < 2s for low latency, < 4s for high latency).

Key Contributions

• First Implementation of Canary in Simultaneous Mode: The paper presents the first application of the Canary model to simultaneous translation, moving beyond the previously used sliding window approach.
• Nemo Framework Enhancements: The authors contributed specific code to the NeMo ecosystem to enable forced-prefix decoding and correct cross-attention handling for incremental generation with Canary.
• Demonstration of Efficiency: The system achieves high-quality translation with only 1 billion parameters, making it a viable candidate for on-device deployment without external APIs.
• Multilingual Capability: The system supports direct translation across 25 source and 25 target languages, covering the IWSLT 2026 task pairs (Czech-English, English-German, English-Italian).

Results
The system was evaluated on the IWSLT 2026 development sets for Czech-to-English, English-to-German, and English-to-Italian.

• Performance vs. Baselines: The Canary-AlignAtt system outperformed the organizers' cascade baseline (Qwen3-ASR + Qwen3-MT) across all metrics (BLEU, chrF, XCOMET-XL) in both high and low-latency regimes.
  • English-to-German/Italian: In the high-latency regime, the system improved BLEU by over 4 points (German) and over 6 points (Italian) compared to the organizers' best baseline.
  • Czech-to-English: The system improved BLEU by nearly 8 points (high latency) and over 5 points (low latency) compared to the SimulStreaming Whisper baseline.
• Comparison with Sliding Window: The AlignAtt policy significantly outperformed a sliding window implementation of Canary, showing gains of over 8 BLEU points (English-German) and over 7 BLEU points (English-Italian).
• Offline vs. Simultaneous: The simultaneous approach yielded better results than running the model in standard offline mode on long audios, suggesting the incremental processing strategy effectively mitigates issues like hallucination on partial sentences.

Significance and Claims
The authors conclude that repurposing a strong offline model for simultaneous translation is a simple, effective, and high-performing strategy. They claim their system serves as a strong baseline for future research, demonstrating that high-quality, multilingual simultaneous translation can be achieved on lightweight models (1B parameters) suitable for edge devices. The results suggest that such systems can compete with or surpass larger, server-side cascade pipelines while eliminating the need for external APIs. The paper positions this work as a practical step toward "pocket-device" deployment, noting that further quantization could enable full pipeline execution on smartphones.

Limitations
The authors note that they were unable to find an optimal setup for injecting in-domain context via the decoder prompt; attempting to use both forced prefixes and context simultaneously caused the model to stall. They hypothesize this may be due to a lack of such data in the training set. Additionally, while the system is designed for computationally unaware simulations, the authors acknowledge that full quantization for constrained devices remains a future step.