Streaming Translation and Transcription Through Speech-to-Text Causal Alignment

1. Problem Statement

Simultaneous Machine Translation (SiMT) aims to translate speech in real-time as it is spoken, balancing accuracy (quality) and latency (delay). Traditional SiMT approaches rely on:

• Cascaded Architectures: An offline speech-to-text (ASR) model followed by a machine translation (MT) model.
• External Policies: Heuristics (e.g., "wait-k") or learned controllers that decide when to read more input (READ) and when to generate output (WRITE).

Key Challenges:

• Syntactic Divergence: In language pairs like English (SVO) and Japanese (SOV), the verb often appears at the end of the sentence. A translator must buffer the subject and object while waiting for the verb to appear to avoid hallucinating or producing grammatically incorrect translations.
• Policy Limitations: Fixed heuristics (like wait-k) struggle with variable sentence structures, often forcing premature translation (low quality) or excessive waiting (high latency).
• Modular Complexity: Existing systems often require separate modules for segmentation, ASR, and translation, complicating inference on unbounded streams.

2. Methodology: The Hikari Model

The authors propose Hikari, a policy-free, fully end-to-end model that unifies translation and transcription into a single probabilistic framework.

A. Core Architecture

• Base: Built upon the Whisper encoder-decoder Transformer architecture.
• Causal Alignment: The model is modified to process audio incrementally.
  • Encoder: Uses a lower-triangular causal mask to prevent attending to future audio.
  • Decoder: Uses a dilated causal mask in cross-attention to ensure strict temporal monotonicity (the decoder cannot attend to future encoder keys).
• Unified READ/WRITE Mechanism:
  • Instead of an external controller, the decision to read or write is encoded directly into the token probability distribution.
  • WAIT Token: A special token (mapped to a rare character like ~) represents a READ action (buffering more audio).
  • Text Tokens: Represent WRITE actions (committing translation).
  • The model learns to emit WAIT tokens when semantic ambiguity exists (e.g., waiting for a verb in an SOV language) and switches to text tokens once the context is resolved.

B. Decoder Time Dilation (DTD)

To address the imbalance where the model might generate excessive WAIT tokens (up to 90% of the sequence), the authors introduce Decoder Time Dilation.

• Mechanism: The decoder generates one token for every $D$ audio embeddings (where $D$ is a dilation factor).
• Optimization:
  • If $D=1$, the model generates too many WAIT tokens, causing training instability and inference latency.
  • If $D$ is too large (e.g., 10), the temporal resolution is too coarse, and target tokens may not fit in the context window for fast speech.
  • Result: The authors found $D=4$ to be optimal, balancing the proportion of WAIT tokens, causal alignment precision, and real-time factor (RTF).

C. Training Strategy

1. Synthetic Data Generation: Since high-quality simultaneous interpretation datasets are rare, the authors created a pipeline:
   • Transcribe audio with word-level timestamps.
   • Translate sentences using an LLM with context.
   • Causal Alignment: Use word embeddings to align source and target words. Target words are delayed (interleaved with WAIT tokens) until the corresponding source audio has been fully processed, ensuring causality.
2. Supervised Fine-Tuning (SFT) for Latency Recovery:
   • Problem: Pre-trained models tend to over-generate WAIT tokens, leading to "latency drift" where the model gets stuck in a loop of pausing.
   • Solution: The SFT strategy artificially introduces delays in the training data (shifting content tokens further back) and masks the loss for the inserted delays. This forces the model to learn to "catch up" and burst out tokens rapidly once the context is clear, recovering from latency without sacrificing quality.

3. Key Contributions

1. Policy-Free End-to-End Framework: Eliminates the need for external READ/WRITE policies or heuristics by embedding decision-making into the token generation process.
2. Decoder Time Dilation: A novel mechanism to reduce autoregressive overhead and balance the training distribution of WAIT vs. text tokens.
3. Latency Recovery via SFT: A fine-tuning technique that explicitly trains the model to recover from delays, significantly improving the quality-latency trade-off.
4. State-of-the-Art Performance: Achieves new SOTA results in English-to-Japanese, German, and Russian simultaneous translation across both low and high-latency regimes.
5. Open Source: Release of model weights and codebase to foster reproducibility.

4. Results

The model was evaluated on IWSLT 2025 benchmarks and held-out podcast data (HQ Podcasts, PodCrawl).

• Translation Quality (BLEU):
  • English-to-Japanese: Achieved 42.17 BLEU (Low Latency) and 41.35 BLEU (High Latency), significantly outperforming baselines like NAIST, OSU, and CUNI.
  • English-to-German: Achieved 29.42 BLEU (Low Latency) and 37.67 BLEU (High Latency).
  • English-to-Russian: Achieved 27.89 BLEU (Low Latency) and 42.75 BLEU (High Latency).
• Latency (StreamLAAL):
  • The SFT model demonstrated a shift in the Pareto frontier, achieving lower latencies (e.g., ~1.59s for en-de) while maintaining high BLEU scores compared to the base model.
  • Inference speed (RTF) was maintained below 1.0 (faster than real-time) on A100/H100 GPUs with batch sizes up to 20.
• Streaming ASR:
  • While trained primarily for translation, Hikari achieved competitive Word Error Rates (WER) on clean datasets (8.6% on TED-LIUM), though performance dropped on noisy, domain-specific data (54.1% on Earnings21), highlighting the need for more diverse training data.
• Ablation Studies:
  • SFT: Removing SFT resulted in significant latency drift and lower BLEU scores.
  • Model Size: The "Medium" model significantly outperformed the "Small" variant, suggesting that policy-free learning requires substantial capacity.
  • Multitask Learning: Removing the ASR auxiliary task degraded translation performance, proving that ASR provides essential acoustic grounding for the causal encoder.

5. Significance and Impact

• Paradigm Shift: Hikari moves SiMT away from modular, heuristic-driven systems toward a unified, learned probabilistic approach. This simplifies the inference pipeline and allows the model to learn complex, language-specific strategies (like buffering for SOV structures) implicitly.
• Real-World Applicability: The ability to handle long-form speech (podcasts) with low latency makes this technology viable for live international conferences, education, and real-time communication tools.
• Future Directions: The authors suggest scaling to larger models (Whisper-large scale), implementing dynamic dilation factors based on speech density, and extending the architecture to speech-to-speech translation.

In summary, Hikari demonstrates that a single, end-to-end model can effectively learn the complex timing strategies required for simultaneous translation without explicit policy engineering, achieving superior performance across multiple language pairs and latency constraints.