Technical Summary: Simulstream

Problem Statement

Streaming Speech-to-Text Translation (StreamST) requires generating translations concurrently with incoming speech under strict latency constraints. A fundamental challenge in this field is the fragmentation of evaluation frameworks. Existing tools, such as the widely used SimulEval, are limited by specific design assumptions: they primarily support only incremental decoding (appending-only outputs), assume short pre-segmented audio inputs rather than continuous long-form streams, and lack native support for interactive system demonstrations. Furthermore, SimulEval does not support token deletion, rendering it incompatible with retranslation paradigms where systems regenerate or revise previous outputs. Consequently, fair, consistent, and reproducible benchmarking across different decoding strategies (incremental vs. retranslation) and system architectures remains difficult.

Methodology

To address these limitations, the authors introduce simulstream, an open-source framework (Apache 2.0 License) designed for the unified evaluation and demonstration of StreamST systems.

Architecture and Core Components

The tool operates on a full-duplex WebSocket client-server architecture:

• Server: Manages connections and processes incoming audio chunks (16-bit integers). It hosts a fixed-size pool of speech processors to handle concurrent clients, preventing resource exhaustion. The server generates transcripts/translations and logs fine-grained metrics (including emitted and deleted tokens) to JSONL files.
• Clients: Includes an HTTP web server for interactive real-time visualization and a command-line WAV client for automated batch evaluation.
• Speech Processors: The framework supports both incremental and retranslation strategies. It includes built-in processors:
  • Sliding Window: Applies fixed-length retranslation with deduplication to handle unsegmented streams.
  • StreamAtt: Uses cross-attention alignments to determine emission points and memory retention for long-form processing.
  • SimulEval Agent Wrapper: Allows seamless integration of legacy SimulEval agents.
  • VAD Wrapper: Integrates Voice Activity Detection (Silero VAD) to filter non-speech segments before processing.
  • The system is extensible, allowing users to implement custom processors by subclassing the base SpeechProcessor class.

Evaluation Metrics

Simulstream computes metrics directly from fine-grained logs, enabling post-hoc analysis without re-running experiments.

• Quality: Supports both BLEU (via sacreBLEU) and COMET. For streaming outputs involving revisions, evaluation is performed on the final output sequence only. The framework uses mweralign to re-segment generated text to match sentence-level references.
• Latency: Implements a reimplementation of the StreamLAAL metric. It computes latency based on the emission time of words relative to a uniform distribution of reference words. It distinguishes between ideal (computational-unaware) and computational-aware (CA) latency. For retranslation systems, latency is calculated based on the last update time of each word.
• Stability and Cost: Computes Normalized Erasure (NE) to measure flickering (ratio of deleted to final tokens) and Real Time Factor (RTF) to measure computational efficiency.

Key Contributions

1. Unified Framework: simulstream is the first open-source framework supporting both incremental and retranslation decoding strategies on long-form speech, with accurate tracking of token emissions and deletions.
2. Legacy Compatibility: It provides a lightweight proxy interface to the SimulEval framework, enabling the direct reuse of existing systems without modification.
3. Interactive Demonstration: It releases a web-based platform for real-time visualization and comparison of different StreamST systems, bridging the gap between research and practical demonstration.

Experimental Results

The authors evaluated the framework using Canary v2 and SeamlessM4T v1 medium models on the MuST-C and MCIF datasets (long-form TED talks and scientific presentations).

• Model Performance: Canary v2 consistently outperformed SeamlessM4T in translation quality (COMET and BLEU) across both retranslation and incremental strategies, while also exhibiting lower flickering (NE) and computational costs (RTF).
• Strategy Comparison:
  • Retranslation vs. Incremental: The comparison revealed that the optimal strategy is model-dependent. For SeamlessM4T, the incremental StreamAtt approach outperformed the sliding-window retranslation in both latency and quality while eliminating flickering, though it incurred higher computational costs. Conversely, for Canary v2, the sliding-window retranslation strategy achieved significantly higher quality than the incremental approach at comparable or lower latency.
  • VAD Impact: Adding a VAD wrapper to the sliding-window approach dramatically reduced flickering and computational costs (RTF reduced by 2-3x) but resulted in a substantial drop in translation quality and reduced the ability to control the latency-quality trade-off.
• Generalizability: Results on the MCIF dataset (featuring diverse accents and recording qualities) confirmed the trends observed on MuST-C, reinforcing that no single decoding strategy is universally superior.

Significance

The paper claims that simulstream establishes a common infrastructure for studying and comparing different StreamST decoding paradigms under controlled and consistent conditions. By enabling reproducible benchmarking and interactive demonstration, the tool facilitates informed design choices for future streaming speech translation systems. The framework has already been adopted as the official evaluation toolkit for the Simultaneous Translation track of the IWSLT 2026 Evaluation Campaign. The authors emphasize that their work supports the community in systematically studying the trade-offs between latency, quality, and stability in both incremental and retranslation approaches.