Technical Summary: Direct Simultaneous Activation for Large Audio-Language Models

1. Problem Statement

Simultaneous speech-to-text translation (Simul-S2TT) aims to translate speech in real-time, outputting text while receiving source audio, rather than waiting for the full utterance. While effective for cross-lingual communication and live subtitles, Simul-S2TT faces distinct challenges compared to offline translation:

• Incomplete Input: The system must generate translations from partial audio segments that may lack complete semantic units or clear linguistic boundaries.
• Distribution Gap: Offline training data aligns full source audio with full target text. In contrast, simultaneous inference involves "waiting" for future information to handle cross-lingual word reordering, creating a distribution shift between training (complete alignment) and inference (incomplete alignment).
• Architectural Rigidity: Traditional Simul-S2TT methods often require modifying model architectures (e.g., new decoder-encoder structures, incremental decoding, or attention constraints) to implement "Read-Write" strategies. This hinders the direct deployment of general Large Audio-Language Models (LALMs), which are typically designed for offline tasks.

The core challenge addressed is how to activate Simul-S2TT capabilities in base LALMs without altering their architecture or decoding strategy.

2. Methodology: Simultaneous Self-Augmentation (SimulSA)

The authors propose Simultaneous Self-Augmentation (SimulSA), a data augmentation strategy that bridges the distribution gap by training LALMs on synthetic simultaneous data derived from their own offline capabilities. The pipeline consists of three stages:

Stage 1: Speech Truncation

From a standard offline speech-text Supervised Fine-Tuning (SFT) dataset $D$, a small subset $D'$ is selected. The full audio $x_{1:s}$ in these pairs is truncated to create partial inputs $x_{1:s'}$.

• Beta Decay Distribution: Instead of uniform random truncation, the authors employ a Beta distribution ($\alpha=1, \beta=3$) mapped to a specific time interval $[l, r]$. This prioritizes truncation points in the earlier stages of the speech, simulating realistic streaming boundaries where early decoding is critical. This avoids generating segments that are too short (lacking information) or too long (resembling offline translation).

Stage 2: Truncated Speech-to-Text Speculation

Since manual annotation of truncated pairs is costly, the method leverages the pre-trained LALM to generate the corresponding target text $y_{1:t'}$ for the truncated audio $x_{1:s'}$.

• Speculative Generation: The model iteratively generates tokens based on the truncated audio and the previously generated prefix.
• Termination Criteria: The generation stops at token $t'$ if the probability of the next token $y_{t'+1}$ drops below the probability of the End-of-Sequence token ($$) or if its rank in the vocabulary exceeds a threshold $\tau$. This ensures the generated text $y_{1:t'}$ contains only information fully supported by the truncated audio $x_{1:s'}$, effectively capturing the "wait" signal required for simultaneous translation.

Stage 3: Mixed Supervised Fine-Tuning

The original full speech-text pairs and the newly generated truncated pairs are mixed and used to fine-tune the base LALM.

• Single-Stage Training: The authors found that a single-stage mixed SFT outperforms a two-stage approach (training first on full data, then on streaming data), which led to performance degradation.
• Data Ratio: The augmented data constitutes a small fraction (approx. 1%) of the total training corpus.

3. Key Contributions

1. Architecture-Free Activation: The paper demonstrates that Simul-S2TT capabilities can be directly activated in base LALMs (specifically Qwen2-Audio) without any architectural modifications or changes to the decoding strategy.
2. SimulSA Strategy: A novel self-augmentation pipeline that constructs high-quality, partially aligned speech-text pairs using the model's own probabilistic outputs and a Beta decay truncation distribution.
3. Efficiency: The method requires augmenting only ~1% of the SFT data to achieve significant performance gains, making it a low-cost solution for enabling real-time capabilities.

4. Experimental Results

Experiments were conducted using the Qwen2-Audio-7B model on the CoVoST2 dataset (English-to-Chinese).

• Performance Gains: Adding ~3,000 augmented samples (increasing the dataset from 232k to 235k) significantly improved BLEU scores.
  • In a low-latency scenario (500ms chunk size, no rollback), SimulSA improved BLEU by 5.1 points over the standard SFT baseline.
  • Gains were most pronounced at lower latencies (smaller chunk sizes) and with fewer rollback tokens.
• Robustness: The method did not compromise offline translation performance. At infinite chunk size (offline mode), BLEU scores remained within random variation compared to the SFT-only baseline.
• Ablation Studies:
  • Data Size: Performance scaled linearly with augmented data size for low-latency settings (no rollback), while gains plateaued at ~1.3% augmentation for higher latency settings.
  • Truncation Distribution: The proposed Beta Decay distribution outperformed uniform truncation, full-span beta decay, and discrete beta decay, confirming the importance of prioritizing early-stage truncation points.

5. Significance and Claims

The paper claims that SimulSA offers a scalable and practical solution for deploying LALMs in real-time translation scenarios. By avoiding architectural changes, it allows general-purpose audio-language models to handle streaming inputs directly. The authors emphasize that this approach:

• Effectively bridges the distribution gap between offline training and simultaneous inference.
• Activates simultaneous capabilities with minimal computational cost (only 1% additional data).
• Maintains compatibility with other tasks and existing model architectures, facilitating easier integration into production systems.

The work suggests that the "wait" mechanism and handling of incomplete semantic units can be learned through data augmentation rather than structural re-engineering, marking a shift in how Simul-S2TT is approached in the era of Large Audio-Language Models.