Technical Summary: Parameter- and Bandwidth-Efficient Edge–cloud Many-to-Many Speech-to-Text Translation

1. Problem Statement

Speech-to-text translation (S2TT) faces a fundamental deployment trade-off between resource constraints and privacy/bandwidth efficiency. Traditional cascaded systems (ASR + MT) suffer from error propagation, while emerging Multimodal Large Language Models (MLLMs) offer unified architectures but encounter significant hurdles in real-world deployment:

• Edge Constraints: Pure on-device models are limited by memory and compute, often resulting in reduced translation quality.
• Cloud Constraints: Centralized cloud inference requires transmitting raw audio, creating bandwidth bottlenecks and exposing sensitive voiceprint biometrics to third-party services.
• Multilingual Imbalance: Existing MLLMs often exhibit English-centric biases, performing well on English-to-X directions but struggling with fully non-English (many-to-many) translation pairs.

2. Methodology: The ESRT Framework

The authors propose Edge–cloud Speech Recognition and Translation (ESRT), a collaborative framework designed to be parameter-efficient, bandwidth-efficient, and privacy-aware.

A. Architecture

ESRT utilizes a split-inference architecture:

• Edge Device: Runs a lightweight, frozen Whisper speech encoder and a trainable Q-Former adapter. It processes raw audio and outputs a compressed tensor of speech tokens.
• Cloud: Receives the compressed tensor and performs high-capacity inference using a Large Language Model (LLM) backbone (based on MiLMMT variants derived from Gemma-3).
• Data Minimization: The system transmits only the intermediate tensor (speech tokens) rather than the raw waveform, keeping the original audio on the device.

B. Multi-Task Weighted Curriculum Learning

To address catastrophic forgetting and improve multilingual balance, the authors introduce a Multi-Task Weighted Curriculum Learning strategy, replacing the strictly sequential training of prior works (like LLM-SRT) with weighted task mixtures across three stages:

1. Stage I: Focuses on Automatic Speech Recognition (ASR) to establish speech-text alignment.
2. Stage II: Samples ASR, Speech-guided Machine Translation (SMT), and Speech-to-Text Translation (SRT) with weights (0.2/0.4/0.4) to introduce translation while maintaining recognition capabilities.
3. Stage III: Shifts weights to (0.2/0.8) for ASR/SRT to emphasize direct translation while preserving acoustic grounding.
   This approach mitigates error propagation where recognition errors would otherwise degrade downstream translation.

C. Bandwidth Efficiency Mechanisms

• Tensor Compression: The Q-Former compresses the encoder output into a fixed number of speech tokens (e.g., 40 or 80 tokens).
• Size Reduction: For a 30-second audio clip, transmitting a 40-token BF16 tensor (approx. 0.059 MB) is significantly smaller than transmitting raw 16-bit mono audio (approx. 0.92 MB), achieving a theoretical reduction of ~10×.
• Feature Caching: The framework supports reusing the transmitted tensor for multiple inference requests (e.g., translating one audio into multiple languages), avoiding repeated transmission.

3. Key Contributions

1. Framework Design: Introduction of an Edge–cloud S2TT framework that maintains high translation quality while eliminating raw audio uploads and reducing data transmission.
2. Model Release: Release of parameter-efficient models (ESRT-1B, ESRT-4B, ESRT-12B) trained with a multi-stage weighted curriculum for many-to-many S2TT.
3. Systematic Evaluation: A comprehensive evaluation across 45 languages covering all 1,980 non-identity translation directions (45 × 44), revealing performance characteristics across different model scales and deployment settings.
4. Hardware Compatibility: Demonstration that the framework supports training and inference on both NPUs (Ascend 910C) and GPUs.

4. Experimental Results

Experiments were conducted on the FLEURS dataset, with additional analysis on CoVoST-2.

• State-of-the-Art Performance: ESRT-12B achieved the best overall performance across all 45 × 44 directions, scoring 23.9 spBLEU and 82.7 COMET. This outperformed the API-based Gemini-3.5-Flash-Lite by 3.6 spBLEU points and 4.4 COMET points, and surpassed the strong prior model MCAT-27B (27B parameters) by 3.1 COMET points, despite using significantly fewer parameters.
• Parameter Efficiency: As shown in Figure 1, ESRT models achieve superior multilingual performance with substantially fewer parameters compared to larger MLLMs (e.g., ESRT-4B outperforms models with ~6× its parameter count).
• Many-to-Many Robustness: Unlike English-centric models (e.g., SeamlessM4T-V2-Large) that degrade significantly on non-English source languages, ESRT-12B maintained a high average COMET score (82.7) across the full grid, with minimal performance drops on low-resource languages like Burmese and Khmer.
• Bandwidth Reduction:
  • ESRT (80 tokens): 5.1× reduction in transmitted size compared to raw audio.
  • ESRT-Lite (40 tokens): 10.2× reduction in transmitted size.
  • The Lite variant (40 tokens) retained competitive quality (87.6 COMET) compared to the full variant (88.1 COMET).
• Privacy Analysis: A qualitative reconstruction attack using an image-inpainting network on the transmitted 40-query tensor resulted in noisy, unintelligible audio, suggesting that direct recovery of the original waveform is difficult under the evaluated setting.

5. Significance and Claims

The paper claims that ESRT offers a viable solution for deploying high-quality, multilingual S2TT in resource-constrained and privacy-sensitive environments. By splitting the inference pipeline, it decouples the heavy computational burden of the LLM from the edge device while simultaneously addressing the bandwidth and privacy costs associated with raw audio transmission.

The authors emphasize that their approach achieves balanced multilingual performance without relying on brute-force scaling, making it suitable for edge scenarios. They position the work as a step toward "privacy-aware" S2TT, noting that while intermediate representations reduce direct exposure, the system is not formally privacy-preserving against all potential inversion attacks.

Limitations Acknowledged:

• The system inherits the 30-second input limit and low-resource weaknesses of the underlying Whisper encoder.
• Language coverage is dependent on the LLM's capabilities.
• The framework is described as "privacy-aware" rather than "privacy-preserving," as intermediate tensors may still expose linguistic or speaker information, and stronger inversion attacks remain a subject for future work.