💬 NLP

Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignment

This paper proposes a fine-grained code-switch speech translation framework that enhances Large Language Models with a language-specialized Mixture-of-Experts projector and a multi-stage training paradigm to achieve significant performance improvements over existing models like SeamlessM4T.

Original authors: Yan Gao, Yazheng Yang, Zhibin Lan, Yidong Chen, Min Zhang, Daimeng Wei, Derek F. Wong, Jinsong Su

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yan Gao, Yazheng Yang, Zhibin Lan, Yidong Chen, Min Zhang, Daimeng Wei, Derek F. Wong, Jinsong Su

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to translate a conversation where two people are speaking a mix of English and Spanish (or Chinese and English) in the same sentence. This is called Code-Switching. It's like a musical duet where the singers switch instruments mid-song.

For computers, this is a nightmare. Most translation software is trained on "pure" songs (all English or all Spanish). When the music switches instruments suddenly, the computer gets confused, loses the rhythm, and produces a bad translation.

This paper presents a new way to teach computers how to handle these musical switches smoothly. Here is the breakdown of their solution using simple analogies:

1. The Problem: The "One-Size-Fits-All" Translator

Previous computers tried to use a single "translator brain" for all languages. The authors found that this is like trying to use a single pair of glasses to see both a microscopic ant and a giant mountain. The computer struggles to see the fine details when languages mix because the "sound space" of English is very different from Spanish or Chinese.

2. The Solution: A "Specialist Team" (MoE Projector)

Instead of one general translator, the authors built a Mixture-of-Experts (MoE) system. Think of this as a high-end translation agency with a team of specialists:

The Team: Instead of one translator, they have a group of "English Experts," a group of "Spanish Experts," and a group of "Chinese Experts."
The Router: When a sentence comes in, a smart "manager" (the router) listens to the words. If the word is in English, the manager instantly hands it to the English team. If it switches to Spanish, the manager passes it to the Spanish team.
The Result: This allows the computer to understand the subtle nuances of each language separately, even when they are mixed together in the same sentence.

3. The Training: A Four-Step "Boot Camp"

You can't just throw a new recruit into a chaotic mixed-language war zone immediately. The authors designed a four-stage training camp to get the model ready:

Stage 1: Learning the Instruments (ASR). First, the specialists practice on pure, single-language recordings (like just English or just Spanish). They learn to recognize the sounds perfectly without trying to translate yet.
Stage 2: Forming the Teams. The specialists are organized into their groups. The "manager" learns to sort the incoming sounds into the correct groups. They use special rules (loss functions) to make sure the manager doesn't get lazy and send everything to just one person, but instead distributes the work evenly.
Stage 3: The Transition (Monolingual Translation). Now, they start practicing translation, but only on pure languages. They slowly mix in translation tasks, helping the team learn how to turn speech into text without getting confused by the language switch.
Stage 4: The Real Challenge (Code-Switching). Finally, they throw in the messy, mixed-language conversations. Because the team has already mastered the individual languages and the translation process, they can now handle the switches smoothly.

4. The Results: A Better Performance

The authors tested this new system on real-world data where people switch languages.

The Competition: They compared their system against the current best models (like SeamlessM4T and Whisper).
The Outcome: Their "Specialist Team" approach won. It translated mixed-language speech more accurately than the others.
The Proof: They showed that when they removed the "Specialist Team" (MoE) or the "Manager" (Router), the performance dropped significantly. This proves that having different experts for different languages is the key to success.

Summary

In short, the paper says: Don't try to force one brain to do everything. Instead, build a team of specialists, train them step-by-step from simple to complex, and use a smart manager to route the work. This approach allows computers to finally understand and translate conversations where people switch languages mid-sentence.

Technical Summary: Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignment

Problem Statement

Code-switching (CS) speech translation (ST) involves translating speech that alternates between multiple languages into a target language text. This task faces two primary challenges:

Semantic Modeling Complexity: The alternation of languages introduces complex semantic modeling requirements, making it difficult for models to capture effective representations from CS speech.
Data Scarcity: High-quality, large-scale CS speech translation datasets are limited and difficult to construct. Previous approaches often rely on costly manual annotations or synthetic data, which suffers from unnatural switching and grammatical inconsistencies.

While Large Language Models (LLMs) have shown promise in cross-modal tasks, existing studies indicate they exhibit limited cross-lingual capabilities on CS data compared to monolingual data. Furthermore, current research on LLMs in CS contexts focuses primarily on machine translation (MT), leaving their potential in CS speech translation largely unexplored.

Methodology

The authors propose a framework that enhances LLMs with a Mixture-of-Experts (MoE) speech projector and a multi-stage training paradigm to address semantic gaps and data scarcity.

1. Model Architecture

The framework consists of three main components:

Speech Encoder: Extracts features from the input speech (e.g., using Whisper-large-v3).
MoE Speech Projector: Connects the speech encoder to the LLM. Unlike a single shared projector, this module organizes experts into language-aware groups. For a scenario involving $m$ languages, there are $m$ expert groups, each containing $n$ experts. A sparse router dynamically assigns speech tokens to specific experts within the appropriate language group.
Large Language Model (LLM): Processes the concatenated prompt and aligned speech representations to generate the translation (e.g., Llama-3-8B-Instruct).

2. Multi-Stage Training Paradigm

To mitigate data scarcity and ensure smooth domain adaptation, the authors introduce a four-stage training process:

Stage 1 (ASR Pre-training): Individual MLP-based projectors are pre-trained for each language using high-resource Automatic Speech Recognition (ASR) data to align speech and text modalities.
Stage 2 (MoE Construction & Specialization): The pre-trained projectors initialize the expert groups. A router is inserted, and the model is trained using:
- Cross-Entropy Loss ( $L_{ce}$ ): Standard translation/recognition objective.
- Language-Specific Loss ( $L_{lang}$ ): Explicitly supervises the routing of tokens to the correct language expert group (using oracle labels available in monolingual ASR data).
- Intra-Group Load Balancing Loss ( $L_{balance}$ ): Encourages balanced token distribution among experts within the same language group to prevent over-reliance on a subset of experts.
Stage 3 (Monolingual ST Transition): The model transitions from ASR to monolingual Speech Translation (ST) data. A Transition Loss ( $L_{transition}$ ) is used to smoothly mix ASR and ST data, bridging the task gap between recognition and translation.
Stage 4 (CS Adaptation): The model adapts to CS speech translation data. The transition loss is applied again to mix monolingual ST and CS ST data. In this stage, $L_{lang}$ and $L_{balance}$ are excluded as oracle language labels for individual tokens are unavailable in CS data.

Key Contributions

First LLM Exploration in E2E CS-ST: The work represents the first exploration of Large Language Models for end-to-end Code-Switch speech translation, providing baseline results for future research.
MoE Speech Projector: A novel architecture that enables fine-grained modeling of speech features from different languages by organizing experts into language-specific groups, enhancing the capture of semantic distinctions in CS speech.
Multi-Stage Training Paradigm: A novel strategy that leverages readily available ASR and monolingual ST data to guide the model through speech-text alignment and gradual adaptation to CS scenarios, effectively mitigating the scarcity of high-quality CS data.
Empirical Validation: Extensive experiments validate the model's effectiveness, with in-depth analyses providing insights into the contributions of the MoE architecture, loss functions, and training stages.

Experimental Results

The approach was evaluated on the Fisher (English-Spanish) and NTUML2021 (Chinese-English) datasets, comparing against baselines such as Whisper, SeamlessM4T, and cascaded Whisper+LLaMA pipelines.

Performance Gains: The proposed model achieved state-of-the-art results, outperforming the strong SeamlessM4T baseline.
- Average Improvements: +0.86 BLEU and +0.93 COMET across test sets.
- Maximum Improvements: +1.49 BLEU and +1.41 COMET on specific test sets.
- Specific Scores: On the NTUML2021 CS test set, the model achieved 39.52 BLEU and 81.33 COMET, surpassing the next best baseline (SeamlessM4T at 38.03 BLEU / 79.93 COMET).
Generalization: The model also demonstrated competitive or superior performance on monolingual ST data, indicating robustness across both monolingual and CS scenarios.
Ablation Studies: Removing the MoE projector, the language-specific loss, the load balancing loss, or the transition stages resulted in significant performance drops, confirming the necessity of each component.
Routing Analysis: Experiments showed that the router successfully directs over 95% of English and Spanish tokens (and >80% of Chinese tokens) to their corresponding language expert groups, validating the semantic space alignment.

Significance and Claims

The paper claims that by explicitly addressing the semantic space gap between languages through a language-aware MoE projector and overcoming data scarcity via a structured multi-stage training paradigm, it is possible to significantly improve the quality of code-switch speech translation. The work establishes a new baseline for LLM-based E2E CS-ST and demonstrates that careful architectural design and data utilization strategies can overcome the limitations of implicit learning and data scarcity in this challenging domain. Future work aims to extend the model to more language pairs and related tasks like ASR and MT.