💬 NLP

Cross-Lingual Interleaving for Speech Language Models

This paper introduces a simple, scalable cross-lingual interleaving method that improves the performance of Speech Language Models across languages without textual supervision, supported by the release of a large EN-FR training dataset and new synthetic benchmarks for evaluation.

Original authors: Adel Moumen, Guangzhi Sun, Philip C. Woodland

Published 2026-02-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Adel Moumen, Guangzhi Sun, Philip C. Woodland

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to speak. Usually, we teach robots by showing them millions of books and articles (text). But what about languages that don't have many books written down? Or what if we want the robot to learn directly from listening to people talk, just like a baby does?

This is the challenge the researchers at Cambridge University tackled. They created a new way to teach "Spoken Language Models" (SLMs) to understand and speak multiple languages, even when they only have audio to learn from.

Here is the story of their discovery, explained simply:

1. The Problem: The "English-Only" Trap

Most AI models are like students who only went to school in English. They are great at English, but if you ask them to speak French, they stumble.

The Bottleneck: To teach a robot a new language, you usually need a teacher who speaks both languages and can translate word-for-word. But for many languages, we don't have those teachers (text data).
The Goal: They wanted to build a robot that learns purely by listening, without needing any written text to help it.

2. The Solution: The "Language Smoothie" (Interleaving)

The researchers came up with a clever trick called Cross-Lingual Interleaving.

Imagine you are making a smoothie.

The Old Way: You make a strawberry smoothie (English), drink it, then make a blueberry smoothie (French), and drink that. The flavors stay separate.
The New Way (Interleaving): You take a sip of strawberry, then a sip of blueberry, then strawberry again, mixing them in the same cup.

In their experiment, they took sentences from English stories and French stories and mixed them together in a single training sequence. The robot hears: "Hello, how are you?" (English) followed immediately by "Bonjour, ça va?" (French), followed by more English.

Why does this work?
By constantly switching languages, the robot is forced to find the "common ground" between them. It realizes that the feeling of a greeting is the same in both languages, even if the sounds are different. It builds a shared mental map where English and French live in the same neighborhood, rather than in separate cities.

3. The Ingredients: A New Library and a New Test

To make this work, they needed two things:

The Library (TinyStories): They created a massive library of 42,000 hours of audio stories. These are short, simple stories (like "The Cat and the Dog") translated perfectly between English and French, spoken by the same voices.
The Test (StoryCloze): They built a test where the robot listens to the first part of a story and has to guess the ending. They tested if the robot could listen to a story in English and guess the ending in French, or vice versa.

4. The Results: A Multilingual Superpower

They tested this on two robot "brains" (one small, one medium-sized) and found some amazing things:

The "Side Effect" Bonus: Even though the robot was learning two languages at once, it actually got better at speaking just English or just French on its own. It's like a student who studies math and physics together and ends up being better at math than if they had only studied math.
The "Switching" Skill: The robot became incredibly good at "code-switching." If you started a sentence in English, it could finish it in French naturally, and vice versa. The old methods (mixing data but keeping languages separate) failed at this completely.
The "Deep Connection": When they looked inside the robot's brain, they saw that the internal "thoughts" (hidden states) for English and French were becoming almost identical. The robot wasn't just memorizing; it was truly understanding that the concepts were the same.

5. The Big Picture

This research is a huge step toward democratizing AI.
Right now, AI is mostly for languages with lots of written books (like English, Chinese, Spanish). This method means we can finally build smart voice assistants for languages that have very few written resources, simply by feeding them audio.

In a nutshell:
The researchers proved that if you mix languages together in a "smoothie" while training an AI, the AI learns to understand the essence of language rather than just the specific words. This allows it to speak, understand, and switch between languages effortlessly, opening the door for AI to converse with the whole world, not just the English-speaking part.

1. Problem Statement

Spoken Language Models (SLMs) aim to learn linguistic competence directly from audio using discrete speech units, bypassing the need for text supervision. This is crucial for low-resource languages where written data is scarce. However, the field faces two major bottlenecks in achieving multilingual capabilities:

English-Centric Evaluation: Most benchmarks and training data are English-focused, making it difficult to measure cross-lingual semantic competence.
Incompatible Training Methods: Existing methods that encourage cross-lingual sharing (e.g., in Text-Speech LMs) rely on interleaving text and speech tokens. This is incompatible with "textless" SLMs, which operate purely on discrete audio units. Furthermore, current cross-lingual approaches often lack mechanisms to mix languages within a single training sequence without text alignment.

The core challenge is how to train a single SLM to understand and converse across multiple languages using only audio data, without relying on text supervision or massive amounts of parallel text data.

2. Methodology

The authors propose a Cross-Lingual Interleaving strategy that operates entirely within the speech token domain.

A. Cross-Lingual Interleaving Scheme

Instead of training on monolingual sequences or mixing languages only at the dataset level, the model is trained on sequences where speech segments from different languages are concatenated within a single context window.

Mechanism: Given a set of languages $\mathcal{L}$ , the model samples a language label $l_j$ for each sentence segment $j$ . The training sequence $s$ is constructed as:
$s = (A^{(l_1)}_1 \parallel A^{(l_2)}_2 \parallel A^{(l_3)}_3 \parallel \dots)$
where $A^{(l)}_j$ is the sequence of speech tokens for the $j$ -th sentence in language $l$ , and $\parallel$ denotes concatenation at sentence boundaries.
Constraint: The model predicts the next token based on preceding tokens that may belong to a different language. This forces the model to learn a shared representational subspace without explicit text alignment.
Tokenization: The system uses the Mimi neural audio codec (RVQ with 32 codebooks) at 12.5 Hz. Only the first (semantic) codebook is modeled to focus on linguistic content rather than acoustic details.

B. Datasets and Benchmarks

To enable this training and evaluation, the authors curated new resources:

TinyStories (EN-FR): A sentence-aligned, bilingual spoken corpus derived from the TinyStories text dataset.
- Scale: ~42,000 hours of speech total (20k hours English, 21k hours French).
- Construction: Sentences were translated using GPT-4 (maintaining context) and synthesized using a multi-speaker TTS system. Crucially, 44 voices (26 female, 18 male) were selected based on speaker verification embeddings to ensure consistent speaker identity across languages.
Evaluation Benchmarks:
- Spoken StoryCloze (sSC): Tests causal/temporal commonsense reasoning (predicting the correct ending of a story).
- Spoken TopicCloze (sTC): Tests topical coherence (predicting the correct ending based on topic consistency).
- Cross-Lingual Setup: These benchmarks include EN→FR and FR→EN evaluation modes to test cross-lingual continuation.

C. Training Protocol

Experiments were conducted on 360M and 1B parameter SLMs (initialized from Llama 3.2 and Qwen2 checkpoints). The training followed a three-stage schedule to isolate the effects of interleaving:

Pre-training: English-only (50k steps) to establish a high-resource baseline.
Interleaving: Cross-lingual mixing (EN/FR) with 50% sampling probability (20k steps).
Stabilization: Alternating monolingual fine-tuning on EN and FR (15k steps) to refine generation capabilities.

Control: All comparisons used a matched training-token budget to ensure performance gains were due to the interleaving strategy, not increased data volume.

3. Key Contributions

Novel Training Strategy: Introduction of a cross-lingual interleaving method that mixes speech tokens across languages without any text supervision, compatible with pure "textless" pipelines.
Open-Source Resources: Release of the TinyStories EN-FR corpus (~42k hours) and bilingual StoryCloze/TopicCloze benchmarks, addressing the scarcity of spoken evaluation data for non-English languages.
Empirical Validation: Demonstration that interleaving improves monolingual semantic accuracy, enables robust cross-lingual continuation, and strengthens hidden-state alignment in models ranging from 360M to 1B parameters.

4. Experimental Results

The results, evaluated on sBLiMP (syntax), sWUGGY (lexicon), sSC, and sTC (semantics), reveal several key findings:

Positive Transfer to Monolingual Tasks:
- Interleaving allowed the model to learn French effectively even with significantly fewer French tokens than a monolingual baseline.
- For the 360M model, interleaving + stabilization achieved 57.08% on French sSC (vs. 56.44% for the monolingual baseline with 3x more French data), proving the gains come from transfer, not data volume.
- A brief stabilization phase recovered English performance (syntax/lexicon) to near baseline levels while retaining French gains.
Emergence of Cross-Lingual Abilities:
- Models trained with interleaving showed robust cross-lingual continuation (e.g., prompting in English, continuing in French).
- On the 360M model, cross-lingual sSC accuracy reached 56.44% (EN→FR) and 55.37% (FR→EN), approaching monolingual performance levels.
- In contrast, a baseline trained on mixed data without interleaving performed significantly worse (approx. 50-51% on sSC), failing to bridge the language gap effectively.
Representation Alignment:
- Layer-wise cosine similarity analysis of hidden states for aligned EN-FR sentences showed that interleaving increased alignment from 0.73 (mixed baseline) to 0.75 (interleaving) and 0.76 (interleaving + stabilization). This confirms the formation of a shared semantic space.
Model Size Dynamics:
- Smaller models (360M) benefited disproportionately from multilingual context under limited French data, suggesting that interleaving helps smaller models avoid overfitting and optimize more efficiently.

5. Significance

This work demonstrates that cross-lingual interleaving is a scalable, simple, and effective route to building multilingual Spoken Language Models.

Scalability: The method does not require text supervision or complex alignment tools, making it applicable to any language pair where speech tokens are available.
Resource Efficiency: It enables high-quality cross-lingual understanding even when data for one language is scarce, leveraging the "high-resource" language (English) to bootstrap the "low-resource" language (French).
Future Impact: By releasing the datasets and benchmarks, the authors provide the community with the necessary tools to move beyond English-centric speech AI, paving the way for truly multilingual voice assistants and NLP tools for low-resource languages.