💬 NLP

DialectS2S: End-to-End Speech Dialogue Modeling for Low-Resource Chinese Dialects

This paper introduces DialectS2S, an end-to-end speech dialogue model for low-resource Chinese dialects that leverages a scalable data synthesis pipeline and a two-stage self-aligned post-training strategy to overcome data scarcity and semantic inconsistency, significantly improving dialect consistency, response quality, and speech intelligibility.

Original authors: Yi Shu, Tianyu Peng, Yingzhuo Deng, Wen Yang, Jun Lin, Changming Xie, Xinyu Yu, Jiajun Zhang

Published 2026-08-11

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Yi Shu, Tianyu Peng, Yingzhuo Deng, Wen Yang, Jun Lin, Changming Xie, Xinyu Yu, Jiajun Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where your phone doesn't just understand the "official" language everyone learns in school, but also the colorful, slang-filled ways your friends and grandparents actually talk. For a long time, the smartest computer brains—called "speech dialogue models"—were like elite students who only spoke perfect textbook Mandarin or English. They could chat fluently in those languages, but if you asked them a question in a local dialect like Sichuanese or Cantonese, they often got confused, spoke in a robotic monotone, or just switched back to the official language. This is a big deal because dialects are the heartbeat of local culture and family connection. The problem is that there just aren't enough recorded conversations in these dialects to teach the computers how to speak them naturally. It's like trying to teach a chef to cook a specific regional dish when you only have one recipe book and a few scattered ingredients.

The researchers behind this paper, DialectS2S, decided to fix this by building a new kind of "speech-to-speech" model that can chat naturally in low-resource Chinese dialects. Instead of just trying to force the computer to memorize a few rare recordings, they invented a clever way to create thousands of new practice conversations using other AI tools. They also noticed something tricky: when you teach a computer a new dialect, its internal "brain" changes how it thinks, but the way we teach it to speak often stays the same, causing a mismatch. To solve this, they developed a two-step training method where the computer teaches itself to speak based on its own new understanding, rather than following an old, rigid script. The result? A model that can hold a natural, clear, and consistent conversation in dialects like Sichuanese, Cantonese, and Tianjin, without losing its ability to speak Mandarin or English.

The Problem: The "Robot" Who Can't Speak Local

Imagine you have a super-smart robot friend who is great at telling jokes in perfect English and Mandarin. But when you try to talk to it in your hometown dialect, it starts stuttering, sounding like a broken radio, or worse, it just ignores your accent and replies in perfect Mandarin. This is exactly what happens with current speech models. They are trained on massive amounts of data for major languages, but for dialects, the data is scarce.

The researchers found that simply trying to teach these models more dialect data often backfires. It's like trying to teach a pianist a new song by making them play the old song faster and louder; the more they try, the more their fingers get tangled. When the model learns a new dialect, its internal "thinking" (how it understands meaning) changes and evolves. However, the "speaking" part (how it turns that thought into sound) is still being taught using the old, unchanged rules. This creates a mismatch: the brain is thinking in dialect, but the mouth is trying to speak in a way that doesn't fit, leading to speech that sounds unnatural or hard to understand.

The Solution: Building a Dialect Playground

To fix this, the team built DialectS2S, a system designed specifically for low-resource dialects. They didn't just wait for people to record more dialect conversations; they built a machine to create them.

1. The Synthetic Pipeline (The "Dialect Factory")
Since real dialect data is hard to find, they created a pipeline to generate it.

Step 1: Rewriting the Script. They took existing, high-quality conversations in Mandarin and used a large language model to "translate" them into dialects like Sichuanese, Cantonese, and Tianjin. It's like taking a standard play and rewriting the dialogue so the characters sound like they are from a specific village, keeping the story the same but changing the flavor.
Step 2: Synthesizing the Voice. They then used speech generation models to turn these new dialect scripts into audio. For the "user" part of the conversation, they used a variety of voices to make it sound like different people. For the "system" (the AI) part, they kept the voice consistent so the AI sounded like one reliable character.
Step 3: The Quality Filter. Not all AI-generated speech sounds good. They used a tool called UTMOS to automatically filter out any clips that sounded robotic or had bad pronunciation, keeping only the high-quality "gold" samples.

2. The Two-Stage Training (The "Self-Correcting Loop")
This is the secret sauce. The researchers realized that the old way of training (just showing the model examples and saying "copy this") wasn't working because the model's internal understanding was shifting.

Stage 1: Mixed-Data Fine-Tuning. First, they taught the model using a mix of Mandarin, English, and the new dialect data. This helped the model learn to understand the dialects.
Stage 2: Self-Aligned Speech Supervision. This is the clever part. Instead of forcing the model to copy old speech targets, they let the model's "Thinker" (the part that understands meaning) generate its own text response. Then, they used a speech synthesizer to turn that text into new speech. They used this new speech as the target for the "Talker" (the part that speaks).
- The Analogy: Imagine a student learning to speak. In the old method, the teacher gives them a script and says, "Read this exactly." In the new method, the student thinks of what they want to say, writes it down, and then practices speaking their own words. This ensures the way they speak matches exactly what they are thinking, eliminating the confusion between "what I mean" and "what I say."

The Results: Speaking Like a Local

The team tested their new model against several other top-tier speech models. The results were impressive.

Language Matching: When asked to speak in a specific dialect, DialectS2S got it right 96% of the time for Sichuanese, 95% for Cantonese, and 91% for Tianjin. Compare this to other models, which often scored 0% or very low numbers (like 2.22% for Sichuanese in one competitor). It's the difference between a tourist who knows a few phrases and a local who can chat all day.
Response Quality: When humans rated how natural and accurate the answers were, DialectS2S scored an average of 4.42 out of 5 across all dialects. Other models struggled, with scores dropping as low as 2.06 for Tianjin.
Speech Intelligibility: This measures how easy it is to understand the speech. The researchers used a metric called Character Error Rate (CER), where a lower number is better. DialectS2S achieved a CER of 8.69% for dialects, which is actually better than the average performance of existing dialect benchmarks (which was 11.67%). This means the speech wasn't just "dialecty," it was clear and easy to understand.

What This Means

The paper suggests that by aligning the speech targets with the model's own evolving understanding, we can teach computers to speak low-resource dialects much more effectively. The team showed that simply adding more training time (like the "sft-3ep" method they tested) didn't help as much as their new self-aligned method. In fact, just training longer sometimes made the speech less clear.

The researchers have made their entire framework, including the code and the data they created, open-source. This means anyone can use DialectS2S to build their own dialect chatbots. While the current model covers five languages and dialects (Mandarin, English, Sichuanese, Cantonese, and Tianjin), the door is now open to teach computers to speak many more local languages, preserving the unique voices of different cultures in the digital age.

Technical Summary: DialectS2S

Problem Statement

Current end-to-end speech dialogue models are predominantly optimized for high-resource mainstream languages (e.g., Mandarin, English) and struggle in low-resource Chinese dialect scenarios. The primary challenges include:

Data Scarcity: High-quality dialect speech dialogue corpora are limited, hindering the training of robust models.
Semantic Inconsistency: During dialect adaptation, the semantic representation space of the "Thinker" module (responsible for understanding and reasoning) evolves. However, conventional speech supervision targets remain static. This misalignment forces the "Talker" module (responsible for speech generation) to implicitly perform semantic recovery alongside speech generation, leading to degraded speech stability, naturalness, and intelligibility.
Limited Scope: Existing research largely focuses on isolated speech recognition (ASR) or text-to-speech (TTS) tasks, leaving unified end-to-end spoken dialogue systems for Chinese dialects underexplored.

Methodology

The authors propose DialectS2S, an end-to-end speech dialogue model designed for low-resource Chinese dialects. The framework operates on a unified Thinker-Talker architecture and employs a two-pronged approach: data construction and a novel training strategy.

1. Scalable Dialect Speech Dialogue Synthesis Pipeline

To overcome data scarcity, the authors developed a pipeline to construct dialect dialogue data:

Dialect Dialogue Generation: Instead of prompting LLMs directly (which yields low diversity), the system rewrites existing open-source Mandarin dialogue texts into dialectal expressions using LLMs while preserving original semantics and structure.
Dialect Speech Synthesis:
- User Queries: Synthesized using CosyVoice2 conditioned on 100 diverse seed utterances per dialect to ensure varied vocal characteristics.
- System Responses: Synthesized using a consistent speaker timbre strategy to maintain voice consistency and improve expressiveness.
Naturalness Filtering: Automated filtering using UTMOS removes samples with unclear pronunciation or unnatural prosody.

2. Two-Stage Post-Training Strategy

To address the semantic inconsistency between evolving hidden representations and static speech targets, the authors introduce a Self-Aligned Speech Supervision strategy:

Stage 1: Mixed-Data Supervised Fine-Tuning (SFT): The model is fine-tuned on a mix of dialect, Mandarin, and English dialogue data. This shifts the Thinker module's hidden representations toward dialect-aware semantic spaces.
Stage 2: Self-Aligned Speech Supervision Training:
- The fine-tuned Thinker module generates text predictions ( $\hat{T}$ ) from input speech.
- A dialect TTS model synthesizes speech ( $\hat{U}$ ) from these predicted texts.
- This synthesized speech serves as the new supervision target for the Talker module.
- Mechanism: By aligning the speech supervision target with the model's own semantic predictions, the semantic content of the supervision matches the evolved hidden representations. This allows the Talker module to focus purely on prosody and pronunciation modeling rather than correcting semantic mismatches.

Key Contributions

DialectS2S Framework: The first end-to-end speech dialogue model specifically designed for low-resource Chinese dialects, supporting multilingual interaction across Mandarin, English, Sichuanese, Cantonese, and the Tianjin dialect. The authors fully open-source the framework, including checkpoints, datasets, and code.
Self-Aligned Speech Supervision: A novel two-stage training strategy that explicitly aligns speech targets with the model's semantic predictions, resolving the mismatch between evolved hidden states and static supervision to improve generation quality.
Scalable Data Synthesis: An efficient pipeline for constructing high-quality dialect dialogue data by rewriting existing corpora and synthesizing speech, addressing the bottleneck of dialect data scarcity.

Experimental Results

The model was evaluated on a benchmark covering three Chinese dialects (Sichuanese, Cantonese, Tianjin) and two mainstream languages.

Language Matching Accuracy: DialectS2S achieved an average accuracy of 95.99%, significantly outperforming baselines like GLM-4-Voice (48.22%) and Qwen2.5-Omni (40.00%). Notably, it achieved 96.00% accuracy for Sichuanese and 95.00% for Cantonese, whereas most baselines scored near 0% for these dialects.
Response Quality: Evaluated via an automatic scorer (Qwen3-Plus), DialectS2S achieved the highest average score (4.42) across all scenarios, demonstrating superior semantic understanding and dialect adherence compared to existing open-source models.
Speech Intelligibility: Measured by Character Error Rate (CER), DialectS2S achieved a dialect average CER of 8.69%, which is lower (better) than the average CER reported by the FRASR2-AED dialect benchmark (11.67%). This indicates highly intelligible generated dialect speech.
Ablation Study: Comparisons between standard SFT (2 and 3 epochs) and the proposed self-aligned strategy showed that while SFT alone improves response quality, it fails to consistently improve speech intelligibility (CER). The self-aligned strategy significantly reduced CER (e.g., 6.98% average vs. 10.15% for sft-2ep) without compromising response quality.

Significance

The paper claims that DialectS2S provides an efficient and scalable solution for end-to-end speech dialogue modeling in low-resource dialect scenarios. By addressing the semantic inconsistency inherent in dialect adaptation through self-aligned supervision, the work enables stable, natural, and intelligible dialect speech interactions. The open-sourcing of the framework aims to facilitate future research and practical applications in regional communication and cultural expression, moving beyond the limitations of current mainstream-language-optimized systems.