Multilingual Multi-Speaker Unit Vocoders: A Systematic Analysis of Discrete Speech Representations

Technical Summary: Multilingual Multi-Speaker Unit Vocoders

Problem Statement
Discrete speech units, derived from self-supervised learning (SSL) embeddings via k-means clustering, have become a pivotal intermediate representation in modern speech systems, including Audio LLMs and speech-to-speech translation. However, these units often entangle phonetic, speaker, and language information. In multilingual multi-speaker settings, this entanglement leads to speaker mixing (identity collapse) and cross-lingual interference during speech generation. While upstream research focuses on mapping speech to discrete units, the downstream unit vocoder responsible for waveform reconstruction remains underexplored. Specifically, the interaction between cluster size (inventory), conditioning strategies, and cross-lingual unit sharing lacks systematic analysis, particularly in non-English, multi-speaker contexts.

Methodology
The authors propose a systematic analysis of a BigVGAN-based unit vocoder adapted for multilingual multi-speaker synthesis. The system replaces the standard mel-spectrogram input with discrete unit sequences.

• Architecture & Conditioning: The generator takes discrete unit sequences (derived from k-means clustering of SSL embeddings) as input. To address identity and language issues, the model employs explicit conditioning:
  • Speaker Conditioning: Utilizes continuous speaker embeddings from a pre-trained ECAPA-TDNN model rather than fixed lookup tables, allowing generalization to unseen speakers.
  • Language Conditioning: Uses learnable language embeddings concatenated with unit representations.
  • Auxiliary Loss: A Language Identification (LID) classifier is trained jointly on mel-spectrograms (both real and generated) to enforce language consistency via an auxiliary cross-entropy loss.
• Data & Clustering: Experiments utilize the IndicVoices-R dataset covering four Indian languages (Bengali, Hindi, Tamil, Telugu) from two distinct language families. Discrete units are extracted using the 21st layer of Data2Vec-AQC (pre-trained on 30k hours of Indian speech). Five multilingual k-means models are trained with cluster sizes ranging from 500 to 10,000.
• Evaluation: The study evaluates intelligibility (Word Error Rate via Indic-Conformer ASR), speaker consistency (cosine similarity of ECAPA-TDNN embeddings), and unit-level metrics (Phoneme Purity, Cluster Purity, and Phone-Normalized Mutual Information).

Key Contributions

1. Framework Extension: The authors extend the BigVGAN architecture to function as a multilingual multi-speaker discrete unit vocoder.
2. Conditioning Analysis: They investigate the specific roles of speaker and language conditioning, demonstrating that explicit speaker conditioning is critical for identity preservation, while language supervision aids intelligibility primarily when unit granularity is coarse.
3. Systematic Cluster Analysis: The paper provides a comprehensive trade-off analysis of cluster sizes (500 to 10k) in a unified multilingual setting, moving beyond the English-centric focus of prior work.
4. Comprehensive Metrics: The evaluation goes beyond standard WER, incorporating speaker similarity and unit-level phonetic metrics to characterize the learned representations.

Results

• Cluster Size and Intelligibility: Cluster size is the primary determinant of intelligibility. As cluster size increases, WER decreases significantly (e.g., Bengali WER drops from 60.42% at 500 clusters to 25.13% at 10k clusters in the unconditioned setting). This correlates with increased Phoneme Purity and PNMI, indicating that larger inventories provide better phonetic discriminability.
• Speaker Conditioning: Without explicit speaker conditioning, generated audio suffers from severe identity collapse and audible gender switches, with speaker similarity scores remaining low (~0.16–0.21). Adding ECAPA-TDNN speaker conditioning increases similarity by 4–5× (reaching ~0.70–0.77), effectively disentangling speaker identity from unit representations.
• Language Conditioning: Language supervision yields the most significant gains at smaller cluster sizes (e.g., 1k), where units are ambiguous and cross-lingual overlap is high. At larger cluster sizes (e.g., 10k), where units are already phonetically distinct, language conditioning provides diminishing returns and can occasionally slightly degrade performance.
• Cross-Lingual Sharing: At small cluster sizes (500), acoustically similar phonemes across different languages (e.g., vowels /a:/ and /i/) collapse into identical cluster IDs, creating ambiguity. As cluster size increases to 10k, these phonemes segregate into language-specific IDs, reducing cross-lingual interference.

Significance and Claims
The paper claims that for multilingual multi-speaker unit vocoders, cluster size governs intelligibility by improving phonetic discriminability, while explicit speaker conditioning is indispensable for preventing identity collapse. Language supervision serves as a supplementary mechanism that is most effective when unit representations are coarse and ambiguous. The analysis reveals that smaller unit inventories force cross-lingual phoneme sharing, which introduces ambiguity, whereas larger inventories allow for finer, language-specific separation. These findings offer practical configuration insights for designing unit vocoders in Audio LLMs and speech-to-speech translation pipelines, emphasizing the need for large cluster inventories and explicit speaker conditioning to achieve high-quality, consistent multilingual synthesis.