LASE: Language-Adversarial Speaker Encoding for Indic Cross-Script Identity Preservation

1. Problem Statement

Multilingual voice cloning and diarisation systems rely on speaker encoders to identify a speaker's identity regardless of the language or script being spoken. However, off-the-shelf speaker encoders (e.g., WavLM-base-plus-sv, ECAPA-TDNN) fail significantly when applied to Indic scripts (Devanagari, Telugu, Tamil) compared to Latin scripts.

• The Failure Mode: When the same speaker switches scripts (e.g., from English to Hindi), standard encoders treat the voice as a different identity.
• Quantitative Impact: On a Western-accented corpus, WavLM loses 0.082 absolute cosine similarity, and ECAPA-TDNN loses 0.105, when comparing same-voice clips across different scripts. This gap is large relative to the margin between same-voice and different-voice pairs, causing production failures such as:
  • Splitting a single customer support agent into multiple identities during code-switching calls.
  • Voice cloning sounding like "a different voice with an accent" rather than the same person.
• Root Cause: Standard encoders are trained primarily on English (Latin script) data (e.g., VoxCeleb). Their embeddings are entangled with phonetic structures specific to English. When exposed to Indic phonemes (e.g., retroflex stops), the embedding drifts, leaking language information into the identity vector.

2. Methodology: LASE (Language-Adversarial Speaker Encoder)

The authors propose LASE, a lightweight architecture designed to disentangle speaker identity from language/script information using Domain-Adversarial Training.

Architecture

LASE consists of three components:

1. Frozen Backbone: Uses WavLM-base-plus (truncated at the transformer stack) to extract frame-level features. This backbone is frozen to preserve pre-trained acoustic knowledge.
2. Trainable Projection Head: A two-layer MLP (768 $\to$ 512 $\to$ 256) with ReLU and dropout. It maps the backbone features to a 256-dimensional speaker embedding ($z$).
3. Gradient-Reversal Language Classifier: A small MLP that attempts to predict the language (English, Hindi, Telugu, Tamil) from the embedding $z$. Crucially, a Gradient Reversal Layer (GRL) is inserted between $z$ and the classifier. During backpropagation, the GRL multiplies the classifier's loss gradient by $-\lambda_t$, forcing the projection head to minimize the classifier's accuracy (i.e., make the embedding language-uninformative) while maximizing speaker discrimination.

Loss Functions

The model is optimized using a sum of two losses:

1. Supervised Contrastive Loss ($L_{spk}$): Pulls embeddings of the same speaker (across different scripts) together and pushes different speakers apart.
2. Adversarial Language Loss ($L_{lang}$): Cross-entropy loss for the language classifier. Due to the GRL, minimizing the total loss forces the encoder to maximize $L_{lang}$ (making the classifier guess randomly, $\approx \ln 4$) while minimizing $L_{spk}$.

Training Schedule

• Warmup: 200 steps with $\lambda=0$ (only speaker loss) to establish a coherent voice geometry.
• Ramp: Linear increase of $\lambda$ from 0 to 0.1 over 500 steps.
• Hold: $\lambda=0.1$ for the remainder.
• Total Cost: Trained for 1,000 steps on a single A10G GPU (~17 minutes, $0.31).

3. Data and Corpus Construction

A major challenge was the lack of natural, large-scale, same-speaker cross-script datasets for Indic languages.

• Synthetic Bootstrapping: The authors synthesized data using ElevenLabs Multilingual v3.
• Setup: 8 distinct multilingual voices generated 50 sentences each in 4 languages (English, Hindi, Telugu, Tamil).
• Quality Gate: Pairs were filtered using a WavLM cosine similarity threshold (0.90) against a reference English clip to ensure the synthesizer preserved identity.
• Resulting Datasets:
  • Training: 1,118 high-quality cross-script pairs.
  • Held-out (Western Accent): 1,043 pairs (same 8 voices, unseen sentences).
  • Held-out (Indian Accent): 1,369 pairs (8 different Indian-accented voices, unseen sentences).

4. Key Contributions

1. First Indic Cross-Script Benchmark: Release of the first same-speaker identity benchmark for Indic languages (1,118 training / 1,043 held-out pairs) constructed via TTS bootstrapping.
2. Measurement Framework: A three-distribution framework (Within-script, Cross-script, Across-speaker) to isolate language-identity entanglement from other confounders.
3. LASE r1 Model: A 256-dim encoder that reduces the cross-script identity gap by 84.3% relative to baselines.
4. Data Efficiency: Achieves cross-script speaker recall matching ECAPA-TDNN (trained on 1M+ VoxCeleb utterances) using ~100x less data (1,118 pairs).
5. Open Source: Release of weights, corpora, and training scripts under MIT/CC-BY-4.0 licenses.

5. Results

The evaluation uses two metrics: Gap ($\Delta$) (drop in similarity from within-script to cross-script; lower is better) and Margin ($M$) (distance from cross-script similarity to the noise floor; higher is better).

Metric	WavLM-base-plus-sv	ECAPA-TDNN	LASE r1 (Ours)
Western Gap ($\Delta$)	0.082	0.105	0.013 (CI includes 0)
Western Margin ($M$)	0.245	0.202	0.662
Indian Gap ($\Delta$)	0.006	0.044	0.026 (CI includes 0)
Indian Margin ($M$)	0.144	0.256	0.344

• Gap Closure: LASE reduces the Western-accent gap from 0.082 to 0.013 (statistically indistinguishable from zero).
• Diarisation Performance: In a synthetic code-switching diarisation task, LASE achieved a cross-script recall of 0.788, statistically tied with ECAPA-TDNN (0.789), despite using 100x less training data.
• Ablation Study:
  • Applying GRL to ECAPA improved it, but WavLM+GRL performed better.
  • The combination of the WavLM backbone and the GRL objective is necessary; neither alone achieves the full effect.
  • WavLM embeddings were easier to make language-invariant than ECAPA embeddings (language loss stayed pinned at random chance for WavLM, but oscillated for ECAPA).

6. Significance and Limitations

Significance:

• Production Impact: Solves a critical failure mode in Indian multilingual TTS and call-center diarisation where script switching causes identity fragmentation.
• Efficiency: Demonstrates that explicit adversarial objectives are more sample-efficient than relying on massive corpus scale to implicitly learn invariance.
• Accessibility: Provides a low-cost, reproducible recipe for building cross-script invariant encoders for any script set where TTS synthesis is viable.

Limitations:

• Synthetic Data: Training and evaluation rely on ElevenLabs synthetic audio. While the gap is real in this distribution, real-world acoustic variability (microphones, background noise, natural prosody) is not yet tested.
• Voice Generalization: The held-out sets use the same voices as training (unseen sentences only). Generalization to completely new voices is a "v2" experiment.
• Code-Mixing: The model was not tested on intra-sentence code-mixing (e.g., English words inside a Hindi sentence), which is common in Indian speech.

Conclusion:
LASE successfully decouples speaker identity from Indic script variations using a small, adversarially trained projection head. It closes the cross-script identity gap to near-zero and matches the performance of massive, production-grade encoders with a fraction of the data, offering a practical solution for multilingual voice technologies in the Indian context.