⚡ electrical engineering

Disentangling Speaker and Language Effects in Cross-Lingual Speaker Verification for Iberian Languages

This paper introduces a bilingual same-speaker evaluation set for five Iberian languages to disentangle speaker and language effects in cross-lingual speaker verification, revealing that while speaker variability contributes to performance degradation, language mismatch remains the primary cause of cross-lingual loss.

Original authors: Pol Buitrago, Javier Hernando

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pol Buitrago, Javier Hernando

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart security guard whose job is to recognize people just by their voice. Usually, this guard is great at spotting a friend whether they are whispering, shouting, or singing. But what happens if your friend speaks to the guard in English, but then tries to prove their identity by speaking in Spanish? The guard often gets confused and says, "Wait, that doesn't sound like the same person!"

This paper investigates exactly that confusion. It asks: Is the guard confused because the person's voice actually changed, or because the language they are speaking changed the sound of their voice?

Here is the breakdown of their investigation using simple analogies:

The Problem: Mixing Up the "Who" and the "What"

In the past, scientists tested these voice guards by having Person A speak Language 1 and Person B speak Language 2. If the guard failed, they didn't know if it was because Person A and Person B sounded different (the "Who" problem) or because Language 1 and Language 2 sound different (the "What" problem). It was like trying to taste the difference between two soups, but you were also changing the bowls, the spoons, and the temperature all at once.

The Solution: The "Same Speaker" Test

To fix this, the researchers created a special test using five languages from the Iberian Peninsula (Spain and Portugal): Spanish, Catalan, Galician, Basque, and Portuguese.

They found people who speak two of these languages fluently. They asked these same people to speak both languages to the voice guard.

The Setup: Imagine asking your friend to say "Hello" in English, and then immediately say "Hola" in Spanish. Since it's the exact same person, any confusion the guard feels must be because of the language switch, not because a different person is talking.

The Tool: The "Transfer Map"

The researchers used a special scoring system they call the Cross-Lingual Transfer Matrix (CLTM). Think of this as a heat map or a compatibility chart.

It measures how well the voice guard learns from one language to understand another.
If the guard learns perfectly from Spanish to understand Catalan, the score is high.
If the guard gets totally confused (like trying to understand a foreign accent without any training), the score is low or even negative.

What They Found

Language is the Main Culprit: Even when using the exact same person, the voice guard still got confused when the language changed. This proves that the "language mismatch" is the biggest reason for errors. The guard has learned that "Spanish voices" sound a certain way and "Portuguese voices" sound another way, and it struggles to ignore those differences.
Speaker Variability Matters Too: When they compared the "Same Person" test to the old "Different People" test, they found that the old tests were extra confusing. This means that having different people talk in different languages makes the problem even worse. It's like trying to recognize a friend's voice over a bad phone line (language change) while they are also wearing a different costume (different speaker).
Not All Languages Are Equally Confusing:
- Spanish and Galician: These are like twins. They sound very similar. The voice guard barely noticed the switch.
- Spanish and Portuguese: These are like cousins who grew up in different houses. They share a family tree, but they have developed very different habits (like nasal sounds in Portuguese). The voice guard got very confused here.
- Spanish and Basque: Basque is a bit of an outlier (it's not even related to the others). The voice guard struggled significantly, especially because Basque has different consonant sounds that the Spanish-trained guard wasn't expecting.

The "Voice Shift" Discovery

The researchers looked inside the computer's "brain" (the mathematical space where voices are stored). They found that even when the same person speaks two different languages, their "voice signature" physically moves to a different spot in that space.

Analogy: Imagine your voice is a dot on a map. When you speak Spanish, the dot is in Paris. When you speak Portuguese, the dot moves to Madrid. The guard is trained to recognize dots in Paris. When the dot moves to Madrid, the guard thinks, "That's not my friend!" even though it is.

The Bottom Line

The paper concludes that while having different speakers makes cross-language voice recognition harder, the language itself is the main reason the technology fails. Even if you use the exact same person, the computer still struggles to recognize them when they switch languages because the "acoustic fingerprint" of the language changes the sound of the voice.

The researchers didn't propose a new app or a medical use for this; they simply provided a clearer way to measure why these systems fail, showing that we need to teach computers to ignore the "accent" of the language to truly recognize the "voice" of the person.

Technical Summary: Disentangling Speaker and Language Effects in Cross-Lingual Speaker Verification for Iberian Languages

Problem Statement

Cross-lingual speaker verification (SV) systems typically suffer from performance degradation when enrollment and test utterances are spoken in different languages. While SV is theoretically expected to rely on extralinguistic cues independent of linguistic content, empirical studies indicate that current models learn representations that are not fully language-agnostic. A critical limitation in existing research is that standard evaluation protocols confound language mismatch with inter-speaker variability. Because large-scale corpora containing the same individuals speaking multiple languages are difficult to collect, standard benchmarks evaluate cross-lingual performance using different speakers for each language. Consequently, observed performance drops cannot be clearly attributed solely to language differences versus speaker variability.

Methodology

To address this confounding factor, the authors propose a bilingual same-speaker evaluation framework applied to five Iberian languages: Spanish, Catalan, Galician, Basque, and Portuguese.

Data Construction:

Source: Mozilla Common Voice corpus (version 25.0).
Training: Strictly balanced subsets with equal sample counts and disjoint speaker identities per language. Each speaker contributes 50 utterances to prevent dataset imbalance or frequency effects.
Evaluation: The authors constructed a bilingual same-speaker test set by selecting speakers who recorded utterances in Spanish and at least one additional Iberian language. This allows for controlled cross-lingual trials where speaker identity is constant.
Dynamic Training Interval: To ensure reliable measurement of performance gains, the study identifies a "dynamic" training interval $[N, 2N]$ (specifically 1,000 to 2,000 samples per language) where performance increases significantly before saturation.

Model Architecture:

Base Model: A multilingual mHuBERT-1471 encoder pretrained on 147 languages.
Fine-tuning: The encoder is fine-tuned for speaker identification using a randomly initialized linear classification head and cross-entropy loss. The head is discarded post-training, and the encoder serves as the embedding extractor.
Training Protocol: Models are trained for a single epoch with AdamW optimization. To mitigate initialization and data composition effects, experiments are repeated across 20 independent data samplings and 5 random seeds (100 runs total).
Embedding Extraction: Temporal pooling of transformer hidden representations followed by L2 normalization. Verification is performed via cosine similarity.

Analysis Framework:
The study utilizes the Cross-Lingual Transfer Matrix (CLTM) to quantify pairwise cross-lingual transfer. The CLTM measures the performance gain on a target language $i$ when adding donor language $j$ data, normalized by the gain from adding equivalent target language data.

Metrics: The analysis employs Relative Frobenius Deviation (RFD1) to measure deviation from language-agnostic behavior, Relative Asymmetry (Asymrel) for directional differences, and Average Row Cosine Similarity (cosrows) for consistency.
Embedding Shift Analysis: For bilingual speakers, the authors compute per-language centroids and measure the displacement vector ( $\delta$ ) between languages to quantify language-induced shifts in the embedding space.

Key Results

1. Language Dependence Persists:
Under standard evaluation (different speakers), the CLTM reveals strong language dependence. Spanish, Catalan, and Galician form a positive transfer cluster, while Basque and Portuguese exhibit negative transfer. The Iberian subset shows reduced deviation from the language-agnostic ideal compared to a 44-language global model but retains a clearly asymmetric structure.

2. Impact of Speaker Variability:
When restricting evaluation to the same bilingual speakers:

Romance Languages (Spanish-Catalan, Spanish-Galician): The bilingual setting consistently reduces both the deviation (RFD1) and asymmetry (Asymrel) of the CLTM. This indicates that a portion of the cross-lingual degradation in standard evaluations is attributable to speaker variability.
Distant/Non-Romance Languages (Spanish-Basque, Spanish-Portuguese): The effect of speaker control is limited. Spanish-Basque remains strongly asymmetric (with RFD1 slightly increasing), and Spanish-Portuguese retains substantial negative transfer. This suggests that for these pairs, language mismatch is the dominant driver of degradation, independent of speaker identity.

3. Embedding Shifts:
Even with the same speaker, switching languages induces systematic shifts in the embedding space.

Portuguese: Exhibits the largest displacement magnitude and highest directional consistency, correlating with the strongest cross-lingual degradation.
Galician: Shows the smallest and least structured shifts, correlating with minimal degradation.
Catalan and Basque: Present intermediate geometric patterns. Notably, the embedding shifts alone do not fully explain the degradation patterns, as Catalan and Basque show similar shift geometries despite different CLTM behaviors.

4. Linguistic Correlation:
The observed transfer patterns align with phonological relationships:

Spanish-Galician: High phonological proximity (overlapping consonants, aligned vowel system) results in minimal shifts and degradation.
Spanish-Portuguese: Maximal phonological divergence (nasal vowels, schwa-like sounds, vowel reduction) leads to the strongest degradation and shifts.
Spanish-Basque: Despite sharing a five-vowel system, divergence in consonantal structure (sibilants, affricates) causes significant representational strain.

Significance and Claims

The paper claims to provide a more precise characterization of language dependence in cross-lingual SV by disentangling speaker variability from language mismatch. The primary findings are:

Speaker variability accounts for part of the observed degradation: Controlling for speaker identity improves cross-lingual performance, particularly for closely related languages.
Language mismatch remains the main driver: Even under same-speaker conditions, significant performance loss persists, driven by intrinsic language effects and phonological divergence.
Systematic Embedding Shifts: Language switching induces systematic shifts in the embedding space even for the same speaker, confirming that current multilingual models do not fully isolate speaker identity from linguistic content.

The work establishes a bilingual same-speaker evaluation set for Iberian languages and demonstrates that while speaker variability contributes to cross-lingual gaps, the fundamental limitation in current SV systems is their inability to be fully language-agnostic.