💻 computer science

Acoustic and perceptual differences between standard and accented Chinese speech and their voice clones

This study reveals that while standard and accented Mandarin voice clones show similar embedding-based distances to their originals, perceptual evaluations demonstrate that accent significantly influences both the perceived identity match and intelligibility improvements, suggesting these should be evaluated as distinct dimensions in voice cloning.

Original authors: Tianle Yang, Chengzhe Sun, Phil Rose, Siwei Lyu

Published 2026-04-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tianle Yang, Chengzhe Sun, Phil Rose, Siwei Lyu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a magical photocopier, but instead of copying paper, it copies voices. This technology, called "voice cloning," can take a 20-second recording of someone speaking and generate new sentences that sound exactly like them.

But here's the catch: What happens if the person has a strong regional accent? Does the magic photocopier keep that unique "flavor," or does it accidentally smooth it out into a generic, standard accent? And does that change how we feel about the voice?

This paper investigates exactly that, using Mandarin Chinese speakers with heavy regional accents versus those with the "standard" accent. The researchers used two different ways to check the results: a computer robot (math) and human listeners (ears).

Here is the breakdown of their findings using some simple analogies:

1. The Computer's View: The "ID Card" Scanner

The researchers first asked a computer system (an AI that acts like a voice ID scanner) to measure how similar the original voice was to the cloned voice.

The Analogy: Imagine the computer is a bouncer at a club checking ID cards. It looks at the "voice ID" of the original person and the cloned person.
The Finding: The bouncer said, "They look the same." Whether the person had a heavy accent or a standard accent, the computer couldn't tell much difference between the original and the clone. The "distance" between the two voices on the computer's map was roughly the same for everyone.
The Takeaway: To the math, the accent didn't seem to disappear. The computer thought the clones were faithful copies.

2. The Human's View: The "Ear" Test

Next, the researchers asked real people to listen to the recordings and rate them. They asked two questions:

Similarity: "Does this sound like the same person?"
Intelligibility: "How easy is it to understand what they are saying?"

The Analogy: Imagine you are listening to a friend tell a story.
- Scenario A (Standard Accent): Your friend speaks clearly. The clone sounds just like them. You say, "Yep, that's definitely them!"
- Scenario B (Heavy Accent): Your friend has a thick accent that makes some words hard to catch. The clone speaks the same words, but suddenly, the accent feels "softer" or more like the standard version.
The Finding:
- Similarity: Humans noticed something the computer missed. They felt the clones of accented speakers sounded less like the original person compared to the standard speakers. It was as if the clone had "washed out" the unique regional flavor, making the speaker feel like a stranger.
- Intelligibility: However, the humans also noticed that the accented clones were much easier to understand than the original recordings. The "magic photocopier" had accidentally cleaned up the speech, making it clearer.

3. The Big Twist: The "Smoothie" Effect

Why did the computer say "Same!" while humans said "Different"?

Think of a voice with a heavy accent like a fruit smoothie with big chunks of fruit.

The Computer (ID Scanner): It only checks the color and the liquid base. It sees the red color and says, "This is a strawberry smoothie." It doesn't care about the chunks.
The Human (Taster): The human tastes the texture. When the voice is cloned, the AI seems to "blend" the smoothie, smoothing out the big chunks (the heavy accent) to make it run more smoothly.
- Result 1: Because the chunks are gone, the smoothie tastes "smoother" and easier to drink (higher intelligibility).
- Result 2: But because the unique chunks are gone, the taster says, "This doesn't taste exactly like my specific smoothie anymore" (lower similarity).

4. Why This Matters

The paper concludes that voice cloning isn't just about copying a voice; it's about copying a person.

The Problem: Current AI tools are great at making voices sound clear, but they might be accidentally "standardizing" people. If you have a strong accent, the AI might make you sound more like a generic news anchor and less like you.
The Safety Risk: If a clone sounds too "clean" and loses the accent, it might be easier to spot as a fake. But if the accent is preserved perfectly, it might be harder to detect, but also harder to understand.
The Solution: We need to test voice cloning in two separate ways:
1. Does it sound like the person? (Identity)
2. Does it sound like the accent? (Style)

In short: The computer thinks the clones are perfect copies. But our ears tell us that for accented speakers, the clones are a bit too "cleaned up." They are easier to understand, but they feel a little less like the real person.

1. Problem Statement

Voice cloning technology is increasingly used for accessibility, dubbing, and personalized avatars, but it also poses security risks regarding impersonation and fraud. While existing research focuses on detection, robustness, and overall synthesis quality, there is a significant gap in understanding how accent variation is represented and altered in voice-cloned speech.

The core research questions address:

Do voice clones of heavily accented speakers diverge more from their source in speaker embedding space than clones of standard speakers?
Is this divergence consistent across different commercial voice-cloning systems?
How does accent attenuation (if it occurs) affect human perception regarding speaker identity similarity and speech intelligibility?

2. Methodology

The study employs a mixed-method approach combining computational analysis of speaker embeddings with a controlled human perception experiment.

A. Data and Stimuli Generation

Source Data:
- Standard Mandarin: Sourced from the AISHELL-3 corpus.
- Accented Mandarin: Sourced from the Mandarin Heavy Accent Speech Corpus.
Voice Cloning Systems: Three commercial systems were utilized: ElevenLabs, MiniMax, and AnyVoice.
Protocol:
- Fixed enrollment duration (~20 seconds) per speaker.
- Identical preprocessing (mono, 16kHz resampling, amplitude normalization) for all inputs.
- Fixed text prompts and synthesis settings to ensure comparability.
- The exact same audio stimuli were used for both computational and perceptual analyses to eliminate material variance.

B. Computational Analysis (Embedding Space)

Model: ECAPA-TDNN (via SpeechBrain toolkit) was used to generate 192-dimensional speaker embeddings.
Metric: Cosine distance ( $d_{cos}$ ) was calculated between embeddings.
Distance Sets:
- $d_{OO}$ : Original vs. Original (baseline intra-speaker variance).
- $d_{CC}$ : Clone vs. Clone.
- $d_{OC}$ : Original vs. Clone.
Divergence Measure: Defined as $\Delta_{div} = d_{OC} - d_{OO}$ . Positive values indicate the clone is further from the original than the original is from itself.
Statistical Approach: Linear mixed-effects models (REML) were fitted on speaker-level mean distances to test for differences between standard/accented conditions and across systems.

C. Perceptual Experiment

Participants: $N=67$ native Mandarin speakers (after exclusions) with normal hearing.
Tasks:
1. Speaker Similarity: Rated on a 1–5 scale how similar the clone sounded to the original speaker.
2. Intelligibility: Rated on a 1–5 scale how understandable the speech was.
Analysis: Cumulative Link Mixed Models (CLMM) were used to analyze ordinal ratings, accounting for random intercepts for listeners and stimuli.

3. Key Results

A. Computational Findings (Embedding Distances)

No Reliable Accent Effect: The ECAPA-TDNN embedding distances did not show a statistically significant difference between standard and accented speakers regarding the divergence between original and clone ( $d_{OC}$ ).
System Dependence:
- ElevenLabs: Showed significant clone divergence ( $\Delta_{div} \approx 0.21$ ) for both standard and accented speech.
- AnyVoice & MiniMax: Showed negligible divergence ( $\Delta_{div} \approx 0$ ), meaning the clones were embedded very close to the originals in this specific vector space.
Conclusion: Standard off-the-shelf speaker embeddings are insensitive to accent-related modifications introduced by cloning.

B. Perceptual Findings

Speaker Similarity (Identity Match):
- Listeners rated clones of standard speakers as significantly more similar to the original than clones of accented speakers.
- This asymmetry was significant for AnyVoice and MiniMax, but less pronounced for ElevenLabs.
- Contradiction: This perceptual difference exists despite the computational analysis showing no difference in embedding distances.
Intelligibility:
- Clones were rated as more intelligible than the original recordings overall.
- Accent Gain: The improvement in intelligibility was significantly larger for accented speech than for standard speech.
- This suggests cloning effectively "standardizes" the speech, making it easier to understand for listeners.

4. Key Contributions

Decoupling Identity and Accent: The study demonstrates that speaker identity preservation and accent preservation are separable dimensions in voice cloning. A clone can be perceived as less "faithful" to the speaker's identity (due to accent loss) even if the embedding distance suggests high similarity.
Limitations of Embedding Metrics: It highlights that current speaker verification embeddings (like ECAPA-TDNN) may downweight accent cues, failing to capture the perceptual reality where accent is a critical component of speaker identity.
Intelligibility vs. Authenticity Trade-off: The research identifies a specific trade-off: voice cloning appears to attenuate heavy accents, thereby improving intelligibility but reducing the perceived authenticity of the speaker's regional identity.

5. Significance and Implications

For Safety and Ethics: Current detection methods relying solely on embedding distances may fail to identify "accent-altered" deepfakes. If a clone sounds more "standard" than the original, it might be perceived as a different person by humans, even if the algorithm says it is the same.
For Accessibility: Voice cloning offers a potential tool to improve the intelligibility of heavily accented speech for broader audiences, though this comes at the cost of erasing regional linguistic identity.
Future Evaluation: The authors argue that future evaluations of voice cloning must treat accent preservation as a distinct metric from speaker identity preservation, rather than assuming they are perfectly correlated. Standard acoustic measures and explicit accent ratings are needed alongside embedding analysis.