Technical Summary: AccentBox: Towards High-Fidelity Zero-Shot Accent Generation

1. Problem Statement

Recent advances in Zero-Shot Text-to-Speech (ZS-TTS) have achieved high naturalness and speaker similarity, allowing for the synthesis of unseen speakers' voices from short audio clips. However, existing systems largely ignore accent variation, often training on biased datasets (e.g., American English) without explicit accent conditioning. This results in poor accent fidelity and a lack of control over the generated accent.

Current approaches to accented speech generation fall into three categories, each with significant limitations:

• Foreign Accent Conversion (FAC): Converts L2 accents to L1 accents but cannot handle arbitrary text or unseen accent pairs.
• Accented TTS: Generates accented speech but is limited to seen speakers and seen accents.
• Standard ZS-TTS: Generates speech for unseen speakers but fails to control or accurately reproduce accents, often leading to accent hallucination or mismatch.

Furthermore, a critical research gap exists regarding speaker-accent entanglement. In both Accent Identification (AID) and ZS-TTS, models often conflate speaker identity with accent, failing to generalize to unseen speakers or disentangle the two attributes. Additionally, existing benchmarks often suffer from data leakage (speaker overlap between training and testing sets), obscuring the true generalization capabilities of AID models.

2. Methodology

The authors propose AccentBox, a novel two-stage pipeline that unifies FAC, accented TTS, and ZS-TTS into a single Zero-Shot Accent Generation task. This task aims to generate any speech content in any given voice and accent from a single audio clip.

Stage 1: Generalisable Accent Identification (GenAID)

The first stage focuses on extracting continuous, speaker-agnostic accent embeddings. The authors propose GenAID, an improved AID model based on the XLSR encoder, designed to overcome speaker-accent entanglement. Key modifications include:

1. Validation on Unseen Speakers: The model is validated exclusively on speakers not present in the training set to prevent memorization of speaker-to-accent mappings.
2. Weighted Sampling: Addresses imbalanced accent label distributions by sampling with probabilities inversely proportional to accent frequency.
3. Data Augmentation: Applies speed and noise perturbations to increase robustness against recording conditions.
4. Information Bottleneck: A two-layer MLP maps high-dimensional encoder outputs to a lower-dimensional (64-dim) embedding, removing redundant information.
5. Adversarial Training: The model is trained to be maximally uncertain about speaker identity using a Mean Square Error (MSE) loss against a uniform distribution, effectively disentangling speaker information from accent features.

Stage 2: Zero-Shot Accent Generation (AccentBox)

The second stage conditions a ZS-TTS system (based on YourTTS) on the pretrained GenAID embeddings.

• Architecture: The system replaces standard one-hot language embeddings with the continuous GenAID accent embeddings. These embeddings condition both the Transformer Text Encoder and the Stochastic Duration Predictor.
• Inference Scenarios: The system supports three modes:
  • Inherent Generation: Generating speech with the target speaker's natural accent.
  • Cross-Accent Generation: Generating a target speaker's voice with a different, specified accent.
  • Unseen Accent Generation: Generating speech with an accent not seen during training.

3. Key Contributions

The paper claims the following contributions:

• Task Definition: Introduction of the Zero-Shot Accent Generation task, which unifies the capabilities of FAC, accented TTS, and standard ZS-TTS.
• Benchmarking: Establishment of the first benchmark for this task, including the verification and quantification of speaker-accent entanglement in AID data/models and the identification of accent mismatch/hallucination issues in ZS-TTS.
• Novel Disentanglement: Implementation of speaker-accent disentanglement in AID using information bottlenecks and adversarial training, achieving speaker-agnostic accent embeddings.
• System Performance: Demonstration of a system capable of high-fidelity zero-shot accent generation without reliance on pre-collected accent labels in TTS data.

4. Experimental Results

GenAID Performance

• Classification: On unseen speakers (13-accent classification), the best GenAID system achieved an F1 score of 0.56 and accuracy of 0.56, significantly outperforming the random baseline (0.08) and previous systems.
• Generalization: The gap in accuracy between seen and unseen speakers was reduced from 0.53 (baseline) to 0.06 (proposed), indicating superior generalization.
• Disentanglement: The Silhouette Coefficient for Speaker Clusters (SCSC) dropped from 0.236 to 0.079, confirming reduced residual speaker information in the accent embeddings.

AccentBox Performance

• Objective Evaluation (Inherent & Cross-Accent): The proposed system achieved the highest Accent Cosine Similarity (AccCos) scores (0.9336 for inherent, 0.9067 for cross-accent) compared to baselines like VALL-E X and a standard YourTTS baseline.
• Subjective Evaluation (Inherent):
  • Accent Similarity: The proposed system was preferred over the Baseline (69.1% for US, 61.3% for Irish) and the Accent_ID system (57.4% for US, 65.7% for Irish).
  • Speaker Similarity: Preferences were mixed; the system was preferred over the Baseline but showed lower objective speaker cosine similarity, potentially due to ASR bias or listener difficulty distinguishing accent from identity.
  • Naturalness: The system outperformed baselines for US accents but received lower preference for Irish accents, attributed to limited training data for Irish and sensitivity to prosody.
• Subjective Evaluation (Cross-Accent): The proposed system demonstrated superior accent fidelity (70.0% preference for US, 61.7% for Irish) compared to the Accent_ID system, which relies on discrete one-hot labels and struggles with consistency.

5. Significance and Claims

The paper positions AccentBox as a significant step forward in inclusive speech technology. By addressing the speaker-accent entanglement issue, the authors claim to have created a system that:

1. Preserves linguistic identity for native speakers and alleviates pressure to conform to native accents for non-native speakers.
2. Enables applications in personalized virtual assistants, movie dubbing, and Computer-Aided Pronunciation Training (CAPT).
3. Provides separate control over speaker and accent, allowing for the generation of unseen accents and voices without requiring pre-collected accent labels in the TTS training data.

The authors note that while the system shows promise, future work is required to experiment with L2 accent generation/conversion and to refine evaluation methods to better understand discrepancies in naturalness across different accents.