⚡ electrical engineering

TokAN: Accent Normalization Using Self-Supervised Speech Tokens

TokAN is a self-supervised, token-based accent normalization framework that converts non-native speech to standard accents using an autoregressive encoder-decoder and reinforcement learning, achieving superior intelligibility and accent reduction without requiring parallel training data or synthetic targets.

Original authors: Qibing Bai, Shuai Wang, Yuhan Du, Bohan Li, Yannan Wang, Haizhou Li

Published 2026-07-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qibing Bai, Shuai Wang, Yuhan Du, Bohan Li, Yannan Wang, Haizhou Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Fixing the "Accent" Without Losing the "Voice"

Imagine you are listening to a friend tell a story, but they have a strong accent that makes some words hard to understand. You want them to sound like they are speaking "standard" English (like a news anchor), but you still want it to sound like your friend telling the story, not a robot or a different person.

This is the problem Accent Normalization tries to solve. The paper introduces a new system called TokAN that does this better than previous methods.

The Problem with Old Methods

Before TokAN, fixing accents was like trying to copy a painting by hand:

The "Reference" Problem: Some old methods needed a recording of a native speaker saying the exact same sentence to use as a guide. This is like needing a photo of the original painting to copy it. In the real world, you rarely have that perfect match.
The "Synthetic" Problem: Other methods tried to use computer-generated speech as a guide. But computer voices often sound robotic or have weird timing. If you teach a model using these "fake" voices, the model learns those robotic errors, making the final result sound unnatural.

The TokAN Solution: The "Digital Lego" Approach

TokAN changes the game by not working with raw sound waves (like a continuous audio file). Instead, it breaks speech down into discrete tokens.

The Analogy:
Think of speech not as a smooth river of sound, but as a sentence written in Morse code or Minecraft blocks.

Tokens: These are the individual "blocks" or "dots and dashes" that represent sounds (phonemes).
The Magic: These blocks contain the meaning of the word but strip away the messy details of how it was said (the specific accent, the breathing, the exact pitch).

How TokAN Works (The 3-Step Process)

1. The Translator (The Tokenizer)

First, the system listens to the accented speech and converts it into these "blocks" (tokens).

The Innovation: In the past, these blocks were assigned randomly (like sorting Lego bricks by color without a plan). TokAN uses a Jointly Trained VQ Tokenizer.
The Analogy: Imagine a master craftsman who doesn't just sort the bricks; they design the bricks specifically so that when you build a house later, the walls are straight and the roof fits perfectly. This tokenizer is trained alongside the speech synthesizer to ensure the "blocks" capture the perfect amount of detail—enough to understand the words, but not so much that the accent gets stuck in the data.

2. The Converter (The Encoder-Decoder)

This is the brain of the operation. It takes the "accented blocks" and rearranges them into "standard blocks."

The Innovation: It uses a special type of AI (an autoregressive encoder-decoder) that looks at the whole sequence of blocks at once.
The Analogy: Think of this as a translator who reads a sentence written in "French-accented English" and rewrites it in "Standard English" without changing the story. Crucially, this translator is accent-universal. It doesn't need to know which accent the speaker has (Chinese, Indian, Spanish); it just looks at the blocks and figures out the standard version on its own.

3. The Builder (The Synthesizer)

Once the blocks are converted to "standard," the system needs to turn them back into sound.

The Innovation: It uses a Flow-Matching Synthesizer.
The Analogy: If the tokens are the blueprint, this is the construction crew. It builds the audio wave.
The "Dubbing" Feature: One of the coolest parts is Duration Control. Sometimes you need the speech to take up the exact same amount of time as the original (like for movie dubbing). TokAN has a special "time-manager" that can stretch or shrink the speech to fit a specific time limit without making it sound like a chipmunk or a sloth.

The Secret Sauce: Reinforcement Learning (The "Coach")

After the system is trained, the authors added a final step called Reinforcement Learning (RL) using a method called GRPO.

The Analogy: Imagine a student who has studied hard (Supervised Fine-Tuning) but still makes small mistakes. Now, they get a Coach.
How it works: The system generates a few versions of the speech. The Coach (an AI judge) listens and gives points based on two things:
1. Did it say the right words? (Low Word Error Rate).
2. Does it sound native? (Accent Classifier confidence).
The system tries again and again, getting "points" for doing better. This teaches the system to fix subtle accent issues that standard training missed, without needing any new human data.

The Results: Why It Matters

The paper tested TokAN on people speaking English with seven different accents (like Chinese, Spanish, Korean, etc.).

Better Clarity: The system reduced the "Word Error Rate" (how often a computer misunderstands the speech) significantly more than any previous method.
More Natural: It sounded more like a native speaker and less like a robot.
Kept the Identity: Even though the accent changed, the listener could still tell it was the original speaker's voice.

Summary

TokAN is like a smart, magical translator that:

Breaks speech into simple "blocks" to ignore the accent noise.
Rearranges those blocks into standard English.
Rebuilds the sound using a high-quality construction crew.
Hires a coach to practice until the accent is gone, but the speaker's unique voice remains.

It solves the problem of needing perfect matching recordings or suffering from robotic-sounding computer voices, making it a major step forward for things like movie dubbing and language learning.

Technical Summary: TokAN: Accent Normalization Using Self-Supervised Speech Tokens

Problem Statement

Accent Normalization (AN), or Foreign Accent Conversion (FAC), aims to convert non-native (L2) accented speech into standard native (L1) speech while preserving the original speaker's identity. Existing approaches face significant limitations:

Reference-based methods require native speech samples at inference, hindering practical deployment.
Reference-free methods often rely on naturally recorded parallel L1–L2 corpora, which are scarce.
Synthetic supervision methods (using cascaded ASR-TTS pipelines) suffer from quality degradation. Direct speech supervision propagates voice cloning artifacts and prosodic errors from synthetic targets, while synthetic duration information often fails to reflect authentic native rhythms.
Token-based methods using pre-trained self-supervised learning (SSL) models with K-Means clustering may not yield code assignments optimally discriminative for accent-specific phonetic content. Furthermore, supervised fine-tuning (SFT) with cross-entropy loss is a weak signal for the ultimate goals of content preservation and accentedness reduction.

Methodology

The authors propose TokAN, a comprehensive framework operating on self-supervised discrete speech tokens. The system consists of three main components and a four-stage training pipeline.

1. Jointly Trained VQ Tokenizer

Instead of using offline K-Means clustering on frozen SSL features, TokAN employs a learnable Vector Quantization (VQ) codebook.

Architecture: A frozen WavLM-Large SSL encoder extracts frame-level features, which pass through a shallow 1-D ResNet and a VQ bottleneck to produce discrete tokens.
Joint Training: The tokenizer is co-optimized with a flow-matching speech synthesizer and a CTC-based ASR module on native English speech. The objective combines VQ reconstruction loss, synthesis loss, and recognition loss.
Benefit: This ensures the codebook captures phonetically rich representations suitable for both high-fidelity reconstruction and content preservation, implicitly providing an initial normalization of accented speech.

2. Autoregressive Token Conversion Model

This module transforms L2-accented token sequences into L1-accented sequences.

Architecture: An encoder-decoder Transformer. The encoder uses Rotary Positional Encodings (RoPE) and is supervised by a CTC phoneme prediction loss to reinforce phonemic content. The decoder is self-attention-only, treating the encoder output as a prefix rather than using separate cross-attention layers. The model is accent-universal, removing explicit source-accent embeddings to rely on phonetic content alone.
Training:
- Pre-training: BART-style corruption (span masking, insertion) on unlabeled native speech to learn token distributions.
- Supervised Fine-Tuning (SFT): Trained on semi-synthetic parallel token pairs (L2 source tokens paired with TTS-synthesized L1 target tokens) using cross-entropy and CTC guidance.
- RL Post-training: A reinforcement learning stage using Group Relative Policy Optimization (GRPO). The decoder is fine-tuned on unpaired real multi-accented speech (GLOBE dataset) using a reward function combining Word Error Rate (WER) and accent classifier confidence. This directly optimizes task-level objectives without requiring additional paired data.

3. Flow-Matching Synthesizer

A non-autoregressive model converts the converted token sequence back to a Mel-spectrogram.

Duration Control: A flow-matching duration predictor assigns durations to tokens. It supports a total-duration-aware mode, conditioning on the average token duration of the source. This allows for "source-length" preservation, critical for applications like voice dubbing, while maintaining natural prosody better than simple scaling.
Output: A pre-trained HiFT vocoder generates the final waveform conditioned on the source speaker embedding.

Key Contributions

Joint VQ Tokenizer: Replaced the conference version's HuBERT/K-Means tokenizer with a jointly trained VQ tokenizer coupled to synthesis and ASR modules, systematically optimizing phonetic representation and reconstruction fidelity.
Accent-Universal Architecture: Redesigned the conversion model with RoPE and a self-attention-only decoder, removing source-accent embeddings to create a more generalizable model.
RL Post-Training: Introduced a GRPO-based RL stage that directly optimizes content preservation (via WER) and accentedness reduction (via accent classifier) using unpaired real data, bridging the gap between SFT behavior and deployment criteria.
Duration-Aware Synthesis: Extended duration control with a flow-matching predictor conditioned on source average duration, enabling high-quality synthesis that respects total duration constraints.
Comprehensive Evaluation: Expanded the scope beyond the conference version with larger datasets, stronger baselines, and detailed ablation studies.

Experimental Results

Experiments were conducted on seven English accents (L2-ARCTIC dataset) comparing TokAN against FramAN (frame-to-frame), CosyAccent (direct flow-matching), and VEVO (prompt-based token conversion).

Intelligibility (WER): TokAN achieved a WER of 9.23% (TokAN-1, duration-free) and 9.40% (TokAN-2, source-length) after RL post-training. This significantly outperformed CosyAccent (12.40%) and FramAN (17.55%).
Accentedness Reduction: TokAN achieved the lowest $\Delta$ PPG (0.2533) and highest L1 probability (99.09%), indicating superior conversion to native-like pronunciation.
Naturalness: TokAN-1 achieved the highest subjective naturalness score (70.73 MUSHRA).
Speaker Similarity: While TokAN showed a trade-off between accent reduction and speaker similarity (consistent with prior findings), it maintained competitive similarity scores. The authors note that the current embedding-conditioned synthesizer may be a bottleneck for speaker identity preservation compared to prompt-based approaches.
Ablation Studies: Confirmed the necessity of each stage: removing RL increased WER to 9.89%; removing SFT caused performance to degrade to near-resynthesis levels; removing pre-training or CTC supervision also significantly hurt performance.

Significance and Claims

The paper claims that TokAN represents a significant advancement in accent normalization by effectively decoupling the need for parallel L1–L2 data while mitigating the quality degradation associated with synthetic supervision. By operating at the token level, the framework abstracts away paralinguistic artifacts, and the joint training of the tokenizer ensures the discrete representations are optimized for the specific downstream tasks of conversion and synthesis.

The integration of GRPO-based RL post-training is highlighted as a novel contribution to the field, allowing the model to refine its output based on task-specific rewards (intelligibility and accent reduction) using unpaired real-world data. Additionally, the total-duration-aware synthesis addresses a critical practical requirement for applications like dubbing, offering a solution that preserves source timing without sacrificing natural prosody.

The authors modestly conclude that while TokAN sets a new state-of-the-art, future work could explore prompt-based synthesizers to better balance accent reduction with speaker identity preservation and extend the modular architecture to non-English languages.