A Comprehensive Analysis of Tokenization and Self-Supervised Learning in End-to-End Automatic Speech Recognition applied on French Language

Technical Summary: A Comprehensive Analysis of Tokenization and Self-Supervised Learning in End-to-End Automatic Speech Recognition applied on French Language

Problem Statement
While end-to-end Automatic Speech Recognition (ASR) systems are increasingly integrated into applications like transcription and voice assistants, their performance evaluation remains limited. Current research predominantly relies on Word Error Rate (WER) and Character Error Rate (CER). However, these metrics are often insufficient to describe the quality of transcripts for downstream applications. Furthermore, the specific impacts of subword tokenization algorithms and Self-Supervised Learning (SSL) model choices on lexical, acoustic, and semantic performance—particularly for the French language—remain under-explored in the context of end-to-end architectures.

Methodology
The authors conducted a qualitative study on French ASR using a comprehensive set of evaluation metrics beyond standard error rates.

• System Architecture: The study utilized 28 end-to-end ASR systems built on the Speechbrain toolkit. Each system consists of an SSL model encoder followed by a Deep Neural Network (DNN) with three linear layers and a Connectionist Temporal Classification (CTC) layer for decoding.
• Tokenization Strategies: The authors compared various tokenization methods, including Character, Byte-Pair Encoding (BPE), SentencePiece, and Unigram. They varied vocabulary sizes (ranging from 150 to 1000 tokens) and investigated the use of grapheme-based initialization (144 graphemes) to respect linguistic and acoustic characteristics.
• Self-Supervised Learning (SSL) Models: The study employed multiple wav2vec 2.0-based models:
  • Monolingual French models trained on 1,000, 3,000, and 7,000 hours of data.
  • A multilingual model (XLSR) trained on 53 languages including French.
  • An English-only model trained on 53,000 hours to test cross-lingual transfer.
• Corpora: Systems were trained on approximately 356 hours of French broadcast data (ESTER 1 & 2, EPAC, ETAPE, REPERE) and evaluated on the 10-hour REPERE test corpus.
• Evaluation Metrics: In addition to WER and CER, the study introduced or utilized:
  • Unseen Word Error Rate (UWER): Measures accuracy on words absent from the training corpus.
  • SemDist: Semantic distance based on sentence embeddings (SentenceBERT/CamemBERT) to assess semantic similarity.
  • Phoneme Error Rate (PhonER): Acoustic metric measuring the Levenshtein distance between reference and hypothesis phoneme sequences.

Key Results

• Impact of SSL Models:
  • Language Specificity: SSL models pre-trained on French data significantly outperformed those trained on English or multilingual data. The French model trained on 7,000 hours achieved a WER of 16.56%, whereas the English model (53k hours) resulted in a WER of 36.41%.
  • Data Size: A direct correlation was observed between the size of the French training data and performance improvements across all metrics.
  • Multilingual Transfer: While the multilingual XLSR model (WER 21.48%) outperformed the English-only model, it underperformed compared to the monolingual French model, suggesting that language-specific information may be diluted in multilingual training.
• Impact of Tokenization:
  • Subword vs. Character: Subword tokenizers (BPE, SentencePiece, Unigram) consistently outperformed character tokenization across all metrics, capturing more nuanced linguistic and contextual information.
  • Optimal Configuration: The Unigram tokenizer with a reduced vocabulary of 150 tokens consistently achieved the best results across WER, CER, SemDist, UWER, and PhonER.
  • Grapheme Initialization: Contrary to the hypothesis that grapheme-based tokenization would improve acoustic alignment, systems using grapheme-initialized BPE yielded inferior results compared to standard subword tokenizers. The best-performing system actually had the lowest percentage of graphemes in its vocabulary.
• Metric Discrepancy:
  • The study found that different metrics often yield conflicting rankings for the "best" system. For instance, a system optimal by WER might not be optimal by CER or SemDist.
  • Spearman correlation analysis revealed low to moderate correlations between metrics (e.g., WER and SemDist correlation of 0.87, but lower correlations with UWER), indicating that no single metric fully captures system performance.

Significance and Contributions
The paper makes four primary contributions to the field of French ASR:

1. Generalization via Vocabulary Reduction: It establishes that reduced vocabulary sizes enhance the generalization capabilities of ASR systems, particularly for unseen words (UWER).
2. Unigram Efficacy: It provides evidence that the Unigram tokenizer with a reduced vocabulary is highly effective for French ASR, outperforming other strategies including grapheme-based approaches.
3. Metric Limitations: It demonstrates that evaluation criteria significantly impact system-level assessments. A system deemed optimal by WER is not necessarily the best from a human, semantic, or phonetic perspective.
4. Grapheme Ineffectiveness: It suggests that phonetically adapted tokens (graphemes) do not yield performance improvements over traditional tokenization methods in this end-to-end context.

The authors conclude that the inconsistency among evaluation metrics necessitates the exploration of alternative approaches for both intrinsic and downstream evaluations, tailored to specific tasks, rather than relying solely on traditional error rates.