Transcribing Bengali Text with Regional Dialects to IPA using District Guided Tokens

Technical Summary: Transcribing Bengali Text with Regional Dialects to IPA using District Guided Tokens

Problem Statement
Accurate transcription of Bengali text to the International Phonetic Alphabet (IPA) is a complex sequence-to-sequence task, particularly when dealing with regional dialects. The challenge arises from the language's intricate phonology, context-dependent sound changes (such as gemination), and significant variation in pronunciation across different districts of Bangladesh. Unlike standard Bengali, regional dialects often exhibit unique spelling conventions, distinct local vocabulary, and divergent phonetic realizations for the same written words. For instance, the word for "my" (amar) is pronounced differently in Chittagong, Rangpur, Narshindi, and other districts. Existing sequence-to-sequence models often struggle with these variations due to out-of-vocabulary (OOV) words and an inability to capture region-specific phonetic patterns without explicit guidance. Furthermore, standard text-to-IPA approaches often fail to account for the specific linguistic nuances of low-resource regional districts.

Methodology
The authors propose a novel technique called District Guided Tokens (DGT) to address the ambiguity of regional pronunciation in sequence-to-sequence models. The core mechanism involves prepending a specific token representing the input text's district of origin (e.g., <Chittagong>, <Rangpur>) to the input sequence before the Bengali text.

• Token Implementation: These tokens are treated as unique words within the tokenizer's vocabulary to ensure they are encoded as single units. To accommodate these new tokens, the embedding size of the transformer models is increased.
• Model Architecture: The study utilizes three state-of-the-art encoder-decoder transformer models: mT5-base, umT5-base, and ByT5-small.
  • ByT5 is highlighted as particularly effective because it operates directly on byte sequences rather than word or subword tokens. This architecture inherently mitigates the OOV problem, which is prevalent in the dataset due to regional spelling variations.
• Dataset: The research utilizes a dataset derived from Fatema et al., containing 150,000 Bengali sentence pairs with corresponding IPA transcriptions. The data covers six districts: Chittagong, Narail, Narsingdi, Rangpur, Tangail, and Kishoreganj. The data was split into 80% training, 10% validation, and 10% testing.
• Pre-processing: The input text underwent cleaning, including the conversion of English numerals to Bengali numerals and the removal of newline characters to create single-sentence inputs.

Key Contributions

1. Novel Technique: The introduction of District Guided Tokens (DGT) as a plug-and-play framework to explicitly inject regional context into sequence-to-sequence models for text-to-IPA transcription.
2. First-of-its-Kind Approach: This study presents the first attempt to solve the sequence-to-sequence problem of transcribing regional Bengali text to IPA, moving beyond standard language transcription.
3. Comprehensive Evaluation: The authors provide a detailed analysis of model performance across different districts and compare DGT-enabled models against non-DGT baselines.
4. Open Source Release: The authors have open-sourced the fine-tuned model checkpoints (via HuggingFace) and the training/inference notebooks (via GitHub) to facilitate further research and application development.

Experimental Results
The study evaluated the models using Word Error Rate (WER) and Character Error Rate (CER) on a held-out test set.

• Overall Performance: The ByT5-small model equipped with DGT achieved the superior performance, recording a WER of 1.24% and a CER of 0.42%. The umT5-base followed with a WER of 2.39% and CER of 1.01%. The mT5-base performed significantly worse (WER 27.79%, CER 5.63%), attributed to its shared positional bias compared to umT5's unique relative positioning bias.
• Impact of DGT: An ablation study confirmed that DGT significantly improves performance. For example, the ByT5-small model's WER improved from 1.80% (without DGT) to 1.24% (with DGT).
• District-wise Variance: Performance varied by district. Narail was the easiest to transcribe (lowest error rates), while Chittagong proved the most challenging (highest error rates), likely due to the high degree of pronunciation and spelling divergence in that region.
• Model Comparison: The byte-based ByT5 model outperformed word-based models (mT5, umT5), demonstrating the advantage of handling OOV words and spelling variations directly at the byte level.

Significance and Claims
The paper claims that incorporating explicit regional district information via DGT significantly boosts the accuracy of IPA transcription for Bengali, a language with high linguistic diversity. The authors assert that this approach effectively bridges the gap between written text and region-specific spoken sounds, a task previously unaddressed for regional dialects.

The study highlights that the token-free ByT5 architecture is particularly well-suited for this task due to its robustness against OOV words and spelling variations. By releasing their models and code, the authors aim to accelerate the development of NLP applications for regional Bengali districts, facilitating linguistic analysis, speech synthesis, and the preservation of understudied dialects. The authors modestly note that while their current models are early language models, future work could explore the capabilities of Large Language Models (LLMs) and dynamic DGT selection mechanisms.