💬 NLP

5-Dialects-BN: Unmasking the Impact of Transliteration on Bangla Dialectal LLMs

This paper introduces 5-Dialects-BN, the first multi-annotation benchmark aligning Romanized transliteration with dialectal, standard, and English texts alongside subjectivity labels for five major Bangla regional varieties, aiming to bridge the resource gap and enable principled evaluation of dialect-aware Large Language Models.

Original authors: Md Mahir Jawad, Galib Mahmud Jim, Rafid Ahmed, Mir Sazzat Hossain, Md Fahim, Md Farhad Alam Bhuiyan

Published 2026-09-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Md Mahir Jawad, Galib Mahmud Jim, Rafid Ahmed, Mir Sazzat Hossain, Md Fahim, Md Farhad Alam Bhuiyan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Language is not a single, solid block; it is a living landscape that shifts and changes as people move across regions. In the world of artificial intelligence, large language models have become incredibly skilled at understanding and generating text, but they have largely learned from a very specific, formal version of languages. When these digital minds encounter the messy, vibrant reality of how people actually speak in their local neighborhoods, they often stumble. This is particularly true for Bengali, a language spoken by hundreds of millions of people across Bangladesh and India. While the standard, formal version of the language is well-documented, the many regional dialects spoken in cities like Chittagong, Sylhet, and Barisal have been largely invisible to the technology. People in these regions often type in a mix of their local dialect and English letters, a practice known as transliteration, creating a unique digital voice that existing systems struggle to hear.

A team of researchers set out to map this uncharted territory by creating a new resource called 5-DIALECTS-BN. They gathered 6,000 real sentences from social media and online forums, capturing the natural way people speak in five distinct regional dialects: Chittagong, Barisal, Noakhali, Sylhet, and Rangpur. For every sentence, they did not just record the words; they built a complete bridge between the different ways people express the same thought. Each entry includes the original text in the local dialect, a version written in English letters (transliteration), a translation into standard Bengali, a translation into English, and a label indicating whether the sentence expresses a personal opinion or a simple fact. This careful, human-verified work ensures that the data reflects the true nuances of the language, from unique slang words to subtle shifts in pronunciation that change the meaning of a sentence.

The researchers then put this new dataset to the test, asking seven different large language models to perform three specific tasks: translating the dialect into English, deciding if a sentence is an opinion or a fact, and converting the local dialect back into standard Bengali. They tested the models in several ways, including giving them no examples, showing them a few examples to learn from, and using a method where the models were allowed to "think" through the problem step-by-step before answering. They also tried a technique called fine-tuning, where they taught the models using just a small number of examples—160 sentences for each dialect—to see if a little bit of direct instruction could help them understand better.

The results revealed a surprising and critical flaw in how these powerful models handle language. The researchers found that when the input text was written in English letters (transliterated), the models performed significantly worse across the board, regardless of how smart the model was or how many examples they were given. This happened because the process of converting local sounds into English letters often loses crucial information. Different sounds in the local dialect can look identical when written in English letters, creating confusion that the models cannot resolve. It is as if a listener is trying to understand a conversation through a wall that muffles specific frequencies; the words are there, but the distinct qualities that make them clear are gone. Even when the models were given a small amount of direct training to help them, the gap between understanding the native script and the English-letter version remained wide, proving that the loss of information in the translation to English letters is difficult to recover.

However, the study also offered a clear path forward. When the researchers used the fine-tuning method with just 160 examples per dialect, the open-source models improved dramatically. They began to outperform even the most advanced, closed-source models that had never seen this specific data before. This suggests that the biggest barrier to understanding these dialects is not a lack of computing power or intelligence, but simply a lack of specific, aligned data. The models are capable of learning these dialects if they are given the right examples. The study concludes that while current technology struggles with the ambiguity of transliterated text, providing it with high-quality, human-verified examples allows it to bridge the gap. This work lays a foundation for building systems that can truly understand the diverse, rich ways people communicate in their daily lives, moving beyond the formal standard to embrace the full spectrum of the language.

Technical Summary: 5-DIALECTS-BN: Unmasking the Impact of Transliteration on Bangla Dialectal LLMs

Problem Statement
While Large Language Models (LLMs) have advanced significantly in Natural Language Processing (NLP), their capabilities degrade sharply in low-resource and dialectally diverse settings. Bangla, spoken by over 270 million people, exemplifies this gap: existing resources overwhelmingly target Standard (Cholito) Bangla, leaving regional dialects without the benchmarks necessary to develop or evaluate dialect-aware systems. Furthermore, online users in South Asia frequently type dialectal Bangla in Latin script (Romanized/Banglish), yet the impact of this transliteration on modern LLM performance remains under-investigated. The absence of multi-annotation benchmarks makes it difficult to diagnose model failures or evaluate systems in a reproducible manner.

Methodology
The authors introduce 5-DIALECTS-BN, a manually curated benchmark comprising 6,000 verified entries across five major regional Bangla dialects: Chittagong, Barisal, Noakhali, Sylhet, and Rangpur. The dataset is constructed through a four-stage pipeline: data harvesting from public online sources (YouTube, Facebook, Reddit, news), authenticity verification by native speakers, tool-assisted multi-annotation, and inter-annotator agreement validation.

Each entry in the dataset contains five aligned fields:

Native dialectal text in the original Bangla script.
Romanized transliteration (using the Avro convention).
Standard (Cholito) Bangla translation.
English translation.
A binary subjectivity label (Subjective vs. Objective).

The authors benchmarked seven contemporary LLMs (three closed-source: Gemini 3 Flash, GPT-4o-mini, Claude Haiku 4.5; and four open-source: Qwen-3-4B, Gemma-4-4B, Llama-3.1-8B, Mistral-7B) across three core tasks:

Dialect-to-English Machine Translation.
Binary Subjectivity Classification.
Dialect-to-Standard Bangla Normalization.

Evaluation was conducted under four regimes: zero-shot, few-shot, Chain-of-Thought (CoT) prompting, and parameter-efficient fine-tuning (LoRA) using 160 examples per dialect. The study also performed causal analyses on transliteration effects, including round-trip inverse transliteration controls and tokenization regression.

Key Contributions

Dataset Release: 5-DIALECTS-BN is the first multi-annotation Bangla dialect benchmark to align Romanized transliteration as a first-class field alongside native script, Standard Bangla, English, and subjectivity labels. It expands the scale of prior work (e.g., DIALTSA-BN) ten-fold and includes the previously under-resourced Rangpur dialect.
Reliability Metrics: The dataset establishes high inter-annotator reliability, with Cohen's $\kappa$ of 0.78 for subjectivity, 0.74 for Standard Bangla rendering, and a mean ROUGE-L of 0.84 for English translations.
Empirical Benchmarking: The paper provides a comprehensive evaluation of LLMs across native and Romanized inputs, revealing specific failure modes such as "demonstration interference" in Llama-3.1-8B and "format collapse" in Qwen-3-4B.
Causal Analysis of Transliteration: The study isolates the mechanisms behind performance degradation in Romanized text, identifying many-to-one information loss and subword token fragmentation as primary causes.

Results

Supervision Bridges the Resource Divide: Fine-tuning open-source models with LoRA on just 160 examples per dialect significantly outperforms closed-source zero-shot and few-shot baselines. For instance, Mistral-7B achieved 73.6 BLEU on translation, surpassing Gemini 3 Flash's zero-shot score of 46.4 BLEU.
Transliteration Harms Current LLMs: Contrary to earlier findings on smaller models, Romanized transliterated input consistently and severely degrades the performance of current LLMs across all models and regimes. This penalty is driven by irreversible phonemic information loss (e.g., vowel length, aspiration) and subword token fragmentation. Round-trip inverse transliteration controls confirmed that restoring the native script recovers almost none of the lost performance, indicating information destruction during the Romanization process.
Prompting Artifacts: The study found that prompting strategies can introduce confounding interactions. Llama-3.1-8B suffered from "demonstration interference," where few-shot prompting caused it to ignore the source input in favor of fluent but unrelated demonstration-register sentences. Conversely, CoT prompting improved subjectivity classification but degraded translation surface fidelity due to paraphrase drift.
Linguistic Divergence: Error rates mirrored linguistic distance from Standard Bangla. High-lexical-divergence items (e.g., Chittagong non-cognate verbs) remained resistant to adaptation, while phonological shifts (e.g., Barisal clusters) were mastered quickly.

Significance
The paper argues that 5-DIALECTS-BN enables the principled evaluation of LLMs on dialectally diverse Bangla and lays a foundation for low-resource, dialect-aware NLP. A central finding is that dialect-aligned supervision is the critical bottleneck for understanding Bangla dialects; while LoRA fine-tuning can overcome the lack of training data, it cannot reconstruct distinctions destroyed by the Romanization channel. The work highlights that for practical deployment in South Asia, where Romanized input is ubiquitous, current models face a severe "transliteration penalty" that requires addressing through better tokenization or orthographic standards, rather than relying solely on prompting or limited fine-tuning. The authors conclude that while 160 examples are sufficient to outperform zero-shot prompting, they are insufficient to fully recover the semantic distinctions lost in transliteration.

More like this