💬 NLP

DunbaaBERT: From Sacrifice to Semantics

The paper introduces DunbaaBERT, a family of Urdu-specific RoBERTa-base models trained from scratch on a large corpus, demonstrating that carefully curated, compact models with smaller vocabularies can achieve competitive performance and favorable efficiency trade-offs across various Urdu NLP tasks compared to larger multilingual baselines.

Original authors: Iffat Maab, Waleed Jamil, Raphael Schmitt

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Iffat Maab, Waleed Jamil, Raphael Schmitt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Building a Specialist vs. a Generalist

Imagine you are trying to teach a robot to understand the Urdu language. Most big tech companies build "Generalist" robots (like mBERT or XLM-R). These robots are like polyglot students who have studied 100 different languages at once. They are very smart, but because they have to split their brainpower across so many languages, they aren't always the deepest experts in any single one, and they are heavy, slow, and expensive to run.

The authors of this paper asked: What if we built a "Specialist" robot that only studies Urdu?

They created DunbaaBERT, a family of three models designed specifically for Urdu. They didn't just copy-paste an existing model; they trained them from scratch using a massive, carefully cleaned library of Urdu text.

The Experiment: The "Vocabulary Size" Test

To see how to build the best specialist, the researchers ran an experiment with three different versions of DunbaaBERT. Think of these versions as three students with different-sized dictionaries:

DunbaaBERT-32k: Has a compact dictionary of 32,000 words.
DunbaaBERT-52k: Has a medium dictionary of 52,000 words.
DunbaaBERT-96k: Has a massive dictionary of 96,000 words.

The researchers wanted to know: Does having a bigger dictionary automatically make the student smarter and faster?

The Ingredients: Cleaning the Library

Before training these models, the team had to gather the "textbooks." They didn't just grab random text from the internet, which is often messy (like a library with torn pages, ads, and gibberish).

The Core: They used high-quality web data and Wikipedia.
The Filter: They had a special "bouncer" (an automated filter) for the messier data. This bouncer checked if a sentence looked like real Urdu or if it was just a broken web link or a list of numbers. If it looked suspicious, the bouncer kicked it out.
The Result: They started with about 22 GB of text but threw away the "junk" to end up with a pristine 17 GB of high-quality Urdu.

The Results: Bigger Isn't Always Better

After training the three models, they put them through a series of tests (like a final exam) covering grammar, news classification, detecting offensive language, and understanding feelings (sentiment).

Here is what they found, which turned out to be a bit surprising:

1. The "Goldilocks" Vocabulary
The model with the medium-sized dictionary (52k) was actually the best at understanding complex grammar rules. It was like the student who had studied just enough to understand the nuance of the language without getting confused by too many rare words.

2. The Efficiency Champion
The model with the smallest dictionary (32k) was the most efficient. It was the "sprinter." It didn't just perform well; it did so while using the least amount of computer power and time.

The Analogy: Imagine a race. The 96k model (big dictionary) was strong but heavy, like a bodybuilder. The 32k model was a lightweight runner. Surprisingly, the lightweight runner often finished the race just as fast, or even faster, because it wasn't weighed down by carrying a massive dictionary it didn't fully need.

3. The "Generalist" Comparison
When they compared their Urdu specialists to the big "Generalist" models (like XLM-R), the specialists won on efficiency. The Generalists could sometimes get slightly higher scores on specific tasks, but they required much more computing power to do it. The DunbaaBERT models were like a local expert who knows the neighborhood better than a tourist with a giant map, and they get there faster.

The Key Takeaway

The paper concludes that for a language as rich and complex as Urdu, you don't need the biggest, heaviest model to get the best results.

Quality over Quantity: A carefully curated, clean dataset is more important than just dumping more data in.
Sweet Spot: A medium vocabulary size (52k) offered the best balance for understanding grammar, while the smallest vocabulary (32k) offered the best balance for speed and cost.
Specialists Win: A model trained specifically for Urdu can compete with (and often beat) massive models trained on 100+ languages, especially when you care about how fast and cheap it is to run.

In short, the authors showed that with the right recipe (clean data) and the right portion size (vocabulary), you can build a highly effective Urdu AI without needing a supercomputer the size of a house.

Technical Summary: DunbaaBERT: From Sacrifice to Semantics

Problem Statement

Despite the rapid advancement of Large Language Models (LLMs) in zero-shot and few-shot learning, Urdu remains a significantly under-resourced language in the NLP ecosystem. While spoken by over 100 million people, Urdu faces challenges due to limited high-quality training data, fragmented evaluation settings, and a lack of systematically evaluated, open-source encoder-only models. Existing resources often vary in script (Standard Urdu vs. Roman Urdu), domain, and annotation schemes, making fair comparisons difficult. Furthermore, prior Urdu-specific models available on platforms like Hugging Face often suffer from poor documentation, unclear training configurations, or potential under-training, while massive multilingual models (e.g., mBERT, XLM-R) distribute their capacity across many languages, potentially diluting performance for specific linguistic structures like Urdu's rich morphology.

Methodology

Corpus Construction

The authors constructed a dedicated 17 GB Urdu pre-training corpus by integrating multiple sources:

Core Web Data: Large-scale web-derived text from mC4 and CulturaX (OSCAR subsets), providing broad topical and stylistic diversity.
Wikipedia: Urdu Wikipedia articles, used both as a clean reference corpus and a source of high-quality text.
Auxiliary Data: Filtered Urdu text extracted from NLLB-derived parallel data (OPUS).
Filtering Strategy: To handle noise in the auxiliary NLLB data without manual annotation, the authors employed an unsupervised filtering pipeline. This involved training a fastText skip-gram model on clean Urdu Wikipedia lines to create a reference centroid. Candidate lines were scored based on cosine similarity to this centroid minus a heuristic penalty term ( $P(x)$ ) accounting for surface-level noise (e.g., short length, high Latin character ratio, boilerplate patterns). This process retained ~92.6% of the auxiliary data while removing clear outliers.
Deduplication: The final corpus underwent exact line-level deduplication, reducing the size from ~22.3 GB to 17 GB.

Model Architecture and Training

The study introduces DunbaaBERT, a family of RoBERTa-base models trained from scratch.

Architecture: Standard RoBERTa-base configuration with dynamic masking and Whole Word Masking (WWM).
Vocabulary Ablation: Three variants were trained with distinct Byte-BPE vocabulary sizes: 32k, 52k, and 96k tokens.
Training Setup: Models were trained for 100k update steps with a sequence length of 512 tokens. Training utilized mixed precision (FP16) on NVIDIA H100 GPUs. All hyperparameters (learning rate warmup, decay, batch size) were kept identical across variants to isolate the effects of vocabulary size.

Evaluation Framework

The models were evaluated across intrinsic and downstream benchmarks:

Intrinsic Evaluation: URBLIMP, a benchmark of 5,696 minimal pairs probing 19 fine-grained linguistic phenomena (e.g., agreement, word order, argument structure).
Downstream Tasks:
- News Classification (COUNT19): Multi-class classification of news articles into seven domains.
- Offensive Language Detection (USADC): Binary classification of abusive vs. normal text in native Urdu script.
- Sentiment Analysis: Evaluated on two datasets: PSL–Kabaddi (Twitter sentiment regarding sports) and IMDB Urdu (document-level movie reviews).
Metrics: Accuracy, Macro-F1, and a novel Normalized Efficiency (Norm. Eff.) metric, defined as $(Macro\text{-}F1/100) \times (SPS / \max(SPS))$ , which balances predictive performance with inference throughput.

Key Contributions

DunbaaBERT Models: The release of three state-of-the-art, monolingual Urdu encoder models (32k, 52k, 96k vocabularies) trained from scratch on a curated 17 GB corpus, released under the MIT license.
Systematic Vocabulary Analysis: A comprehensive investigation into how vocabulary size affects Urdu NLP, challenging the assumption that larger vocabularies always yield better downstream performance.
Efficiency Benchmarking: The introduction of normalized efficiency metrics to evaluate the trade-off between model performance and computational cost, demonstrating that smaller, specialized models can outperform larger multilingual baselines in cost-effectiveness.
Curated Corpus: A detailed methodology for constructing a high-quality, deduplicated Urdu corpus using unsupervised filtering techniques, addressing the scarcity of clean training data.

Results

Linguistic Acceptability (URBLIMP)

DunbaaBERT variants significantly outperformed multilingual baselines (mBERT, XLM-R, mmBERT).
The 52k vocabulary variant achieved the highest overall score (97.0), followed by 32k (95.1) and 96k (94.6).
Notably, the 96k model, despite achieving the lowest pre-training perplexity, did not translate this into superior downstream linguistic competence. The 52k model showed the most balanced performance across complex grammatical categories.

Downstream Tasks

Performance: DunbaaBERT variants achieved competitive or superior results compared to strong multilingual baselines. For instance, on the USADC offensive language detection task, DunbaaBERT32k achieved a Macro-F1 of 94.08, outperforming multilingual models and previously reported GPT-4-based results.
Efficiency: The 32k variant consistently provided the strongest Normalized Efficiency across most benchmarks (e.g., 0.944 on COUNT19, 0.897 on IMDB Urdu).
Vocabulary Impact: Larger vocabularies (96k) did not consistently improve downstream effectiveness. In several cases, the 32k model offered the best trade-off between accuracy and inference speed.

Efficiency vs. Search Cost

Analysis of hyperparameter search costs revealed that DunbaaBERT models, particularly the 32k and 52k variants, occupy favorable regions in the performance-efficiency-cost space. They require less total search time to find optimal configurations compared to larger multilingual models like XLM-Rlarge or HPLT-BERTur.

Significance and Claims

The paper claims that carefully curated, Urdu-specific encoder models can remain highly competitive for practical NLP applications, even with comparatively compact model and training scales. The authors argue that vocabulary scaling alone is not sufficient to guarantee improved downstream effectiveness; instead, a balanced vocabulary size (around 32k–52k) combined with high-quality corpus curation offers the optimal trade-off for Urdu.

The study highlights that for morphologically rich languages like Urdu, the "sacrifice" of using a smaller, specialized vocabulary often yields better semantic and syntactic generalization than simply increasing token counts. The authors position DunbaaBERT as a robust, efficient alternative to massive multilingual models, particularly in resource-constrained deployment scenarios. They acknowledge limitations regarding the focus on classification benchmarks and the lack of dialectal variation in the corpus, but emphasize that their work provides a reproducible foundation for future Urdu NLP research.