Benchmarking BERT-based Models for Sentence-level Topic Classification in Nepali Language

This study benchmarks ten BERT-based models for Nepali sentence-level topic classification, demonstrating that the Indic-specific MuRIL-large model achieves the highest performance with a 90.60% F1-score, thereby establishing a robust baseline for future Nepali NLP applications.

The Gist

Imagine you are trying to teach a group of robots how to understand the Nepali language. Specifically, you want them to read a sentence and instantly guess what it's about: Is it about farming? Health? School? Culture? Or just general chat?

This paper is like a report card for ten different "robot brains" (AI models) to see which one is the best at this specific task.

Here is the story of their race, explained simply:

1. The Problem: The "Empty Library"

Most AI models are like students who have read millions of books in English, French, or Spanish. They are super smart. But for languages like Nepali, the library is very empty. There aren't enough books (data) to teach them properly. This is called a "low-resource" language.

The researchers wanted to see: If we don't have enough Nepali books, can we teach these robots using books from related languages (like Hindi or other Indian languages) instead?

2. The Contestants: Ten Different "Students"

The researchers lined up ten different AI models to take the test. Think of them as students with different backgrounds:

• The Global Travelers (Multilingual): These models studied many languages at once (like mBERT and XLM-R). They are well-traveled but maybe not experts in any single one.
• The Regional Specialists (Indic Models): These models studied a group of related Indian languages together (like MuRIL and IndicBERT). Since Nepali is related to Hindi and other Indian languages, these students had a head start.
• The Local Experts (Monolingual): These models studied only Nepali (like NepBERTa). They are the locals who know the slang and culture best.
• The English Native: One model that only knows English (RoBERTa) was also tested, just to see how it would do without any Nepali training.

3. The Exam: The "Topic Test"

The researchers created a fair test with 25,000 sentences divided into five categories:

1. 🌾 Agriculture (Farming)
2. 🏥 Health (Medicine)
3. 🎓 Education & Tech
4. 🏔️ Culture & Tourism
5. 💬 General Chat

They asked each robot to read a sentence and pick the right category.

4. The Results: Who Won?

Here is the surprising outcome:

• 🏆 The Champion: MuRIL-large
  The "Regional Specialist" named MuRIL-large won the race! It got about 90.6% of the answers right.

  • Why? It's like a student who studied a whole group of related languages (Indic languages). Because Nepali is a cousin to Hindi and other Indian languages, this model understood the "family resemblance" perfectly. It knew the grammar and words better than the others.

• 🥈 The Runner-Up: NepBERTa
  The "Local Expert" (NepBERTa) came in second with 88.3%.

  • The Twist: Even though it only studied Nepali, it did almost as well as the giant regional model. Plus, it was faster and cheaper to run. It's like a local guide who knows the city streets better than a tourist, even if the tourist has a bigger map.

• 🥉 The Others:
  The "Global Travelers" (like XLM-R) did pretty well, but the "English Native" (RoBERTa) struggled the most, getting only about 83% right. It's hard to guess the topic of a Nepali sentence if you've never heard the language before!

5. The Tricky Part: "General Chat"

The researchers noticed something interesting. All the robots were great at guessing topics like "Farming" or "Health" because those words are very specific. But they got confused by "General Communication" (like stories or arts).

• Analogy: It's easy to guess a sentence is about "Apples" if you see the word "Apple." But if the sentence is just "It was a nice day," it's hard to guess if it's about culture, tourism, or just chatting. This category was the hardest for everyone.

6. The Takeaway: What Does This Mean?

This study is a big step forward for technology in Nepal. It tells us:

1. Don't ignore your neighbors: Using models trained on related languages (like MuRIL) works better than just using generic global models.
2. Local knowledge is gold: Even a model trained only on Nepali (NepBERTa) is very powerful and efficient.
3. The foundation is built: Now that we know which models work best, we can build bigger, smarter tools for Nepal—like automatic news summarizers, better search engines, or chatbots that actually understand Nepali people.

In short: The researchers found that to teach a robot Nepali, it helps to teach it the "family languages" first, but a dedicated local teacher is also a very strong contender. The future of Nepali AI looks bright!

Technical Summary

1. Problem Statement

Natural Language Processing (NLP) has advanced significantly with Transformer-based models like BERT, yet low-resource languages such as Nepali remain underexplored. Despite being an Indo-Aryan language with a rich morphological structure and significant cultural importance, Nepali NLP faces challenges due to:

• A lack of large, annotated corpora.
• Limited computational resources.
• Insufficient comparative studies on how different pre-trained model families (Multilingual, Indic, Hindi-specific, and Monolingual Nepali) perform on Nepali text.

The specific problem addressed is the lack of a systematic benchmark evaluating diverse BERT variants for sentence-level topic classification in Nepali, which is necessary to establish robust baselines for future document-level applications.

2. Methodology

Dataset

• Source: Information and Language Processing Research Lab (ILPRL), Kathmandu University.
• Size: 25,006 sentences in the Devanagari script.
• Structure: A balanced dataset distributed across five conceptual domains:
  1. Agriculture (D1A)
  2. Health (D2H)
  3. Education & Technology (D3E)
  4. Culture & Tourism (D4C)
  5. General Communication (D5G)
• Preprocessing: Sentences were filtered and deduplicated to remove noise.

Models Evaluated

The study benchmarked 10 pre-trained BERT-based models categorized into four groups to analyze the impact of pre-training corpora and language coverage:

1. Multilingual: mBERT, XLM-RoBERTa (XLM-R), mDeBERTa.
2. Indic (Region-specific): MuRIL (base & large), DevBERT, IndicBERT.
3. Hindi-specific: HindiBERT.
4. Nepali-specific (Monolingual): NepBERTa.
5. English Baseline: RoBERTa.

Experimental Setup

• Data Split: Training (20,005), Validation (2,500), Testing (2,501).
• Fine-tuning: All models were fine-tuned using the Hugging Face library.
• Hyperparameters: 10 epochs, learning rate of $2 \times 10^{-5}$, batch size of 8, gradient accumulation of 2, and max sequence length of 256.
• Hardware: Intel Xeon Platinum 8255C CPU and GeForce RTX 4060 Ti (16 GB) GPU.
• Metrics: Accuracy, Weighted Precision, Weighted Recall, Weighted F1-Score, and AUROC.

3. Key Contributions

• Comprehensive Benchmarking: This is one of the first studies to systematically compare Multilingual, Indic, Hindi, and Monolingual Nepali models on a single, balanced Nepali corpus.
• Granular Evaluation: Unlike previous works that focused on broad categories or non-transformer models, this study provides sentence-level classification results across five distinct domains.
• Baseline Establishment: The paper establishes a robust performance baseline for Nepali NLP, specifically for sentence-level classification, which serves as a foundation for future document-level tasks.
• Efficiency Analysis: The study evaluates not just accuracy but also the trade-off between model performance, parameter size, and training time.

4. Results

Overall Performance

• Top Performer: MuRIL-large achieved the highest performance across all metrics, with a Weighted F1-Score of 90.60%. It outperformed both multilingual and monolingual models.
• Strong Contender: NepBERTa (monolingual Nepali) performed competitively with an F1-score of 88.26%. Notably, it achieved this with significantly lower computational cost (34.78 minutes training time, 110M parameters) compared to MuRIL-large (65.75 minutes, 304M parameters).
• Multilingual Models: XLM-R performed well (89.67% F1), while mBERT and mDeBERTa scored slightly lower (87.78% and 86.92%, respectively).
• Lowest Performer: IndicBERT had the lowest F1-score (80.66%), and the English-only RoBERTa lagged significantly (83.83%), highlighting the necessity of language-specific or regionally aligned pre-training.

Category-wise Analysis

• Models consistently performed better on Agriculture, Health, Education, and Culture categories.
• General Communication was the most difficult category to classify (lowest F1-scores across all models). The authors attribute this to the category being derived from multiple sub-categories (Arts, Literature, Stories), leading to higher semantic ambiguity.
• MuRIL-large maintained the lead in every category, with F1-scores ranging from 84.37% (General Comm) to 94.86% (Culture).

Key Insight

The results suggest that while monolingual pre-training (NepBERTa) is highly effective when sufficient data exists, Indic models (specifically MuRIL) that leverage shared linguistic and scriptural features across South Asian languages (including Hindi and other Indic languages) currently offer superior representation for Nepali text.

5. Significance and Future Directions

• Validation of Regional Pre-training: The study confirms that models pre-trained on linguistically related Indic data (MuRIL, DevBERT) or specific monolingual data (NepBERTa) significantly outperform generic multilingual models for low-resource languages.
• Practical Application: The established baselines allow researchers to select optimal models based on their constraints (e.g., choosing NepBERTa for speed/efficiency or MuRIL-large for maximum accuracy).
• Future Work:
  • Extending the dataset to include more categories and diverse text styles.
  • Scaling from sentence-level to document-level classification to capture broader context.
  • Conducting detailed error analysis and exploring ensemble learning techniques to further boost accuracy.
  • Investigating Named Entity Recognition (NER) and Sentiment Analysis using these models.

Limitations: The study is confined to text classification and may not generalize directly to other tasks like NER or QA. Additionally, the dataset, while balanced, covers limited domains, and the internal training corpora of some models (like mDeBERTa) are not fully transparent.