Technical Summary: NE-BERT

Problem Statement

Modern Natural Language Processing (NLP) systems exhibit a significant performance disparity between high-resource and low-resource languages, reinforcing digital inequities. While general multilingual models like mBERT and regional models like IndicBERT-V2 offer broad coverage, they fail to adequately serve the indigenous languages of Northeast India. This region, home to over 200 distinct languages, faces unique challenges including extreme resource scarcity (some languages have fewer than 1,000 digitized sentences), agglutinative morphological structures, and script diversity (Latin and Bengali-Assamese). Existing models often suffer from "vocabulary fragmentation" in these contexts, where rare words are broken into suboptimal subword units, severely degrading inference efficiency and semantic coherence.

Methodology

The author introduces NE-BERT, a domain-specific multilingual encoder model based on the ModernBERT architecture, specifically designed for nine Northeast Indian languages and two anchor languages (Hindi and English).

1. Dataset Construction

The training corpus comprises approximately 8.3 million sentences covering:

• 9 Northeast Indian Languages: Assamese, Garo, Khasi, Meitei, Mizo, Naga, Nyishi, Pnar, and Kokborok.
• 2 Anchor Languages: Hindi and English.
• Sources: Data was curated from government documents, news archives, educational materials, and cultural texts. Specific parallel corpora for Nyishi and Kokborok were sourced from the WMT 2025 Shared Task.

2. Weighted Sampling Strategy

To address extreme data imbalance (ranging from 1,002 sentences for Pnar to 3.4 million for Hindi), the author employed aggressive weighted sampling during tokenizer training:

• Ultra-low-resource languages (e.g., Pnar, Kokborok) received a 100× upsampling weight to ensure adequate vocabulary representation and prevent character-level fragmentation.
• Anchor languages were downweighted (Hindi 0.05×, English 0.2×) to prioritize Northeast language vocabulary while maintaining cross-lingual transfer capabilities.
• Note: These virtual counts applied only to tokenizer training; the actual Masked Language Modeling (MLM) training used raw sentence counts to prevent overfitting.

3. Tokenization

The paper utilizes a custom SentencePiece Unigram tokenizer (50,368 tokens) rather than the more common Byte-Pair Encoding (BPE).

• Rationale: Unigram's probabilistic approach better preserves the linguistic structures of agglutinative languages compared to BPE's greedy merging strategy.
• Configuration: The tokenizer was trained on the weighted virtual counts to ensure common words in low-resource languages form single tokens, reducing sequence length and improving semantic coherence.

4. Model Architecture and Training

• Base: ModernBERT-base (Warner et al., 2025) with 149M parameters (22 layers, 768 hidden dimension, 12 attention heads).
• Features: Rotary Position Embeddings (RoPE), Flash Attention 2, and unpadding for throughput improvement.
• Training: Trained using MLM with 15% masking probability and dynamic masking over 10 epochs.
• Efficiency: The model was trained on a single NVIDIA A40 GPU (48GB VRAM) for ~17 hours at a cost of $7.31.

Key Results

Perplexity Performance

NE-BERT was evaluated on held-out test sets (500 sentences per language) against IndicBERT-V2, MuRIL, and mBERT.

• Overall Performance: NE-BERT achieved an average perplexity of 2.21 across the 9 Northeast languages, significantly outperforming IndicBERT-V2 (35.29) and MuRIL (16.88), and surpassing mBERT (2.76).
• Average Improvement: NE-BERT achieved 15.97× and 7.64× lower average perplexity respectively compared to IndicBERT-V2 and MuRIL across the 9 Northeast Indian languages.
• Ultra-Low-Resource Gains: The most dramatic improvements were observed in languages with minimal data. For Pnar (1,002 sentences) and Nyishi (55,870 sentences), NE-BERT reduced perplexity to 2.92 and 4.33 respectively, whereas IndicBERT-V2 exhibited catastrophic failure with perplexity scores of 66.92 and 187.20.
• High-Resource Performance: On languages with extensive Wikipedia coverage (Assamese, Meitei), mBERT remained competitive, but NE-BERT still achieved superior or comparable results.

Tokenization Efficiency

• Fertility: NE-BERT achieved an average tokenization fertility of 1.68 tokens/word for Northeast languages, compared to 2.08 for IndicBERT-V2, 2.14 for MuRIL, and 2.51 for mBERT. This represents a 1.50× improvement over mBERT.
• Bits Per Character (BPC): NE-BERT achieved an average BPC of 0.347, demonstrating superior compression efficiency compared to IndicBERT-V2 (1.497) and MuRIL (1.271).

Downstream Evaluation

In Part-of-Speech (POS) tagging tasks across Khasi, Mizo, and Nagamese:

• NE-BERT achieved an average accuracy of 82.4%, outperforming mBERT (73.3%) by 9.1 percentage points and IndicBERT-V2 (59.2%) by 23.2 percentage points.

Significance and Claims

The paper posits that NE-BERT demonstrates that domain-specific models with appropriate tokenization can effectively serve ultra-low-resource languages with as few as 1,000 training sentences.

• Vocabulary Optimization over Scale: The results challenge the notion that scaling model size alone is sufficient for low-resource performance. The author argues that careful vocabulary optimization (via weighted Unigram tokenization) and targeted training data are more critical than raw parameter count for these specific linguistic contexts.
• Cost-Effectiveness: The successful training of a competitive model on a single GPU for under $10 provides a practical blueprint for developing language models for underrepresented languages worldwide.
• Digital Inclusion: By addressing the "curse of multilinguality" and vocabulary fragmentation, NE-BERT aims to support NLP research and digital inclusion for Northeast Indian communities, which have been largely absent from mainstream NLP research.

The author explicitly releases the model, tokenizer, training corpus, and test sets under a CC-BY-4.0 license to facilitate reproducibility and community-driven applications. They acknowledge limitations, including the encoder-only architecture (unsuitable for generation tasks) and the need for further downstream evaluation across diverse tasks and all nine languages.