Pretraining and Benchmarking Modern Encoders for Latvian

1. Problem Statement

Despite the dominance of decoder-only Large Language Models (LLMs) in generative tasks, encoder-only Transformers remain critical for practical NLP applications such as classification, sequence labeling, retrieval-augmented generation (RAG), and embedding models due to their efficiency and reliability in token-level outputs.

However, low-resource languages like Latvian face significant challenges:

• Underrepresentation: Latvian is poorly represented in multilingual pretraining corpora, leading to diluted capacity and suboptimal representations in models like mBERT and XLM-R.
• Lack of Monolingual Models: Few high-quality, monolingual Latvian encoders exist. Existing ones (e.g., LVBERT, LitLat BERT) are often based on older architectures or smaller datasets.
• Architectural Uncertainty: It is unclear how modern architectural choices (e.g., ModernBERT's efficiency, DeBERTaV3's disentangled attention) translate to low-resource, monolingual settings.
• Fragmented Evaluation: There is no unified benchmark or leaderboard for Latvian encoder models, making systematic comparison difficult.

2. Methodology

2.1 Dataset Construction

The authors constructed a high-quality, large-scale Latvian corpus totaling 6.43 billion words after filtering and deduplication.

• Sources: A mixture of web-scale data (FineWeb2, HPLT-v2 cleaned) and curated resources (Latvian National Corpus Collection, news, legal texts, academic papers, Wikipedia, and books).
• Preprocessing:
  • Strict deduplication (exact and near-duplicate removal using MinHash LSH).
  • Quality filtering using a 5-gram KenLM language model trained on the Balanced Corpus of Modern Latvian to discard high-perplexity (noisy/non-fluent) text.
  • Context Extension: A specific subset of 2.5B tokens was curated to support long-context training, prioritizing documents longer than 1,024 tokens (up to 8,192 tokens).

2.2 Tokenization

• Utilized the HPLTv2 Latvian tokenizer (GPT-2 style byte-level WordPiece) with a vocabulary size of 32,768.

2.3 Model Architectures

Three distinct architectural families were pretrained on 100 billion tokens using a consistent recipe (Effective batch size: 4.2M tokens; Hardware: 8x NVIDIA GPUs):

1. RoBERTa-based: Follows the standard HuggingFace recipe with 30% span-based masking.
2. DeBERTaV3-based: Follows the CamemBERTaV2 recipe using Disentangled Attention and Replaced Token Detection (RTD) objectives.
3. ModernBERT-based: Follows the ModernBERT recipe with three-stage training (stable, context extension, cooldown). Features include:
   • FlashAttention2 and RoPE positional embeddings.
   • Token-level unpadding for >99% packing efficiency.
   • Variants: mini (59M), base (136M), and large (377M parameters).
   • Long-context support up to 8,192 tokens.

2.4 Evaluation Framework

The authors introduced a unified benchmark suite covering three regimes:

1. Lightweight Diagnostics (EuroEval-style): Minimal-data fine-tuning on tasks like Sentiment (LTEC), Linguistic Acceptability (ScaLA), NER (FSNER), Reading Comprehension (WikiQA), and Commonsense Reasoning (COPA).
2. Universal Dependencies (UD): Morphosyntactic modeling including POS tagging, lemmatization, and dependency parsing on the Latvian UD treebank.
3. Word Sense Disambiguation (WSD): A newly constructed, manually annotated Latvian WSD dataset based on Latvian WordNet, treating the task as context-gloss matching.

3. Key Contributions

1. Suite of Pretrained Encoders: Release of multiple Latvian-specific models (RoBERTa, DeBERTaV3, ModernBERT) in various sizes, including long-context variants.
2. Unified Benchmark: Introduction of a comprehensive evaluation suite for Latvian NLP, including a new Word Sense Disambiguation (WSD) dataset.
3. State-of-the-Art Performance: Demonstration that a relatively small monolingual model can outperform significantly larger multilingual baselines.

4. Key Results

4.1 Overall Performance

• Best Model: lv-deberta-base (111M parameters) achieved the strongest overall performance across almost all tasks.
• Efficiency: Despite having ~5x fewer parameters than xlm-roberta-large (560M), lv-deberta-base consistently outperformed it and other multilingual baselines (mDeBERTa, mmBERT).
• Comparison to Prior Work: It substantially outperformed previous Latvian-specific models like hplt-bert-base-lvs and litlat-bert.

4.2 Task-Specific Highlights

• Commonsense Reasoning (COPA): A major breakthrough. While most encoders scored an MCC < 20, lv-deberta-base achieved 52.5, indicating a marked improvement in reasoning capabilities.
• Word Sense Disambiguation (WSD): lv-deberta-base achieved 78.9% sense selection accuracy and 83.6% context-sense matching accuracy, outperforming all baselines.
• Universal Dependencies: lv-deberta-base showed the most consistent gains in structural prediction (dependency parsing metrics like LAS, CLAS), suggesting monolingual pretraining is crucial for syntactic structure.
• Long-Context Models: ModernBERT variants (lv-mbert) offered high throughput and long-context support (8k tokens) but generally trailed lv-deberta-base in raw accuracy, though they performed competitively on extractive QA.

4.3 Comparison with LLMs

• When compared to commercial LLMs (e.g., GPT-5), the specialized encoder lv-deberta-base often outperformed them on specific NLU tasks (ScaLA, FSNER, WikiQA) in zero-shot settings, despite LLMs being orders of magnitude larger.
• LLMs dominated the COPA reasoning task, but encoders closed the gap significantly compared to previous multilingual models.

5. Significance and Conclusion

• Monolingual Advantage: The study confirms that for low-resource languages, monolingual pretraining on high-quality, curated data yields superior representations compared to relying on shared multilingual capacity, even when the monolingual model is much smaller.
• Architectural Insight: While ModernBERT offers efficiency and long-context benefits, DeBERTaV3 architecture proved most effective for the Latvian language in terms of raw performance on diagnostic and linguistic tasks.
• Resource Availability: The authors have open-sourced all models and the evaluation benchmark, addressing the fragmentation in Latvian NLP research and enabling reproducible downstream applications.

Limitations: The models are strictly monolingual, limiting cross-lingual transfer. The training data, while curated, still inherits biases from web/news sources (formal register bias) and lacks sufficient conversational or dialectal variation.