moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERT

Technical Summary: moBERTo

Problem Statement
While encoder-only transformer models remain essential for production NLP pipelines due to their favorable trade-off between performance, latency, and computational cost, the Portuguese NLP landscape lacks a modern, efficient encoder. The dominant monolingual model, BERTimbau, relies on the original BERT architecture, inheriting limitations such as a 512-token context window and absolute positional embeddings. Although multilingual models exist, they typically underperform dedicated monolingual models on language-specific benchmarks. Furthermore, recent architectural advancements in English encoders—such as Rotary Positional Embeddings (RoPE), alternating local-global attention, Flash Attention, and sequence unpadding introduced by ModernBERT—have not been effectively adapted for Portuguese.

Methodology
The authors introduce moBERTo, a Portuguese adaptation of the ModernBERT-base architecture. The approach focuses on continued pretraining rather than training from scratch to leverage existing representations.

• Data Curation: The model is trained on a curated corpus of approximately 12 billion tokens (5 epochs over 60 billion total tokens) extracted from FineWeb2. The data was filtered using educational and STEM classifiers from ClassiCC-PT to ensure high-quality content.
• Architecture Preservation: The original ModernBERT-base architecture (149M parameters) is preserved, including RoPE, alternating local-global attention, Flash Attention, and unpadding. The RoPE frequency parameters (160k global, 10k local) are kept at their original values.
• Training Strategy:
  • Phase 1 (Continued Pretraining): The model undergoes continued pretraining on the 60B-token Portuguese corpus at a sequence length of 1,024 tokens using Masked Language Modeling (MLM) with a 30% masking rate.
  • Phase 2 (Long-Context Post-training): An optional phase involves an additional 10 billion tokens at a sequence length of 8,192 tokens to explicitly enhance long-context capabilities.
• Adaptation Variants: Several model variants were trained to isolate specific factors:
  • Tokenizer Adaptation: Some variants use a Portuguese-specific tokenizer. To address the misalignment this causes with the original embedding space, a Subword-Matching (SWM) transfer approach is employed, where new token embeddings are reconstructed by combining the original embeddings of their constituent subwords.
  • Initialization: Ablation studies compare continued pretraining against training from scratch (random initialization).

Key Contributions

1. moBERTo Release: The first Portuguese adaptation of the ModernBERT architecture, bringing modern encoder design (RoPE, alternating attention, etc.) to the Portuguese NLP ecosystem.
2. Comprehensive Evaluation: A rigorous evaluation across five task families: Information Retrieval (IR), Long-Context Retrieval, Document Classification, Named Entity Recognition (NER), and Natural Language Understanding (NLU).
3. Ablation Studies: Systematic analysis of the impact of continued pretraining vs. scratch training, tokenizer adaptation, SWM embedding transfer, and long-context post-training.
4. Open Resources: Public release of model weights and the curated pretraining dataset on Hugging Face.

Results

• Information Retrieval: The best variant, moBERTo-SWM-8k (combining Portuguese tokenizer, SWM, and long-context post-training), achieved the highest average reranking nDCG@10 (0.5255) across three Portuguese benchmarks (QUATI, mMARCO-PT, Robust04-PT), outperforming the previous state-of-the-art BERTimbau (0.4671).
• Long-Context Retrieval: Remarkably, moBERTo (orig. tok.) achieved the strongest results on the MLDR benchmark at 8,192 tokens (0.6166) despite being trained exclusively at 1,024 tokens. This suggests that long-context capabilities from the original English pretraining transfer effectively without dedicated long-context training in Portuguese. However, tokenizer adaptation (without SWM) significantly degraded long-context performance.
• Token-Level Tasks (NER & NLU): Variants using the Portuguese tokenizer with SWM transfer (moBERTo-SWM and moBERTo-SWM-8k) consistently outperformed original-tokenizer variants on NER (LeNER-Br) and NLU (PLUE-PT), indicating that language-specific tokenization benefits these tasks.
• Classification: Performance gains were marginal across classification tasks, with many models reaching saturation.
• Language Retention: Continued pretraining on Portuguese resulted in a degradation of English performance (GLUE benchmark), with SWM variants showing the largest decrease due to the new tokenizer.

Significance and Claims
The paper claims that moBERTo successfully bridges the gap between modern encoder architectures and Portuguese NLP. The authors emphasize four key findings derived from their ablation studies:

1. Continued pretraining is superior to training from scratch, particularly for preserving long-context capabilities inherited from the base model.
2. Tokenizer adaptation improves token-level tasks (NER, NLU) but degrades long-context retrieval due to positional misalignment.
3. Subword-Matching (SWM) embedding transfer effectively mitigates the long-context degradation caused by tokenizer changes while improving reranking and NER performance.
4. Encoder-only architectures remain competitive with larger decoder-only alternatives for discriminative tasks, offering a more efficient solution for production pipelines.

The work demonstrates that modernizing the encoder paradigm for Portuguese is feasible and beneficial, providing a new baseline for future research and applications in the region.