Technical Summary: BERTomelo

Problem Statement

While encoder models have become the state-of-the-art for Natural Language Processing (NLP), the landscape for Portuguese language models remains constrained by legacy architectures. Existing monolingual Portuguese encoders, such as BERTimbau and Albertina, largely adhere to the original BERT architecture, characterized by limited 512-token context windows, standard attention mechanisms, and a lack of modern efficiency optimizations. Although multilingual models offer broad coverage, they often suffer from the "Curse of Multilinguality," where per-language performance degrades compared to dedicated monolingual models. Furthermore, recent architectural breakthroughs in English, such as ModernBERT, which incorporate FlashAttention, Alternating Attention, and Rotary Positional Embeddings (RoPE) to support extended context windows (up to 8,192 tokens), have not been fully realized for Portuguese. There is a critical gap in the availability of a next-generation, monolingual Portuguese encoder that leverages these modern efficiency mechanisms and is trained on a contemporary, high-quality corpus.

Methodology

To address this gap, the authors introduce BERTomelo, a suite of monolingual encoders pre-trained from scratch specifically for Brazilian Portuguese. The methodology encompasses three core components: architecture, data, and training protocol.

Architecture

BERTomelo is built upon the ModernBERT architecture. It is available in two variants:

• Base: 22 layers, 12 attention heads, hidden size of 768, and ~136 million parameters.
• Large: 28 layers, 16 attention heads, hidden size of 1,024, and ~378 million parameters.

Both variants adopt a 1,024-token context window, doubling the capacity of standard legacy models. Key architectural features include:

• Efficiency Mechanisms: Integration of FlashAttention-2, Unpadding, and Sequence Packing to reduce memory bottlenecks and computational waste.
• Attention Strategy: Implementation of Alternating Attention, where two consecutive layers apply local attention (window size 128) followed by a layer with global attention.
• Positional Encoding: Use of Rotary Positional Embeddings (RoPE) with an adjusted base frequency (10,000) to facilitate future context scaling.
• Structural Optimizations: Bias-free linear layers, pre-normalization blocks, and Gaussian Error Gated Linear Unit (GeGLU) activation functions.
• Training Objective: The Next Sentence Prediction (NSP) task is omitted; training relies solely on Masked Language Modeling (MLM) with an increased masking rate of 30%.

Data and Tokenization

The models were pre-trained on ClassiCC-PT, a massive, high-quality corpus comprising 96 million curated Portuguese documents extracted from Common Crawl. This dataset was selected to replace the outdated BrWaC corpus used by previous models.

• Tokenization: The authors utilized a custom fast tokenizer based on the Unigram algorithm (previously used in ModBERTBr), trained on a combination of Wikipedia and BrWaC. The vocabulary size is 32,768 tokens.
• Preprocessing: The corpus was segmented by newlines and filtered based on word length (2–15 characters), stop-word density (minimum 2 per sequence), and sequence length (20–1,024 words).

Training Protocol

Training was conducted on AMD High-Performance Computing clusters (MI300X and MI325X GPUs) using the Hugging Face ecosystem.

• Configuration: Both models were trained for 2 million steps with a global batch size of 2,048.
• Precision: Training utilized bf16 mixed precision to ensure stability.
• Duration: The Base model required approximately 121 hours, and the Large model 170 hours, processing roughly 386 billion tokens (8.67 epochs).

Key Contributions

1. ModernBERT Adaptation: BERTomelo is the first Portuguese encoder to systematically integrate the ModernBERT architecture, bringing advanced efficiency mechanisms (FlashAttention, Alternating Attention) and extended context windows to the Portuguese NLP ecosystem.
2. Extended Context Window: By implementing a 1,024-token context window, BERTomelo addresses the limitations of previous 512-token models, enabling better handling of longer-range dependencies.
3. High-Quality Pre-training: The use of the ClassiCC-PT corpus ensures the model is aligned with contemporary language usage, moving beyond the constraints of older datasets like BrWaC.
4. Open Source Availability: Both Base and Large models, along with source code, are publicly released on Hugging Face and GitHub to foster community research.

Results

The paper evaluates BERTomelo through intrinsic and extrinsic metrics, comparing it against state-of-the-art Portuguese models (BERTimbau, Albertina, ModBERTBr) and multilingual baselines (mBERT, XLM-RoBERTa).

Intrinsic Evaluation

• Tokenization: The tokenizer demonstrated high efficiency with a fertility of 1.506 and an unknown token ratio of only 0.0017% on the training set.
• Training Stability: Loss curves showed smooth, progressive decay. The Large model exhibited transient gradient norm spikes after 1 million steps but maintained overall stability.
• MLM Performance: On the test set, the Base model achieved 79.26% accuracy (loss 0.9495), while the Large model reached 82.28% accuracy (loss 0.7799), indicating robust capture of Portuguese syntactic and semantic patterns.

Extrinsic Evaluation (Downstream Tasks)

Evaluated on ASSIN2 (STS and RTE) and LeNER-Br (NER):

• Semantic Textual Similarity (STS): BERTomelo achieved the lowest Mean Squared Error (MSE) for both variants (0.425 for Base, 0.401 for Large), outperforming all other models including BERTimbau and ModBERTBr.
• Recognizing Textual Entailment (RTE): BERTomelo Base achieved 87.70% accuracy, and Large achieved 89.26%. While BERTimbau Large slightly outperformed BERTomelo Large (90.04%), BERTomelo remained highly competitive.
• Named Entity Recognition (NER): BERTomelo Base secured the highest Micro F1-score (91.37%) and Precision (90.46%). BERTomelo Large achieved the highest Recall (92.75%) and a competitive Micro F1-score (91.05%).

Significance and Claims

The paper claims that BERTomelo successfully bridges the architectural gap in Brazilian Portuguese NLP. By integrating ModernBERT innovations, it offers a more robust and efficient alternative to massive multilingual models and legacy monolingual encoders. The results demonstrate that dedicated monolingual pre-training on contemporary data yields superior performance in tasks like NER and STS compared to multilingual counterparts.

The authors modestly note that while the 1,024-token context window is a core capability, it was not explicitly evaluated in the current benchmarks (which are largely sentence-level). However, they assert that this feature provides a foundation for processing longer-range dependencies. The work validates the necessity of architectural adaptation for specific languages and sets a new baseline for future Portuguese encoder development. Future work is proposed to fully leverage the RoPE implementation to extend the context window to 8,192 tokens and to implement sequence packing for further efficiency gains.