NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpus

1. Problem Statement

Despite the advancements in Natural Language Processing (NLP) for Portuguese, the field faces two critical bottlenecks compared to English:

• Data Scarcity: Existing monolingual Portuguese corpora (e.g., brWaC with ~2.68B tokens, Aroeira with ~15B tokens) are significantly smaller than those used for English models, limiting the scale and performance of pre-trained models.
• Architectural Lag: Most Portuguese models rely on older BERT architectures. While modern architectures like ModernBERT (featuring Rotary Positional Embeddings, long-context support, and efficient attention mechanisms) have shown superior efficiency and accuracy in English, they have not been fully adopted or trained on massive, high-quality Portuguese datasets.
• LLM Limitations: While Large Language Models (LLMs) offer generative capabilities, they suffer from high computational costs, latency, privacy concerns, and hallucination risks. There is a need for "right-sized," efficient encoder-only models for discriminative tasks (classification, retrieval, entailment) that do not require the overhead of generative LLMs.

2. Methodology

A. Corpus Construction: Aurora-PT

The authors created Aurora-PT, a new monolingual Brazilian Portuguese corpus, which is currently the largest openly available resource of its kind.

• Scale: Contains 331 billion tokens (using GPT-2 tokenization) and approximately 226 billion tokens using a custom tokenizer.
• Data Sources: Aggregated from nine diverse sources, including Common Crawl subsets (CC100, mOSCAR, mC4), Wikipedia, FineWeb v2, Aroeira, and specialized datasets like Blogset-br and HPLT 2.0.
• Preprocessing Pipeline: Inspired by FineWeb v1, the pipeline included:
  • Language Filtering: Using GlotLID (threshold 0.799).
  • Deduplication: MinHash with 112 hashes across 14 buckets.
  • Quality Filtering: Removal of low-quality content (e.g., lines with <3 words, code snippets like "Javascript", "lorem ipsum", and documents with poor punctuation or character distribution).
  • Result: A clean, deduplicated corpus of ~700 million documents (2.2 TB).

B. Model Architecture: NorBERTo

The authors trained NorBERTo, a ModernBERT-style encoder tailored for Portuguese, trained from scratch on Aurora-PT.

• Architecture Features:
  • Rotary Positional Embeddings (RoPE): Enables better extrapolation to longer contexts.
  • Hybrid Attention: Alternates between global attention (1st layer and every 3rd layer) and local attention (fixed window) to optimize efficiency for long sequences.
  • Optimizations: Sequence packing, unpadding during training, bias-free linear projections, and Gated Linear Units (GeGLU).
• Configurations:
  • NorBERTo-base: ~150M parameters, 22 layers, hidden size 768.
  • NorBERTo-large: ~395M parameters, 28 layers, hidden size 1,024.

3. Key Contributions

1. Aurora-PT: The release of the largest open monolingual Portuguese corpus (331B tokens), surpassing previous resources like Aroeira and GigaVerbo.
2. NorBERTo: The first ModernBERT-based encoder for Portuguese, trained from scratch, offering long-context support and modern efficiency.
3. Comprehensive Benchmarking: A systematic evaluation of NorBERTo against state-of-the-art baselines (BERTimbau, Albertina, XLM-RoBERTa) on standardized Portuguese benchmarks (ASSIN 2, PLUE).
4. Right-Sized AI Advocacy: Demonstrating that compact, specialized encoders can outperform larger, older models and offer a more efficient alternative to LLMs for specific NLP tasks.

4. Results

The models were evaluated on ASSIN 2 (Semantic Similarity and Textual Entailment) and PLUE (Portuguese Language Understanding Evaluation, a Portuguese adaptation of GLUE).

A. Perplexity

NorBERTo achieved the lowest perplexity (20,437.97) on Brazilian Portuguese corpora, significantly outperforming baselines like brWaC (107,721) and Aroeira (306,549), indicating superior language modeling capabilities.

B. PLUE Benchmark (Classification & Inference)

NorBERTo-large established new state-of-the-art (SOTA) results across all evaluated tasks:

• MRPC (Paraphrase Detection): 0.9191 F1 (surpassing BERTimbau-large by +3.2 points).
• RTE (Recognizing Textual Entailment): 0.7689 Accuracy (surpassing BERTimbau-large by +1.4 points).
• WNLI (Coreference): 0.5774 F1 (highest among all models).
• Note: The base model also outperformed larger legacy models (e.g., BERTimbau-large) on MRPC.

C. ASSIN 2 Benchmark

• Textual Entailment: NorBERTo-large (sequence classification) achieved the highest F1 score of ~0.904, surpassing BERTimbau-large (0.881) and Albertina.
• Semantic Similarity: NorBERTo lagged slightly behind BERTimbau-large (Pearson correlation 0.766 vs. 0.852). The authors attribute this to the fact that BERTimbau benefited from continued pre-training on English-initialized checkpoints, which aids in transferring syntactic/semantic knowledge, whereas NorBERTo was trained from scratch.

5. Significance and Future Work

• Efficiency vs. Performance: The paper proves that combining modern architectures (ModernBERT) with massive monolingual data (Aurora-PT) allows mid-sized models (150M–400M parameters) to match or exceed the performance of older, larger models (900M parameters) and compete with LLMs in discriminative tasks.
• Deployment Viability: NorBERTo is designed for realistic deployment: it is efficient to serve, easy to fine-tune, and suitable as a backbone for Retrieval-Augmented Generation (RAG) systems.
• Limitations: The authors acknowledge that the training token count (226B) is smaller than the trillions used for the original English ModernBERT, which may limit generalization on rare linguistic patterns or broad factual knowledge.
• Future Directions: Plans include scaling NorBERTo to 1–2B parameters, exploring contrastive learning for similarity tasks, and training decoder-only and encoder-decoder models on Aurora-PT.

In conclusion, NorBERTo represents a significant leap forward for Portuguese NLP, providing a robust, modern, and open foundation for building efficient, high-performance language systems without relying on the prohibitive costs of general-purpose LLMs.