Technical Summary: PortBERT

Problem Statement

While transformer-based models have revolutionized Natural Language Processing (NLP), efficient, language-specific models for Portuguese remain scarce. Existing Portuguese models often prioritize scale or raw accuracy, neglecting training and deployment efficiency. Furthermore, many advancements in Portuguese NLP come with high computational costs, making them difficult to deploy in latency-sensitive or resource-constrained environments. There is a lack of transparent reporting regarding training times, hardware utilization, and inference throughput for Portuguese-specific models, creating a gap in understanding the compute-performance trade-offs in this domain.

Methodology

To address these gaps, the authors introduce PortBERT, a family of RoBERTa-based encoder models designed specifically for Portuguese.

• Data Corpus: The models were pre-trained from scratch on over 450 GB of deduplicated and filtered text derived from the CulturaX dataset, which aggregates Portuguese portions of mC4 and OSCAR23. This represents a significant reduction in redundancy compared to raw crawled corpora.
• Tokenization: A dedicated byte-level Byte Pair Encoding (BPE) vocabulary of 52,000 tokens was constructed using 40 GB of randomly sampled Portuguese data. This approach, inspired by prior work on German and Dutch, aims to improve token efficiency and compression.
• Architecture & Training:
  • The models adhere to the standard RoBERTa architecture without modifications like sparse attention or extended context windows, ensuring a fair baseline for comparison.
  • Training was performed using the fairseq framework.
  • PortBERTbase was trained on a cluster of 8 NVIDIA A40 GPUs.
  • PortBERTlarge was trained on a TPUv4-128 pod.
  • Both models were trained in full precision (fp32) to ensure a direct comparison of hardware efficiency without the confounding factor of mixed-precision optimizations.
  • The training schedule included 100k update steps with a batch size of 8k, a 10k-step warmup, and polynomial learning rate decay.
• Evaluation: The models were evaluated on ExtraGLUE, a suite of translated GLUE and SuperGLUE tasks (STS-B, RTE, WNLI, MRPC). The evaluation protocol involved fine-tuning each task 28 times with varying hyperparameters (batch size and learning rate) to identify the best-performing checkpoint.

Key Contributions

1. Model Release: The authors release two open-source variants, PortBERTbase (126M parameters) and PortBERTlarge (357M parameters), trained on GPUs and TPUs respectively.
2. Comprehensive Benchmarking: The paper provides a rigorous evaluation of PortBERT against eleven other Portuguese language models (both monolingual and multilingual) on the ExtraGLUE benchmark.
3. Efficiency Reporting: Unlike many prior works, this study explicitly reports training times, inference throughput, and fine-tuning throughput, offering practical insights into model efficiency and hardware utilization.
4. Reproducibility: All models are released on Hugging Face, and fairseq checkpoints are provided to support further research.

Results

• Downstream Performance:
  • PortBERTbase achieved an average score (AVG) of 80.57, outperforming several existing monolingual models and ranking competitively against multilingual baselines. It notably achieved the highest accuracy on the WNLI task (60.56) among base models.
  • PortBERTlarge achieved an AVG of 82.26, slightly surpassing BERTimbau-large (82.23) and demonstrating the highest WNLI accuracy (61.97) among large models. While it did not dominate every single task, it consistently ranked among the top performers.
  • Both models matched or surpassed existing monolingual and multilingual models in several metrics, validating the effectiveness of training on recent, high-quality, deduplicated corpora.
• Efficiency vs. Accuracy:
  • PortBERTbase demonstrated a favorable balance, offering strong downstream performance with competitive training throughput (~25–26 samples/sec) and inference throughput (>65 samples/sec).
  • PortBERTlarge achieved a high efficiency-performance trade-off. Despite being a large model, it maintained higher training (23.3 samples/sec) and inference (70.7 samples/sec) throughput compared to other large models like XLM-RoBERTa-large and EuroBERT-610M, which suffered from lower throughput metrics.
  • The study highlights that PortBERT models offer superior parameter efficiency, with PortBERTbase (126M) outperforming larger multilingual models like XLM-RoBERTa-base (278M) on average.

Significance and Claims

The paper positions PortBERT as a robust, reproducible, and accessible alternative for practical Portuguese NLP. Its significance lies in addressing the underexplored dimension of compute-performance trade-offs.

• Complement to Large Models: While large multilingual encoders (e.g., XLM-RoBERTa, EuroBERT) offer strong performance, their high computational demands restrict practical deployment. PortBERT complements these by offering a more efficient option for monolingual applications where zero-shot transfer is not required.
• Sustainable Modeling: The work demonstrates that thoughtful model design, combined with optimized pre-training on recent corpora, can yield strong models without relying on massive, inaccessible infrastructure.
• Baseline Establishment: By using standard RoBERTa architecture and transparent training pipelines, PortBERT establishes a strong baseline for future research, allowing for fair comparisons of computational costs and efficiency in Portuguese NLP.

The authors acknowledge limitations, including the lack of dialect-specific control (Brazilian vs. European Portuguese), the absence of Whole Word Masking (WWM) due to framework constraints, and the reliance on web-based corpora which may contain residual biases. However, they assert that PortBERT provides a foundational step toward sustainable and accessible language modeling for Portuguese.