Technical Summary: BamiBERT

Problem Statement

Despite the prevalence of BERT-based models in the English NLP ecosystem, Vietnamese language processing faces significant limitations. While PhoBERT has emerged as the de facto standard for Vietnamese text encoding, it suffers from two critical architectural constraints:

1. Short Context Window: It supports a maximum context length of only 256 subword tokens, limiting its ability to process long documents or complex sentence structures.
2. Dependency on External Segmentation: It requires input text to be pre-segmented into words by an external tool before processing, adding latency and complexity to deployment pipelines.

Other existing Vietnamese models, such as ViDeBERTa and ViSoBERT, either rely on smaller corpora or share similar segmentation dependencies. Multilingual models like XLM-RoBERTa offer broader coverage but often lack the monolingual specialization required for high-precision Vietnamese tasks. There is a clear need for a model that operates directly on raw text, supports longer contexts, and is trained on a larger, cleaner corpus to improve generalization.

Methodology

The authors introduce BamiBERT, a new pre-trained language model designed to overcome these limitations. The methodology involves training a model from scratch with the following specifications:

• Architecture: BamiBERT utilizes the standard "base" BERT architecture, consisting of 12 Transformer block layers.
• Pre-training Objective: The model employs the Masked Language Modeling (MLM) objective, optimized using the RoBERTa pre-training approach. This includes a dynamic masking strategy and the removal of the Next Sentence Prediction (NSP) objective.
• Tokenization: The authors extended the PhoGPT Vietnamese-specific byte-level BPE tokenizer by adding a <mask> token. This resulted in a final vocabulary size of 20,481 token types.
• Context Length: A maximum sequence length of 2048 tokens was set, a significant increase over the 256-token limit of PhoBERT.
• Training Data: The model was trained from scratch on a 129 GB corpus of uncompressed, general-domain Vietnamese text.
• Optimization: Training was conducted for 20 epochs using the Adam optimizer. The process utilized a batch size of 1024 sequence blocks distributed across 8 A100 GPUs (40GB memory each), with a peak learning rate of 0.00015. The first two epochs were dedicated to learning rate warm-up.

Key Contributions

1. Raw Text Processing: Unlike PhoBERT and ViDeBERTa, BamiBERT operates directly on raw input text, eliminating the need for external word segmentation tools.
2. Extended Context: It supports a context length of up to 2048 tokens, enabling the processing of longer documents and more complex linguistic dependencies.
3. Large-Scale Monolingual Training: It is the first Vietnamese BERT-based model trained from scratch on a 129 GB corpus, significantly larger than the 20 GB used for PhoBERT or the 60 GB used for ViDeBERTa.
4. State-of-the-Art Performance: The model establishes a new state-of-the-art (SOTA) for "base"-sized Vietnamese encoders across multiple benchmarks.

Experimental Results

The authors evaluated BamiBERT against four baselines (ViDeBERTa, ViSoBERT, XLM-RoBERTa, and PhoBERT) across 8 Vietnamese benchmark datasets covering Natural Language Inference (NLI), Named Entity Recognition (NER), sentiment analysis, spam detection, and aspect-based sentiment analysis (ABSA).

Overall Performance:

• BamiBERT achieved the best score on 11 out of 15 metrics and the second-best score on 3 others.
• It set a new SOTA for "base"-sized Vietnamese encoders.

Specific Domain Highlights:

• ViNLI (NLI): BamiBERT achieved 81.01% Accuracy and 81.15 F1, outperforming the second-ranked PhoBERT by +3.01 Accuracy and +3.10 F1.
• PhoNER_COVID19 (NER): It achieved the highest F1 score of 94.90, surpassing PhoBERT by 0.70 points.
• UIT-ViSFD & UIT-ABSA (Sentiment): BamiBERT demonstrated strong performance in aspect detection, achieving the highest F1 scores in both Hotel (79.99) and Restaurant (88.01) domains. While it trailed ViSoBERT slightly in aspect classification for the Hotel domain, it led in the Restaurant domain and maintained strong end-to-end performance.
• Cross-Domain Generalization: The model performed consistently well across both general-domain texts (ViNLI, PhoNER_COVID19) and social media/forum content (UIT-VSFC, ViSpamReviews), demonstrating resilience to domain shifts.

Significance and Claims

The paper positions BamiBERT as a robust, flexible, and high-performance alternative to the current standard, PhoBERT. Its primary significance lies in:

• Operational Efficiency: By removing the dependency on external word segmentation, BamiBERT simplifies integration into downstream applications and reduces processing latency.
• Scalability: The extended 2048-token context window allows for applications requiring long-document understanding, which were previously constrained by the 256-token limit.
• Reliability: The model's consistent top-tier performance across diverse tasks (NLI, NER, classification, and detection) and domains suggests it is a reliable default choice for real-world Vietnamese NLP pipelines, particularly those operating under domain heterogeneity.

The authors conclude that BamiBERT effectively addresses the key limitations of existing monolingual encoders, offering strong cross-domain generalization and setting a new benchmark for Vietnamese language models. The model and code are released publicly on Hugging Face.