Technical Summary: HalleluBERT

Problem Statement

Despite the transformative impact of transformer-based models like BERT and RoBERTa on Natural Language Processing (NLP), the Hebrew language lacks a large-scale, RoBERTa-style encoder trained from scratch and released in both base and large variants. Existing Hebrew models face significant limitations:

• Corpus and Scale: Previous models like HeBERT and AlephBERT were trained on comparatively modest corpora (approx. 10–18 GB), limiting their coverage of Hebrew's complex morphology and domain diversity.
• Training Transparency: While HeRo introduced RoBERTa-style pre-training for Hebrew, it lacked full reporting of critical training details (e.g., effective batch size, total update steps) and appeared to be trained under limited compute resources.
• Model Variants: There was no publicly available large-variant Hebrew model to test the benefits of scaling parameters for this specific language.

Consequently, there is a gap in high-quality, reproducible resources for Hebrew NLP that can match the performance of multilingual models (like XLM-RoBERTa) while leveraging the advantages of monolingual pre-training.

Methodology

Data and Pre-processing

The authors constructed a pre-training corpus totaling approximately 49.1 GB of deduplicated Hebrew text. This dataset combines:

• HeDC4: A cleaned, language-identified web corpus derived from mC4 and OSCAR22 (approx. 47.5 GB).
• Wikipedia: A Hebrew Wikipedia dump (approx. 1.6 GB).
  Documents were shuffled prior to training to mitigate ordering effects from source-specific crawls.

For tokenization, the authors employed Byte-Pair Encoding (BPE) operating at the byte level, consistent with RoBERTa. Instead of using an English-optimized tokenizer (e.g., GPT-2), they trained a Hebrew-specific vocabulary on 20 GB of shuffled text. This resulted in a 52k subword inventory tailored to Hebrew's orthographic and morphological properties.

Model Architecture and Pre-training

The paper introduces HalleluBERT, a RoBERTa-style encoder family with two variants:

• HalleluBERTbase: Comparable in size to existing Hebrew BERT models (~126M parameters).
• HalleluBERTlarge: A scaled-up variant (~357M parameters).

Training was conducted using the fairseq framework on a 128-core TPUv4 pod. Key training hyperparameters included:

• Precision: Full precision (no mixed precision was used due to framework limitations).
• Schedule: 100,000 update steps with a 10k-step warmup and polynomial learning rate decay.
• Batch Size: Global batch size of 8k.
• Learning Rates: 0.0004 for the base model and 0.00015 for the large model.
• Duration: The base model trained in ~30.2 hours; the large model required ~6.0 days.

Evaluation Protocol

The models were evaluated on native Hebrew benchmarks for Named Entity Recognition (NER) and Sentiment Classification:

• NER: Evaluated on the BMC dataset and the token-based variant of NEMO2.
• Sentiment: Evaluated on SMCD (Social Media Comments Deduplicated), a cleaned version of the OmiLab dataset containing ~8,465 unique samples.

The evaluation pipeline utilized a grid search over batch sizes {16, 32} and learning rates {5e-6 to 5e-5} on NVIDIA RTX 3090 GPUs. Models were fine-tuned for up to 30 epochs with early stopping. Performance was measured using micro-averaged F1 for NER and macro-averaged F1 for sentiment.

Key Contributions

1. Release of HalleluBERT: The authors released both base and large variants of a RoBERTa-style encoder for Hebrew, trained from scratch on a large-scale, cleaned corpus.
2. Custom Tokenizer: Development and release of a Hebrew-specific byte-level BPE tokenizer (52k vocabulary).
3. Comprehensive Benchmarking: A rigorous comparison against existing monolingual models (HeBERT, AlephBERT, HeRo) and multilingual baselines (mBERT, XLM-RoBERTa).
4. Open Science: All model weights and the tokenizer are released under the MIT license to support reproducible research.

Results

HalleluBERT achieved state-of-the-art performance across the evaluated tasks:

• Named Entity Recognition:
  • HalleluBERTbase achieved the highest micro-F1 scores on both BMC (93.33) and NEMO (87.06), outperforming all other base models.
  • HalleluBERTlarge maintained a consistent advantage in NEMO (88.70) and the NER average, confirming that scaling benefits NER performance.
• Sentiment Classification:
  • While AlephBERT led among base models (83.66), HalleluBERTlarge achieved the highest classification score overall (84.91), surpassing XLM-RoBERTalarge by over one point.
• Overall Performance:
  • HalleluBERTlarge attained the highest unweighted mean score across all three benchmarks (BMC, NEMO, SMCD) with 88.95, ranking first among all evaluated models.
  • HalleluBERTbase ranked second overall (87.83), outperforming all other baselines in the combined average.

Notably, the results indicate that monolingual pre-training on large, high-quality corpora yields superior performance compared to multilingual encoders for Hebrew-specific tasks.

Significance and Claims

The paper posits that HalleluBERT demonstrates the value of large-batch, monolingual pre-training for Hebrew. The authors claim that substantial performance gains are achievable even with conservative hyperparameter choices (e.g., full precision training, standard learning rates) when the scale of data and compute is sufficient.

The work highlights that:

1. Monolingual superiority: For Hebrew, a model trained exclusively on high-quality Hebrew text outperforms multilingual counterparts, reinforcing the need for language-specific resources.
2. Scalability: Scaling from base to large variants provides consistent, albeit sometimes marginal, improvements, particularly in NER tasks.
3. Reproducibility: By releasing the model and tokenizer, the authors aim to establish a new standard baseline for Hebrew NLP, addressing previous gaps in training transparency and model availability.

The authors modestly acknowledge limitations, including the lack of confidence intervals (due to fixed random seeds), the absence of mixed-precision training (which limited computational efficiency), and the exclusion of dialectal or domain-specific variations (e.g., Biblical Hebrew, legal text) in the pre-training data. They suggest future work could explore Whole Word Masking (WWM), vocabulary ablations, and broader evaluation suites.