Technical Summary: Bridging the Gap via Transfer Learning to Malaysian English

Problem Statement

Malaysian English is a low-resource creole language characterized by distinctive morphosyntactic adaptations, semantic features, and frequent code-switching between English, Malay, Chinese, and Tamil. While Pre-trained Language Models (PLMs) like BERT and XLM-RoBERTa have shown success in standard English and other languages, they underperform on Malaysian English text. Existing State-of-the-Art (SOTA) solutions fail to adequately capture entities and relations in this context due to the language's unique blend of loanwords, compound words, and derivations from local languages. This gap hinders effective processing for downstream Natural Language Processing (NLP) tasks, specifically Named Entity Recognition (NER) and Relation Extraction (RE).

Methodology

The authors propose a transfer learning approach involving the development and evaluation of two new models: MENmBERT and MENBERT. The methodology is structured into three main phases:

1. Data Resources

• MEN-Corpus: A corpus of 14,320 Malaysian English news articles used for pre-training.
• MEN-Dataset: A benchmark dataset containing 200 news articles with 6,061 manually annotated entities and 4,095 relation instances (adapted primarily from DocRED and ACE-2005, focusing on DocRED labels to ensure label distribution robustness).

2. Model Development Strategies

The authors explored three pre-training strategies to determine the most effective approach for Malaysian English:

• MENmBERT-FP (Further Pre-training): Further pre-training the bert-base-multilingual-cased model on the MEN-Corpus. This leverages the multilingual nature of the base model to handle the code-switching inherent in Malaysian English.
• MENBERT-FP (Further Pre-training): Further pre-training the bert-base-cased model on the MEN-Corpus.
• MENBERT-SC (Pre-training from Scratch): Training a bert-base-cased architecture from scratch using the MEN-Corpus and a custom vocabulary generated via BertWordPieceTokenizer.

3. Fine-Tuning and Evaluation

The pre-trained models were fine-tuned for NER and RE tasks using the MEN-Dataset.

• Hyperparameter Optimization: A grid search was conducted to optimize learning rates, batch sizes, epochs, and weight decay for both tasks.
• Baselines: Performance was compared against standard bert-base-cased and bert-base-multilingual-cased models fine-tuned on the same dataset.
• Evaluation Metrics: Models were evaluated using Precision, Recall, and F1-Score. For NER, performance was analyzed both globally and across 12 specific entity labels. For RE, the study focused on relation instances derived from DocRED.

Key Contributions

1. MENmBERT: The introduction of a BERT-based model pre-trained specifically on a Malaysian English news corpus. This model is designed to capture the contextual nuances of a low-resource creole language.
2. Benchmarking and Analysis: The authors provide a comprehensive evaluation of transfer learning strategies (Further Pre-training vs. Training from Scratch) for Malaysian English. They release the MEN-Dataset, code, and pre-trained models to facilitate future NLP research in this domain.
3. Granular Performance Insights: The study moves beyond aggregate metrics to analyze performance across specific entity labels and relation types, revealing where language-specific pre-training offers the most significant advantages.

Experimental Results

Pre-training Validation

In a masked token prediction task using 100 Malaysian English sentences, MENmBERT-FP demonstrated the highest accuracy in predicting masked tokens, particularly for local language terms (e.g., Bumiputera, Taman). This confirmed that further pre-training on language-specific data enhances contextual understanding compared to generic multilingual or English-only baselines.

Named Entity Recognition (NER)

• Overall Performance: MENmBERT-FP achieved the highest overall Micro F1-Score of 0.831, representing a 1.52% improvement over the bert-base-multilingual-cased baseline (0.819).
• Label-Specific Gains: While the overall improvement appeared modest, a deeper analysis revealed significant gains for 6 out of 11 evaluated entity labels. On average, the F1-Score difference between MENmBERT-FP and the multilingual baseline was +10%.
• Training from Scratch: The MENBERT-SC model performed poorly (F1-Score: 0.316), suggesting that training from scratch on this specific low-resource dataset without the benefit of a pre-trained multilingual foundation is ineffective.

Relation Extraction (RE)

• Overall Performance: MENmBERT-FP achieved an F1-Score of 0.353, a significant 26.27% improvement over the bert-base-multilingual-cased baseline.
• Correlation with NER: The authors observed that the improvement in RE was closely linked to the model's enhanced ability to correctly classify entity pairs. Since RE relies on accurate entity detection, the superior NER performance of MENmBERT-FP directly contributed to better RE outcomes.
• Comparison to SOTA: The authors note that the absolute F1-scores for RE are lower than those reported on larger, standard datasets like DocRED (e.g., ~54% F1). They attribute this to the smaller size of the MEN-Dataset (2,237 relation instances) and the inherent complexity of the low-resource setting.

Significance and Claims

The paper claims that pre-training language models on language-specific and geographically-focused corpora is a promising approach for improving NLP performance in low-resource settings.

• Inclusivity in NLP: The work contributes to the field by demonstrating the applicability of PLMs to non-standard English varieties, promoting inclusivity in NLP research.
• Strategic Transfer Learning: The findings suggest that for low-resource creole languages like Malaysian English, further pre-training a multilingual model (MENmBERT-FP) is superior to both training from scratch and using generic multilingual models without adaptation.
• Resource Availability: By publishing the MEN-Dataset and code, the authors provide a foundational resource for the community to build upon, addressing the scarcity of annotated data for Malaysian English.

The authors remain modest in their claims, acknowledging that while MENmBERT improves performance, the RE task still faces challenges due to dataset size, and future work will focus on data augmentation and expanding to other downstream tasks.