Malaysian English News Decoded: A Linguistic Resource for Named Entity and Relation Extraction

Technical Summary: Malaysian English News Decoded

Problem Statement
Natural Language Processing (NLP) tasks, specifically Named Entity Recognition (NER) and Relation Extraction (RE), face significant challenges when applied to Malaysian English (ME). ME is a creole language variant that incorporates morphosyntactic and semantic adaptations from Malay, Chinese, and Tamil, alongside Standard English. Existing state-of-the-art NLP models and datasets are predominantly trained on Standard English (American or British), rendering them inadequate for ME. The authors demonstrate that standard NER tools (spaCy, Flair, Stanza, and Malaya), when applied to ME news articles, fail to accurately detect entities exhibiting local morphosyntactic variations, achieving micro F1-Scores below 0.6. Furthermore, no annotated dataset tailored specifically to the Malaysian English context existed prior to this work, hindering the development of robust NLP resources for this low-resource language.

Methodology
To address these gaps, the authors constructed the Malaysian English News (MEN) dataset and conducted a fine-tuning experiment using the spaCy NER framework.

• Data Acquisition and Corpus Construction: The authors scraped 14,320 news articles from prominent Malaysian English portals (New Straits Times, Malay Mail, and Bernama English) across Nation, Business, and Sports categories. From this corpus, 200 articles were selected for manual annotation, comprising 120 Nation, 60 Business, and 20 Sports articles.
• Annotation Process: Four proficient annotators, trained in both Malaysian English and Bahasa Malaysia, performed the annotation over an eight-week period. The process involved:
  • Entity Annotation: Labels were adapted from OntoNotes 5.0 but expanded to include two ME-specific categories: TITLE (e.g., Datuk, Tan Sri) and ROLE (e.g., Menteri Besar). A total of 12 entity labels were used.
  • Relation Annotation: Relation labels were adapted from DocRED and ACE-2005, with a specific "NO_RELATION" label introduced for instances lacking a suitable relation.
  • Quality Control: Inter-Annotator Agreement (IAA) was calculated using F1-Scores (rather than Cohen's Kappa, due to the lack of negative cases). Disagreements were resolved by a subject matter expert acting as an adjudicator.
• Model Fine-Tuning: The authors fine-tuned the spaCy NER pipeline using the MEN dataset. They evaluated several configurations: a blank spaCy model trained from scratch, and pre-trained models (en_core_web_sm, en_core_web_lg, en_core_web_trf) fine-tuned on the MEN data. The dataset was split into training (75%), validation (15%), and testing (10%) sets.

Key Contributions

1. The MEN Dataset: The primary contribution is the release of a manually annotated dataset containing 200 news articles, 6,061 entity mentions, and 3,268 relation instances. Approximately 60% of the entity mentions (specifically PERSON, ORGANIZATION, ROLE, TITLE, and FACILITY) exhibit morphosyntactic adaptations unique to the Malaysian context.
2. Annotation Guidelines: The paper provides a detailed annotation guideline that defines 12 entity labels and 101 relation labels, including specific adaptations for ME (e.g., handling loan words and compound blends).
3. Empirical Validation of Fine-Tuning: The study demonstrates that fine-tuning NER models on a domain-specific dataset significantly improves performance for low-resource language variants compared to using pre-trained Standard English models.

Results

• Baseline Performance: Initial experiments with off-the-shelf NER tools on 30 ME sentences yielded poor results, with the best model (spaCy) achieving an F1-Score of only 0.58.
• Inter-Annotator Agreement: The entity annotation achieved a macro F1-Score of 0.818, while relation annotation achieved 0.51. Lower agreement was observed for rare entity types (e.g., WORK_OF_ART) and complex relation labels from ACE-2005.
• Fine-Tuning Performance:
  • The spacy-blank model (trained from scratch on MEN) achieved the highest overall Micro F1-Score of 0.94.
  • Fine-tuned pre-trained models (en_core_web_sm-FT, en_core_web_lg-FT, en_core_web_trf-FT) showed significant improvements over their non-fine-tuned counterparts. For instance, en_core_web_sm-FT improved from a baseline of 0.211 to 0.832.
  • On average, fine-tuned models demonstrated a +230% improvement in performance compared to pre-trained models.
  • Specific improvements were noted for ME-specific entities like TITLE and ROLE, which pre-trained models failed to detect entirely (F1-Score of 0) without fine-tuning.

Significance
The paper claims that the MEN dataset is a unique resource that addresses the scarcity of annotated data for Malaysian English. By providing a dataset that captures the specific morphosyntactic adaptations of ME, the authors enable researchers to accelerate progress in NER and Relation Extraction for this language variant. The study validates that while existing SOTA models struggle with ME, a tailored dataset combined with fine-tuning can bridge the performance gap, offering a pathway to more accurate NLP applications for Malaysian English news and similar creole language contexts. The dataset and guidelines have been made publicly available to facilitate further research.