YoNER: A New Yorùbá Multi-domain Named Entity Recognition Dataset

1. Problem Statement

Named Entity Recognition (NER) is a foundational Natural Language Processing (NLP) task, yet research for low-resource African languages like Yorùbá is severely constrained by a lack of diverse, high-quality resources.

• Current Limitations: Existing Yorùbá NER datasets (e.g., MasakhaNER, WikiAnn) are restricted primarily to the News domain or automatically generated from Wikipedia.
• The Gap: These resources fail to capture the linguistic diversity of Yorùbá as spoken in informal contexts (blogs, movies, radio) or specific cultural domains (Bible). Consequently, models trained on news data struggle to generalize to other domains due to shifts in vocabulary, entity distribution, and writing styles.
• Research Questions: The paper seeks to answer:
  1. How well does News-based NER transfer to diverse domains (Bible, Blogs, Movies, Radio)?
  2. Can small amounts of in-domain data (e.g., 200 sentences) improve cross-domain transfer?
  3. Do monolingual BERT models outperform multilingual models in low-resource, multi-domain settings?

2. Methodology

A. Dataset Curation (YoNER)

The authors created YoNER, the first multi-domain, human-annotated Yorùbá NER dataset.

• Domains: Five distinct domains were selected: Bible, Blogs, Movies, Radio Broadcasts, and Wikipedia.
• Data Collection:
  • Bible: Selected chapters from the Yorùbá Bible rich in proper nouns.
  • Blogs: Posts from Nairaland and social media (informal, code-mixed).
  • Movies: Transcripts from Yorùbá movies (e.g., Oleku, Apoti Eri) obtained from YouTube/Netflix.
  • Radio: Transcripts from stations like Fresh FM and Tiwa-n-Tiwa.
  • Wikipedia: Encyclopedic articles.
• Statistics: The corpus contains 5,148 sentences and 100,795 tokens.
• Annotation: Three native Yorùbá speakers manually annotated the data using IOB2 tagging for three entity types: Person (PER), Organization (ORG), and Location (LOC).
  • Quality Control: Inter-annotator agreement (Fleiss' Kappa) was substantial at the token level (0.71–0.88) but lower at the entity span level (0.30–0.61), a common NER challenge. Disagreements were resolved via majority vote.
• Split: Data was split into Train (200 sentences per domain), Dev (100), and Test (500–1,128 per domain).

B. Model Development (OyoBERT)

To address the lack of a strong monolingual model, the authors pre-trained OyoBERT.

• Architecture: Based on BERT-base and BERT-large configurations.
• Training Data: A mix of ~347MB of monolingual Yorùbá text (mC4, MT5, Wikipedia, news sites) and >1GB of synthetic data generated by translating English texts (from the WURA corpus) using NLLB-200.
• Objective: Standard masked language modeling with a token-dropping objective for efficiency.

C. Experimental Setup

The authors conducted three main experimental series:

1. Cross-Domain Transfer (News $\to$ Others): Fine-tuned models on MasakhaNER 2.0 (News) and evaluated on YoNER domains.
2. Few-Shot In-Domain Transfer: Fine-tuned models on small in-domain subsets (200 sentences) to test adaptability.
3. Monolingual vs. Multilingual: Compared the new OyoBERT (monolingual) against African-centric multilingual models (AfroXLMR, AfriBERTa) and general multilingual models (mBERT, XLM-R).
4. Cross-Lingual Baseline: Compared English-to-Yorùbá transfer (using English CoNLL/OntoNotes) against Yorùbá-to-Yorùbá transfer.

3. Key Contributions

1. YoNER Dataset: A high-quality, multi-domain NER dataset for Yorùbá, expanding beyond the News/Wikipedia limitation to include culturally rich domains like Movies and Radio.
2. OyoBERT: The introduction of the first strong monolingual Yorùbá BERT model, which outperforms general multilingual models in in-domain tasks.
3. Comprehensive Benchmarking: A systematic evaluation of cross-domain transfer, few-shot learning, and the efficacy of monolingual vs. multilingual models in a low-resource African context.
4. Public Release: Both the YoNER dataset and pretrained OyoBERT models are released publicly to foster future research.

4. Key Results

A. Cross-Domain Transfer Performance

• Domain Drop: Models trained on News data (MasakhaNER 2.0) performed well on formal domains like Wikipedia and Bible (F1 $\approx$ 77%) but dropped significantly on Blogs (F1 $\approx$ 59%) and Movies (F1 $\approx$ 50%).
• Entity Scarcity: The poor performance on Blogs and Movies correlates with the low frequency of annotated entities in these domains.
• Model Superiority: African-centric models (AfroXLMR-large-76L) consistently outperformed general multilingual models (mBERT, XLM-R). XLM-R performed poorly because Yorùbá was not a primary pre-training language, and mBERT's Yorùbá data is limited.

B. Impact of Small In-Domain Data

• Source Domain Matters: Closely related formal domains (News $\leftrightarrow$ Wikipedia) transfer best.
• In-Domain Data: Adding small amounts of in-domain data (200 sentences) significantly improved performance, but the News domain remained the strongest source for generalization due to its larger training size.
• Cross-Lingual vs. Cross-Domain: Surprisingly, transferring from English (large data) to Yorùbá performed better than transferring from small Yorùbá in-domain data. However, Yorùbá News $\to$ Yorùbá Other (same language, different domain) still outperformed English $\to$ Yorùbá (cross-lingual) when the source data size was comparable, highlighting that language proximity is crucial.

C. Monolingual (OyoBERT) vs. Multilingual

• In-Domain: OyoBERT outperformed multilingual models (AfroXLMR) in culturally specific domains like Movies and Radio.
• Cross-Domain: Multilingual models (AfroXLMR) generally held an advantage in cross-domain transfer, likely due to exposure to broader data distributions during pre-training.
• Conclusion: Monolingual models are superior for specific, culturally rich tasks, while multilingual models offer better generalization when data is scarce across domains.

D. Entity Type Analysis

• ORG Entities: Models consistently struggled with Organization (ORG) entities, often achieving 0% F1 in domains like Bible and Movies where ORG entities were absent or rare.
• PER Entities: Person entities were the most frequent and easiest to detect across all domains.

5. Significance and Future Work

• Cultural Representation: This work addresses the "digital divide" by providing resources that recognize African names and cultural concepts often missed by generic AI systems.
• Low-Resource Strategy: The findings suggest that for low-resource languages, monolingual pre-training (even with synthetic data) is highly effective for in-domain tasks, while multilingual models remain necessary for broad cross-domain generalization.
• Future Directions:
  • Extending YoNER to include more entity types (e.g., DATE, GPE) and additional domains.
  • Investigating Large Language Models (LLMs) on this dataset.
  • Analyzing biases introduced by machine-translated synthetic training data.

In summary, YoNER establishes a new benchmark for Yorùbá NLP, demonstrating that while African-centric models are essential, the domain gap remains a significant challenge that requires tailored, multi-domain datasets and strategic model selection.