Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark

1. Problem Statement

Despite the rapid advancement of multilingual Large Language Models (LLMs), there is a critical scarcity of gold-standard evaluation benchmarks for Named Entity Recognition (NER) across diverse languages. Existing resources often lack:

• Cross-lingual consistency: Annotations are rarely aligned across parallel datasets, making it difficult to evaluate cross-lingual transfer.
• Typological diversity: Many benchmarks focus on high-resource Indo-European languages, neglecting agglutinative, analytic, and introflexive languages.
• Standardization: Different projects use varying tagsets and annotation guidelines, hindering unified research.

The authors aim to address these gaps by expanding the Universal NER (UNER) project to provide a massive, standardized, and typologically diverse benchmark for evaluating NER systems in a cross-lingual setting.

2. Methodology

Dataset Construction and Expansion

The paper introduces UNER v2, a significant extension of the 2024 release (UNER v1).

• Scale: The repository now contains 30 datasets across 22 languages, totaling 3 million annotated tokens and 59,000 entities.
• New Additions: 11 new datasets were added in 10 language varieties (Greek, Hebrew, Norwegian Nynorsk, Norwegian Bokmål, Slovenian, Swedish, Czech, Indonesian, Japanese, Korean, and Romanian).
• Parallel Benchmarks: A key feature is the inclusion of Parallel Universal Dependencies (PUD) datasets for Czech, Indonesian, Japanese, and Korean, aligned with existing PUD datasets in other languages. This allows for direct cross-lingual comparison on parallel text.
• Domain Diversity: The datasets cover diverse domains including news, fiction, spoken transcripts, and a new legal domain (Romanian LegalNERo).

Annotation Guidelines and Process

• Tagset: The project maintains a simple, transferable tagset of three core entity types: Person (per), Location (loc), and Organization (org). Other entities are labeled as "Other" but excluded from the final release to maintain focus.
• Guidelines: Annotations follow rigorous, shared guidelines to ensure consistency.
• Quality Control:
  • Inter-Annotator Agreement (IAA): At least 5% of data in each new dataset is annotated by multiple volunteers to calculate IAA.
  • Tools: Annotations are collected using TALEN, a web-based span-level sequence labeling tool.
  • Format: Data is released in a custom .conllu format (.iob2) compatible with standard NER pipelines.
• Data Transfer: Existing datasets (Norwegian NDT, Slovenian SSJ, Romanian LegalNERo) were converted to the UNER standard, involving manual correction and label mapping to ensure alignment with UNER guidelines.

Experimental Setup

The authors benchmarked the new datasets using two distinct approaches:

1. Encoder-Based Models: Fine-tuning XLM-RoBERTa-Large (560M parameters) on the training data. Experiments included both in-language and cross-lingual transfer scenarios.
2. Large Language Models (LLMs): Prompting three state-of-the-art models (Gemini 2.5 Flash Lite, Claude Sonnet 4, GPT-5 Mini) with the exact annotation guidelines (no prompt engineering) to act as annotators. Their outputs were compared against human gold standards.

3. Key Contributions

• UNER v2 Release: A massive expansion of the UNER resource, adding 11 new datasets and bringing the total to 30 datasets across 22 languages.
• Typological Coverage: The dataset now spans five macro-language families (Indo-European, Afroasiatic, Austronesian, Koreanic, Japonic) and includes diverse morphological types (fusional, agglutinative, analytic, introflexive) and scripts (Latin, Greek, Hebrew, CJK).
• Parallel Evaluation Benchmarks: The addition of PUD-based datasets for four new languages enables rigorous evaluation of cross-lingual transfer capabilities on parallel text.
• LLM Baseline Analysis: A comprehensive study evaluating the current capabilities of generative LLMs as NER annotators compared to human standards.
• Open Resources: All data, guidelines, and code are publicly available under a CC-BY-SA-4.0 license on GitHub and Hugging Face.

4. Results

Encoder-Based Models (XLM-R)

• In-Language Performance: Fine-tuned XLM-R achieved the highest performance on in-language test sets (diagonal in Figure 3).
• Cross-Lingual Transfer:
  • Transfer performed well between typologically similar European languages (F1 > 0.60).
  • Performance dropped significantly for typologically distant languages (e.g., Japanese and Korean), confirming that cross-lingual transfer remains a challenge for distant language pairs.
• Entity Type Difficulty:
  • Person (per): Highest F1 scores (least ambiguous).
  • Organization (org) & Location (loc): Consistently lower F1 scores due to ambiguity (e.g., metonymy where "The White House" can be an org or a location).

Large Language Models (LLMs)

• Overall Performance: LLMs achieved a maximum average F1 of 0.497 (Claude Sonnet 4), which is substantially lower than human Inter-Annotator Agreement (0.741).
• Variability: Performance varied by language and model. LLMs performed best on Greek and worst on Czech.
• Systematic Errors:
  • Over-annotation: On PUD datasets, models like GPT and Gemini annotated 30–90% more entities than humans, often labeling generic references as named entities.
  • Under-annotation: On Hebrew and Korean, models under-annotated by 30–60%.
  • Guideline Adherence: Models struggled with specific nuances, such as distinguishing geopolitical entities (Org vs. Loc) and nationalities vs. locations.

5. Significance

• Standardization: UNER v2 establishes a unified, gold-standard benchmark that allows researchers to fairly compare NER methods across a wide spectrum of languages and typologies.
• Robustness Testing: The dataset highlights the limitations of current SOTA models, particularly their struggle with typologically distant languages and ambiguous entity types (Org/Loc).
• LLM Limitations: The study provides empirical evidence that while LLMs are useful, they currently cannot replace human annotators for high-stakes NER tasks without significant prompt engineering or agentic workflows.
• Community Growth: By integrating with major resources like the Aya Dataset and skLEP, and by inviting community collaboration (via CA21167 UniDive), UNER v2 fosters a more inclusive approach to language technology, specifically targeting under-resourced and typologically diverse languages.

In conclusion, UNER v2 represents a critical step toward truly universal NLP, providing the necessary infrastructure to evaluate and improve multilingual models beyond the current limitations of high-resource language bias.