NameBERT: Scaling Name-Based Nationality Classification with LLM-Augmented Open Academic Data

1. Problem Statement

Inferring nationality from personal names is essential for equity monitoring, personalization, and sociological research. However, existing solutions face three critical limitations:

• Data Scarcity and Bias: Most models rely on small, proprietary, or regionally biased datasets (e.g., U.S.-centric voter records), leading to poor performance on underrepresented (long-tail) countries.
• Granularity Issues: Many models predict broad ethnic groups rather than fine-grained nationalities, or they are constrained by a limited number of output classes.
• LLM Trade-offs: While Large Language Models (LLMs) demonstrate strong zero-shot performance, their high computational cost, latency, and per-query pricing make them impractical for large-scale, real-time deployment.

The authors aim to build a scalable, fine-grained nationality classifier that overcomes data imbalance without relying on expensive LLM inference.

2. Methodology

A. Dataset Construction (Open Academic Graph - OAG)

The authors constructed a massive dataset of 1.4 million name-country pairs derived from the Open Academic Graph (OAG), which contains metadata for 35 million authors.

• Label Extraction: Nationality labels were heuristically extracted from author affiliation strings (e.g., extracting "UK" from "University of Oxford, UK").
• Noise Mitigation: Ambiguous entries (authors with multiple country affiliations) were removed. The dataset spans 99 countries but exhibits a severe long-tail distribution (e.g., China and USA dominate, while countries like Bhutan have <50 entries).
• Validation: An LLM (GPT-4o) was used to screen a subset of the data, verifying that names are plausible for their assigned countries, resulting in a "filtered" evaluation set.

B. LLM-Augmented Training Framework

Instead of using LLMs as inference engines, the authors used them as dataset enrichers to address class imbalance.

• Augmentation Strategy: For the 57 underrepresented countries (those with <6,000 samples), GPT-5 was prompted to generate 5,000 synthetic, realistic full names per country.
• Data Splitting: These synthetic names were split into training, validation, and test sets and merged with the original OAG data to create an augmented dataset (train_aug).
• Goal: To increase supervision for minority classes without distorting the distribution of majority classes.

C. Model Architecture: NameBERT

The core model, NameBERT, is a transformer-based classifier:

• Base: Fine-tuned bert-base-multilingual-cased.
• Input: Names tokenized via WordPiece (max sequence length 40).
• Output: A linear classification head over 99 nationalities.
• Training: Optimized with AdamW, using Macro-F1 for early stopping to ensure long-tail robustness.

D. Evaluation Protocol

The study utilized multiple evaluation splits to test different aspects of performance:

1. test_oag: Raw OAG data (in-domain, noisy labels).
2. test_filter: LLM-validated real names (cleaner, balanced).
3. test_filter_aug: Mix of validated real names and synthetic tail names (diagnostic for tail performance).
4. test_gold: Fully synthetic stress-test set (100 LLM-generated names per country) to test cross-domain generalization.
5. NaNa_test: Out-of-domain test set using Wikipedia names (from the Name2Nat dataset).

3. Key Contributions

1. New Dataset: Curated a large-scale, open-source dataset of 1.4M name-nationality pairs from OAG, covering 99 countries.
2. LLM-as-Enricher Framework: Proposed and empirically validated a pipeline where LLMs generate synthetic data to augment long-tail classes rather than performing inference.
3. NameBERT Model: Developed a transformer-based classifier that outperforms state-of-the-art (SOTA) baselines (NamePrism, EthnicSeer, Name2Nat) across in-domain and out-of-domain tasks.
4. Efficiency Benchmark: Demonstrated that a local NameBERT model is orders of magnitude faster and cheaper than LLM APIs for large-scale inference.

4. Key Results

Performance vs. Baselines

• In-Domain (OAG Data): NameBERT achieved significantly higher accuracy and Macro-F1 scores than all baselines across 12, 39, and 99-class taxonomies.
  • Example (99 classes): NameBERT (0.699 Acc) vs. Name2Nat (0.467 Acc).
• Out-of-Domain (Wikipedia/NaNa): NameBERT maintained superior performance over baselines, proving that representations learned from academic metadata transfer well to general Wikipedia names.
• LLM Zero-Shot: While GPT-5 achieved ~58.6% accuracy on 99 classes (zero-shot), NameBERT (trained) achieved ~69.9%, demonstrating the value of supervised fine-tuning.

Impact of LLM Augmentation

• Tail Performance: Augmentation yielded substantial gains on Macro-F1 for long-tail countries, particularly in the test_filter_aug setting (Macro-F1 increased from 0.350 to 0.569 for NameBERT).
• Trade-offs: In out-of-domain settings (NaNa dataset), augmentation improved tail accuracy slightly but slightly reduced overall accuracy, suggesting that synthetic data helps when the evaluation distribution matches the augmentation strategy but may introduce distribution shifts otherwise.

Computational Efficiency

• Throughput: NameBERT on a local GPU processed 1,831 names/second, compared to 3.7–4.8 names/second for LLM APIs.
• Latency: NameBERT latency was <1 ms/name (at batch size 100+), whereas LLMs had latencies of 200–270 ms/name.
• Cost: NameBERT incurred near-zero per-query costs, whereas LLM APIs cost hundreds of dollars per million names.

5. Significance and Conclusion

This work establishes a new paradigm for demographic inference: using LLMs to build better datasets for efficient, specialized models.

• Scalability: NameBERT enables the processing of millions of names in real-time, a task impossible with direct LLM inference due to cost and latency.
• Fairness: The augmentation strategy specifically targets the "long tail" of underrepresented nationalities, improving equity in demographic monitoring without sacrificing overall accuracy.
• Practicality: The study provides a reproducible, open-source framework (dataset + model) that balances the high accuracy of LLMs with the efficiency of traditional classifiers, making large-scale bias monitoring feasible for researchers and organizations.

The authors conclude that while LLM augmentation is not a universal fix for all generalization tasks, it is highly effective for improving performance on underrepresented classes when the evaluation strategy accounts for tail distributions.