Technical Summary: Large Language Models Naively Recover Ethnicity from Individual Records

Problem Statement

Individual-level data on race, ethnicity, caste, and religious sect are critical for social science research but are frequently absent from administrative records. Researchers often rely on Bayesian Improved Surname Geocoding (BISG) to infer these identities using surnames and geographic data. However, BISG faces fundamental limitations:

1. Data Dependency: It requires specific training data (e.g., Census surname-race distributions) that rarely exists outside the United States.
2. Categorical Rigidity: It is constrained to official US Census categories, making it unsuitable for studying religious sects, castes, or non-US ethnic groups.
3. Information Loss: It typically utilizes only surnames, discarding potentially informative first and middle names.
4. Socioeconomic Bias: BISG systematically misclassifies racial minorities in wealthier neighborhoods as White, introducing endogeneity into downstream analyses.

Methodology

The paper proposes using Large Language Models (LLMs) to infer ethnicity directly from names and optional metadata without labeled training data. The approach involves:

• Prompting: Constructing zero-shot prompts that request classification into context-specific categories (e.g., US racial groups, Lebanese religious sects, Indian castes) based on a name and location.
• Models Tested: Six models were evaluated, including proprietary systems (Gemini 3 Flash, GPT-4o, GPT-4.1-mini) and open-source alternatives (DeepSeek v3.2, GLM-4.7, GLM-4.7-Flash).
• Validation Strategy:
  • Individual-Level (US): Stratified samples from Florida and North Carolina voter files with self-reported race.
  • Individual-Level (International): Lebanese voter rolls (religious sect), Indian Members of Parliament from reserved constituencies (Scheduled Castes/Tribes), and Indian land records (caste).
  • Aggregate-Level: Comparison of inferred ethnic distributions against census data for India, Uganda, Nepal, Armenia, Chile, and Costa Rica using original full-count voter rolls.
  • Scaling: Knowledge distillation experiments where small transformer models (Qwen3) were fine-tuned on LLM-generated labels.
  • Feature Analysis: Testing the impact of additional metadata (age, party registration, gender, zip code) and "reasoning" modes.

Key Results

1. Accuracy vs. BISG (United States)
On balanced stratified samples from Florida and North Carolina, LLM-based classification significantly outperformed BISG:

• Gemini 3 Flash achieved 83.8% (Florida) and 83.5% (North Carolina) accuracy.
• BISG (surname + county) achieved 68.2% and 68.9%, respectively.
• Recall Improvements: LLMs showed substantial gains for minority groups. For Black voters, recall improved from ~49% (BISG) to ~81% (Gemini). For Hispanic voters, recall exceeded 90% in Florida.
• Input Sensitivity: Removing first names reduced accuracy by 8.7 percentage points, confirming that full names carry significant demographic signal beyond surnames.

2. International and Cross-Categorical Performance

• Lebanon (Religious Sect): Gemini 3 Flash achieved 64.3% overall accuracy. Performance was high for groups with distinctive naming conventions (e.g., Armenian Orthodox at 97.4%, Shia at 86.4%) but lower for overlapping groups (Roman Catholic/Orthodox).
• India (Caste):
  • MPs: Gemini 3 Flash achieved 99.2% accuracy in distinguishing Scheduled Castes (SC) from Scheduled Tribes (ST) based on names alone.
  • Land Records/Sarpanch: Accuracy varied by region (74.0% in Bihar vs. 57.0% in Rajasthan), correlating with the presence of explicit caste markers in local naming conventions.
• Aggregate Validation: In countries with distinctive naming conventions (India, Armenia), LLMs recovered population distributions with high fidelity (e.g., India's Hindu/Muslim shares within 0.5 percentage points of census data). Performance degraded where naming conventions overlapped significantly (e.g., Indigenous populations in Costa Rica).

3. Impact of Metadata and Reasoning

• Metadata: Adding party registration to the prompt increased accuracy by 2.0 percentage points (to 86.7%) in Florida. Age provided a marginal gain (+0.3pp), while gender and finer geography (zip code) offered no improvement or slight degradation.
• Reasoning: Enabling "extended reasoning" modes yielded mixed results. It improved accuracy for smaller models (e.g., Qwen 8B +3.4pp in Florida) but offered negligible or negative gains for larger models, while significantly increasing inference time and token costs.

4. Scaling and Cost
Fine-tuning small models (e.g., Qwen3-1.7B) on a small set of LLM-labeled data (n=1,200) allowed local deployment with accuracy approaching the "teacher" model (Gemini) and exceeding BISG (79.7% vs. 76.2% in Florida), eliminating API costs.

5. Mitigation of Income Bias
The study replicated findings that BISG misclassification rates for Black voters correlate positively with neighborhood income (wealthier areas = higher misclassification as White). In contrast, LLM classification showed no statistically significant relationship between misclassification rates and neighborhood income, suggesting LLMs do not rely on geographic priors in the same biased manner as Bayesian models.

Significance and Claims

The paper claims that LLMs provide a viable, flexible alternative to BISG for ethnicity inference that:

1. Requires no labeled training data, enabling application in countries lacking Census surname-geography distributions.
2. Supports arbitrary classification schemes, allowing researchers to study religious sects, castes, and non-US ethnic categories without being constrained by official administrative definitions.
3. Leverages full name information (first and middle names) and diverse metadata (e.g., party registration) that traditional methods discard.
4. Reduces systematic socioeconomic bias inherent in geographic-prior-based methods like BISG.
5. Enables local, low-cost deployment through the fine-tuning of small open-source models on LLM-generated labels.

The authors conclude that while performance varies by context depending on the distinctiveness of naming conventions, the method opens new possibilities for studying ethnic politics in global contexts where traditional inference methods are inapplicable.