Benchmarking Open-Weight Foundation Models for Global AI Technical Governance

Technical Summary: Benchmarking Open-Weight Foundation Models for Global AI Technical Governance

Problem Statement
Large language models (LLMs) are increasingly utilized for synthesizing cross-national AI governance data. However, existing literature suggests these models suffer from "geographic bias," producing less accurate responses for countries underrepresented in training data (primarily Global South nations). Previous empirical studies on this phenomenon are methodologically compromised by three critical limitations: (1) reliance on proprietary, closed-source models that prevent independent replication; (2) evaluation of model knowledge regarding years falling after the training data cutoff, conflating temporal ignorance with geographic bias; and (3) the use of coarse binary classification schemes that fail to distinguish between confident fabrication (hallucination) and honest acknowledgments of uncertainty. These limitations undermine the validity of findings regarding geographic equity in AI governance analysis.

Methodology
This study addresses the aforementioned limitations through a rigorous, reproducible benchmarking framework involving four open-weight frontier language models: Meta's Llama 4 Maverick, Mistral AI's Mistral Large 3, Alibaba Cloud's Qwen3-235B-A22B, and DeepSeek's DeepSeek-V3-0324. The selection ensures geopolitical balance, with two models from Western developers and two from Chinese developers.

The evaluation utilizes the Global AI Dataset v2 (GAID v2), a verified ground-truth database containing 24,453 indicators across 227 countries. From this, 18 indicators mapped to the eight thematic dimensions of the IEEE IRAI 2026 framework (Ethics, Safety, Security, Transparency, Fairness, Accountability, Regulation, and Adoption) were selected. The study generated approximately 2,990 country-metric-year observations across six evaluation years (2010, 2013, 2016, 2019, 2022, 2023).

For each observation, three query variants were generated (direct numeric, comparative regional, and contextual governance framing), resulting in a total of 35,880 prompts. Responses were classified using a five-category rule-based scheme: Verified Accuracy (VF), Confident Fabrication (HF), Honest Refusal (HR), Qualitative Hedging (QH), and Misattribution (MF). Statistical analysis employed mixed-effects logistic regression to estimate geographic disparities and a difference-in-differences (DiD) design to assess whether the national origin of the model developer moderates accuracy differentials.

Key Results
Contrary to the dominant hypothesis in existing literature, the study found that fabrication rates were significantly higher for Global North countries than for Global South countries across all four models.

• Regional Variation: Africa exhibited the highest mean Verified Accuracy (35.1%), followed by the Americas (30.2%), while Europe (24.4%) and Oceania (23.5%) showed lower accuracy. Africa had significantly lower odds of fabrication compared to Europe (OR=0.52).
• Model Performance: Mistral Large 3 demonstrated the highest accuracy (lowest fabrication rate at 61.2%), while Llama 4 Maverick had the highest fabrication rate (85.1%). Model identity was the strongest predictor of fabrication rate.
• Developer Origin: The DiD analysis provided only marginal evidence (p=0.076) that developer origin moderates geographic bias. Chinese-origin models showed a slightly larger accuracy advantage for Global South queries compared to Western-origin models, but the effect was small and not robust to multiple comparison corrections.
• Thematic Variation: Accuracy varied drastically by theme. The "Safety" dimension (specifically AI compute capacity and model parameters) exhibited near-total fabrication (VF rates near 0%), whereas the "Regulation" dimension showed the highest accuracy (42.2% VF). Principal Component Analysis (PCA) revealed that geographic bias is not a single, general factor but is diffuse and thematically structured, requiring 15 components to explain 90% of the variance.
• Temporal Effects: Proximity to the training data cutoff did not independently explain geographic accuracy patterns once model identity and thematic dimensions were controlled.
• Response Behavior: Honest Refusals (HR) were effectively absent (<0.1%), indicating that frontier models almost never decline to answer factual queries, instead opting to provide numeric estimates regardless of knowledge certainty.

Significance and Claims
The paper claims to provide the first rigorous, reproducible assessment of geographic bias using open-weight models and a verified ground-truth dataset. Its primary contributions include:

1. Methodological Correction: It demonstrates that previous findings of geographic bias may be confounded by temporal limits and binary classification schemes. By distinguishing fabrication from refusal, the study reveals a more complex reality where models often fabricate confidently for high-value Global North indicators rather than simply refusing Global South queries.
2. Thematic Specificity: The study establishes that geographic accuracy differentials are not uniform but are heavily dependent on the thematic domain. The extreme inaccuracy regarding AI compute metrics (Safety) suggests a structural limitation in LLMs' ability to handle quantitative hardware data, regardless of geography.
3. Governance Implications: The findings suggest that no current open-weight frontier model is reliable for precise quantitative AI governance data. The study concludes that human verification against primary sources remains essential and that practitioners should treat numeric responses regarding AI infrastructure with extreme skepticism.
4. Dataset Limitations: The study highlights a structural gap in global AI ethics measurement infrastructure, noting that dedicated ethics indicators were absent from the ground-truth dataset for the majority of the evaluation window (2010–2022).

The paper explicitly refrains from claiming that geographic bias is absent; rather, it argues that its empirical manifestation is more complex than a simple "Global North advantage," shaped significantly by the scale and specificity of the verified values in the evaluation frame.