Technical Summary: From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch

Problem Statement

Large Language Models (LLMs) are increasingly deployed in governmental settings to improve public communication and administrative efficiency. However, existing evaluation frameworks suffer from two critical gaps when applied to non-English, public-sector contexts:

1. Linguistic Limitation: The vast majority of benchmarks are English-centric. Performance on English benchmarks does not reliably transfer to other languages, and existing Dutch benchmarks often rely on machine-translated datasets or address only isolated dimensions (e.g., knowledge or reasoning) without considering governmental values.
2. Contextual Misalignment: Current leaderboards (e.g., MMLU) provide raw technical scores that are unintelligible to non-technical stakeholders (policymakers, civil servants). Furthermore, no existing suite holistically evaluates models against the specific ethical and operational values required for public administration, such as sustainability, social bias, and training data transparency.

This lack of a unified, value-aligned evaluation framework creates significant risks for governments, including the deployment of opaque, biased, or factually unreliable systems, as evidenced by historical failures in automated decision-making in Canada, the Netherlands, and Poland.

Methodology

The authors developed the "Grip on LLMs" framework through a participatory process involving the City of Amsterdam and domain experts. The methodology consists of three phases:

1. Value Identification and Operationalization

The evaluation dimensions were not imposed by the researchers but derived from three sources:

• Advisory Board: Nine internal experts (innovation, AI policy, sustainability, diversity) identified and prioritized values. "Inclusion" and "Factuality" were rated highest.
• Practitioner Interviews: 18 interviews with data scientists and policymakers revealed that raw benchmark scores are uninterpretable and that selection processes require tools legible to non-experts.
• User Survey: 429 responses from internal chatbot users confirmed the priority of quality and factuality, though with less emphasis on sustainability than the advisory board.

These inputs were distilled into six evaluation dimensions:

1. Factuality: Accuracy in answering objective questions.
2. Honesty: The ability to acknowledge limitations rather than hallucinating.
3. Social Bias: Systematic preferences toward demographic groups.
4. Sustainability: Energy consumption (carbon footprint).
5. Cost: Financial cost per 1,000 prompts.
6. Training Data Transparency: Openness regarding corpus composition and licensing.

Two specific use cases were also defined: Text Simplification (making complex government text accessible) and Summarization (processing internal documents).

2. Benchmark Selection and Construction

To evaluate over 30 models, the authors curated a suite of benchmarks, prioritizing Dutch-native or human-verified translated datasets over machine translations:

• Factuality: Used tinyBenchmarks (MMLU, ARC-Challenge, TruthfulQA) with Dutch translations. Scores were estimated using a GP-IRT estimator to account for item difficulty.
• Honesty: A new benchmark, HONESTCITYBENCH, was developed containing 530 Dutch prompts across five limitation categories (e.g., outdated info, insufficient data). Evaluation used an LLM-as-a-judge protocol validated against human annotations.
• Social Bias: Combined Dutch BBQ (ambiguous QA scenarios) and BZK Social Bias (simulated hiring decisions) to measure bias across age, disability, gender, and country of origin.
• Use Cases: Simplification was measured on the Dutch Municipal Simplification benchmark and INT Duidelijke Taal using SARI. Summarization used machine-translated CNN/Daily Mail and XSum datasets, measured via BERTScore.
• Sustainability & Cost: Energy was measured using CodeCarbon on H100 GPUs for open-weight models. Cost was calculated based on token pricing (API) or GPU hourly rates (self-hosted).

3. Evaluation Protocol

The study evaluated 31 instruction-tuned LLMs, including European initiatives (EuroLLM, Apertus), Dutch-specific models (GEITje, Fietje), and major commercial/multilingual models (GPT-4o/5, Mistral, Qwen, Llama). All experiments were run under identical conditions (temperature 0, single H100 GPU) to ensure reproducibility.

Key Contributions

1. Value-Driven Framework: A systematic evaluation suite that translates public administration values into measurable technical dimensions, specifically tailored for the Dutch language.
2. Large-Scale Evaluation: The first comprehensive benchmarking of over 30 models (multilingual and Dutch-specific) across six dimensions relevant to government use.
3. New Benchmarks: The introduction of HONESTCITYBENCH for Dutch honesty evaluation and the integration of bias and sustainability metrics into a unified suite.
4. User-Centric Tooling: The release of a publicly accessible, user-friendly LLM Overview (leaderboard) designed for non-technical stakeholders, translating complex metrics into interpretable categorical ratings (1–5 scales) and visualizations.

Results

The evaluation yielded several critical findings regarding the trade-offs inherent in model selection:

• No Dominant Model: No single model excels across all dimensions. High performance in one area often necessitates trade-offs in others.
• Quality vs. Efficiency: Higher quality (factuality, honesty, summarization) consistently correlates with higher financial costs and greater energy consumption.
• Factuality vs. Honesty: These are distinct properties. The study found that high factuality does not imply high honesty. Notably, the most recent high-capability models (e.g., GPT-5, Mistral Large 3) achieved top factuality scores but scored lowest on honesty, often providing confident but incorrect answers.
• Bias Independence: Social bias scores are largely independent of both model quality and cost. Spending more or using a "smarter" model does not guarantee a less biased outcome.
• Open vs. Closed Models: European open initiatives (e.g., EuroLLM, Apertus) offer full transparency and competitive efficiency but generally score lower on quality metrics compared to top-tier commercial models. However, they provide necessary transparency for public sector compliance.
• Dutch-Specific Models: Models like Fietje and GEITje showed limited world knowledge (low factuality) but low bias, suggesting they are niche tools rather than general-purpose replacements.

Significance and Claims

The paper claims that responsible AI deployment in the public sector requires moving beyond single-metric optimization. The authors argue that:

• Trade-offs are Unavoidable: Policymakers must explicitly choose which dimensions to prioritize (e.g., accepting higher costs for better transparency or lower bias) rather than seeking a "best" model.
• Interpretability is Crucial: Technical scores must be translated into value-aligned categories to enable decision-making by non-experts.
• Honesty is a Critical Safety Feature: In government contexts, a model that admits ignorance is often safer than one that hallucinates with high confidence. The decoupling of factuality and honesty is a key finding for risk assessment.

The work serves as a practical starting point for Dutch public organizations to select LLMs, emphasizing that benchmark scores are proxies for values, not definitions of them, and should be used as part of a broader, task-specific evaluation process.