Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandic

This paper critiques the validity of current Large Language Model benchmarks for Icelandic, demonstrating that unverified synthetic or machine-translated data introduces severe flaws and arguing for the necessity of human-verified evaluation methods in low- and medium-resource language settings.

The Gist

Imagine you are a teacher trying to grade a new student's essay. But instead of reading the essay yourself, you ask a robot to translate the essay into your language, grade it, and then tell you how the student did.

This paper is essentially a report card on how we are currently testing AI (Large Language Models) in Icelandic, a language spoken by a relatively small number of people. The authors, who are experts in Icelandic and AI, are raising a huge red flag: Many of the tests we are using are broken, and the broken tests are giving us a false picture of how smart these AIs really are.

Here is the breakdown of their findings using some everyday analogies:

1. The "Copy-Paste" Problem (Machine Translation)

Currently, because creating tests from scratch is hard and expensive, many researchers take famous English tests (like a 5th-grade science quiz) and just run them through a machine translator to make them Icelandic.

• The Analogy: Imagine you want to test a student's knowledge of Icelandic history. Instead of writing a test in Icelandic, you take a test about American history, run it through Google Translate, and hand it to the student.
• The Result: The translation might be full of weird errors. A "turkey" (the bird) might get translated as "Turkey" (the country). A famous scientist's name might get changed to a random local name.
• The Paper's Finding: The authors found that many of these "translated" tests are so full of errors that they are useless. It's like giving a math test where the numbers are written in a language the student doesn't understand. If the AI gets the answer right, it might just be guessing because the question was nonsense, not because it actually understands Icelandic.

2. The "Fake News" Factory (Synthetic Data)

Some tests aren't just translated; they are generated entirely by other AIs. An AI is asked to "make up 1,000 questions about Iceland."

• The Analogy: This is like asking a student to write their own history textbook, and then using that textbook to grade them. The student might make up facts that sound plausible but are completely wrong.
• The Paper's Finding: When AIs generate these questions, they often hallucinate (make things up). They might create a question about a person who never existed or a fact that is scientifically impossible. The authors found that some of these AI-generated tests were almost 100% flawed.

3. The "Native Speaker" Gap

The most successful tests in the paper were the ones where actual humans who speak Icelandic wrote or checked the questions.

• The Analogy: This is like having a real Icelandic teacher write the test. They know the local culture, the slang, the correct grammar, and the history. They don't rely on a robot to guess what a sentence means.
• The Paper's Finding: Tests made by humans were much cleaner and more accurate. Tests made by machines (or translated by them) were full of "translationese" (awkward phrasing that no real human would ever say).

4. Why Does This Matter? (The "Garbage In, Garbage Out" Rule)

The authors argue that if we keep using these broken tests, we are lying to ourselves about how good AI is.

• The Analogy: Imagine you are training a dog. If you use a broken clicker that makes a noise every time the dog sits, even if the dog is just standing still, the dog will think it's doing something right. Eventually, the dog will be confused and won't actually learn to sit.
• The Real Consequence: If AI models are trained or evaluated on these broken tests, they might learn to "game the system." They might learn to spot the weird translation errors and answer based on those errors rather than actually understanding the language. This makes the AI look smarter than it is, but in the real world, it will fail at actual tasks.

The Main Takeaway

The paper asks a simple question: "Who is checking the people who make the tests?"

Right now, the answer is often "no one," or "a robot." The authors are calling for a change:

1. Stop blindly translating tests. If you translate a test, a human native speaker must check it to make sure it makes sense.
2. Involve real humans. Native speakers need to be part of the process to ensure the tests reflect real culture and language, not just robotic translations.
3. Quality over Quantity. It's better to have a small, perfect test than a huge, broken one.

In short: We can't trust the scoreboard if the referees are robots that don't speak the language. We need human experts to ensure the tests are fair and accurate before we declare an AI a genius.

Technical Summary

1. Problem Statement

The paper addresses a critical issue in Natural Language Processing (NLP): the validity of Large Language Model (LLM) benchmarks for low-to-medium resource languages, specifically Icelandic.

• The Core Issue: As LLMs advance, the creation of benchmarks has shifted from human-curated datasets to automated methods, including machine translation (MT) of existing English benchmarks and the generation of synthetic data via LLMs.
• The Risk: In low-resource settings, where native linguistic expertise is scarce, creators often rely on these automated methods without rigorous verification. This leads to benchmarks containing severe flaws (factual errors, translation artifacts, cultural misalignment, and grammatical hallucinations) that skew evaluation results.
• The Consequence: Models may appear to perform well on these flawed benchmarks by exploiting superficial heuristics or "translationese" rather than demonstrating genuine linguistic competence or cultural understanding. This creates a false sense of progress and risks reinforcing inaccuracies in future model training cycles.

2. Methodology

The authors conducted a comprehensive case study on Icelandic benchmarks using a quantitative error analysis approach.

• Scope of Analysis: The study reviewed two primary public leaderboards for Icelandic:
  1. Miðeind's Leaderboard: Maintained by an Icelandic language technology company.
  2. EuroEval (formerly ScandEval): A broader European project.
  • Datasets Analyzed: The authors sampled subsets of 12 distinct benchmarks, including WinoGrande-IS, ARC-Challenge (two versions), Belebele, WikiQA, MultiWikiQA, HellaSwag, MMLU, ScaLA, and others.
• Sampling Strategy:
  • For datasets >250 rows, 250 random examples were selected.
  • For smaller datasets, 10% of the data was sampled.
  • Specific splits (e.g., train sets) were used where available.
• Annotation Protocol:
  • Three authors (experts in Icelandic linguistics and NLP) manually annotated the samples.
  • Labeling Scheme:
    • IC (Incorrect/Severely Flawed): Wrong answers, severe flaws, or poor machine translation.
    • F (Flawed): Contains errors (spelling, grammar, vocabulary) but potentially solvable.
    • OK (Valid): A correct, valid example.
  • Reliability: Inter-Rater Reliability (IRR) was calculated using Krippendorff's alpha and weighted agreement coefficients.
• Comparative Approach: The study explicitly compared benchmarks created/translated by humans (and verified) against those generated via MT or LLMs without verification.

3. Key Contributions

• Systematic Audit of Icelandic Benchmarks: This is the first quantitative error analysis of the current state of LLM evaluation in Icelandic, revealing that a significant portion of widely used benchmarks are fundamentally flawed.
• Differentiation of Data Sources: The paper establishes a clear empirical distinction between human-authored/verified benchmarks and synthetic/MT-generated benchmarks, showing the latter are significantly more error-prone.
• Identification of Specific Failure Modes: The authors categorize specific types of errors, including:
  • Translation Artifacts: Incorrect terminology (e.g., translating "turkey" the bird as the country "Iceland" in a geopolitical context).
  • Cultural Misalignment: US-centric science questions (ARC-Challenge) that lack relevance or are poorly localized for Icelandic culture.
  • Hallucinated Linguistic Features: In gender-bias benchmarks, the generation of non-existent feminine forms for masculine occupation nouns in Icelandic.
  • Data Contamination: Datasets originally intended for training (e.g., NQiI) being repurposed as evaluation benchmarks without adaptation.
• Call for "Construct Validity": The paper argues for a "Benchmark-Validation-First Approach," emphasizing that benchmarks must measure what they intend to measure (construct validity) rather than just maximizing statistical power through volume.

4. Key Results

The quantitative analysis yielded stark differences between benchmark types:

• High Error Rates in MT/Synthetic Data:
  • HellaSwag-IS (MT): 0% of the sampled examples were labeled "OK" (Valid). All were severely flawed.
  • MMLU-IS (MT): Less than 10% of examples were valid; the rest were severely flawed.
  • MultiWikiQA-IS (LLM-generated): Less than 50% of examples were valid.
  • ARC-Challenge (EuroEval/MT): Only 20% of examples were valid; 37% were severely flawed.
• Superiority of Human-Verified Data:
  • ARC-Challenge (Miðeind/Human-localized): 55% valid, with only 7% severely flawed.
  • WinoGrande-IS (Human-translated): 0% severely flawed in the sample; high validity.
  • IcelandicQA (LLM-generated but Human-verified): Significantly better than unverified synthetic counterparts.
• Specific Examples of Failure:
  • In the ScaLA benchmark (linguistic acceptability), procedural speech insertions from the Icelandic parliament (e.g., "Speaker rings") were treated as grammatical errors or corrupted incorrectly, disrupting syntactic structure.
  • In GAMBIT+ (gender bias), the dataset hallucinated feminine forms for Icelandic occupation nouns that do not exist linguistically, rendering the evaluation invalid for Icelandic.
  • In NQiI, context and questions were mismatched (e.g., Kim Jong-Un context paired with a question about Bolivia's capital).

5. Significance and Recommendations

The paper concludes that the current state of benchmarking for low-resource languages is precarious and calls for an immediate shift in methodology.

• Avoid Unverified MT: Machine-translated benchmarks should not be used for evaluation unless the output is rigorously verified and corrected by native speakers.
• Native Speaker Involvement: Benchmark creation must involve native speakers of the target language to ensure cultural alignment, correct terminology, and grammatical validity.
• Quality over Quantity: Small, carefully curated, human-verified datasets are often better indicators of model performance than large, synthetic, or unverified sets.
• Continuous Scrutiny: Benchmark construction is not a one-time task. It requires continuous auditing, documentation, and critical evaluation ("Who benchmarks the benchmarks?").
• Future Direction: The authors advocate for participatory design involving the language community to prevent models from optimizing for "translation artifacts" or superficial patterns, ensuring that evaluation metrics genuinely reflect the linguistic and cultural specificities of the language.

In summary, the paper serves as a warning that automated benchmark generation in low-resource languages often produces invalid data that misleads the research community. It demands a return to human-centric, culturally grounded evaluation practices to ensure the true capabilities of LLMs are measured accurately.