💬 NLP

Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translation

This paper introduces a culturally adapted Estonian translation of the WinoGrande benchmark, demonstrating that human-translated datasets yield more reliable evaluations of large language model reasoning than machine-translated ones, while highlighting the limited effectiveness of prompt engineering in overcoming translation challenges.

Original authors: Marii Ojastu, Hele-Andra Kuulmets, Aleksei Dorkin, Marika Borovikova, Dage Särg, Kairit Sirts

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Marii Ojastu, Hele-Andra Kuulmets, Aleksei Dorkin, Marika Borovikova, Dage Särg, Kairit Sirts

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to test how smart a new student is. You give them a riddle: "The trophy doesn't fit in the suitcase because it is too big." The student has to guess: Is "it" the trophy or the suitcase?

This is the WinoGrande test. It's a famous puzzle game used to see if AI computers have "common sense" (the ability to understand the world like a human does). But until now, this game only existed in English.

This paper is about a team in Estonia who tried to bring this game to the Estonian language to test if AI works just as well there. Here is the story of what they found, told simply.

1. The Challenge: Translating a Riddle

Translating a riddle isn't like translating a menu. If you translate a menu, "burger" is just "burger." But in a riddle, every word matters. If you change the grammar or the cultural context, the answer might change, or the riddle might become broken.

The team had two ways to translate the 1,767 riddles into Estonian:

Method A (The Human Expert): A professional translator carefully rewrote every sentence, making sure the cultural references made sense to an Estonian (e.g., changing a "desert cactus" to a "forest juniper" because cacti don't grow in Estonia).
Method B (The Robot Translator): They asked a powerful AI (GPT) to translate the riddles for them, first with a simple instruction, and then with a very long, detailed instruction manual.

2. The Experiment: Testing the AI Students

Once they had the three versions of the test (Original English, Human-Translated Estonian, and Robot-Translated Estonian), they asked various AI models to take the test.

The Results:

The English Test: The AIs did great. They were like top students acing a test in their native language.
The Human-Translated Test: The AIs did almost as well as in English. The human translator had fixed tricky grammar issues and cultural mismatches, so the riddles were still fair and solvable.
The Robot-Translated Test: The AIs struggled significantly. Their scores dropped.

3. Why Did the Robot Translation Fail?

The researchers looked closely at the robot-translated riddles and found three main "bugs" in the system:

The "Grammar Trap": In English, the riddles are balanced. In the robot version, the grammar got messy. Sometimes, the sentence structure gave away the answer just by looking at the verb ending, rather than using logic. It was like a math test where the answer was written in the question mark.
The "Lost Meaning" (The Twist): Sometimes, the robot changed the story entirely.
- Original: "The cat bit the dog."
- Robot Translation: "The dog bit the cat."
- Result: The AI got the answer wrong, not because it was stupid, but because the question it was reading was different from the one the humans intended.
The "Cultural Disconnect": The robot didn't understand that some things just don't exist in Estonia. It kept references that felt weird or confusing to a local speaker, making the riddles harder to solve.

4. The "Prompt Engineering" Experiment

The team thought, "Maybe if we give the robot a better set of instructions (a 'prompt'), it will do a better job." They wrote a very detailed manual for the robot, explaining exactly how to handle Estonian grammar and culture.

The Verdict: It helped a little bit, but not enough. The robot still couldn't match the quality of the human translator. It's like giving a robot a perfect recipe book but asking it to bake a cake without ever having tasted one; it can follow the steps, but it can't "feel" the texture.

5. The Big Lesson

The paper concludes with a very important message for the future of AI:

You cannot just use a robot to translate a test for another robot.

If you want to know if an AI is truly smart in a new language, you need a human expert to build the test. If you use a robot to translate the test, you end up testing the robot's ability to translate, not its ability to think.

The Analogy:
Imagine you want to test if a driver can drive in the rain.

Human Translation: You take the driver out into a real rainstorm.
Machine Translation: You show the driver a video of a rainstorm that was filmed in a studio with fog machines.
The Result: The driver might pass the video test but crash in the real rain.

The Estonian team proved that for AI to be truly tested, we need human hands to hold the steering wheel of the translation process.

1. Problem Statement

The WinoGrande dataset is a standard benchmark for evaluating commonsense reasoning in Large Language Models (LLMs) via pronoun disambiguation. However, it is exclusively in English, limiting the evaluation of multilingual capabilities.

The Challenge: Creating benchmarks for low-resource or morphologically complex languages (like Estonian) often relies on Machine Translation (MT).
The Gap: MT-generated datasets often differ significantly from human-annotated ones due to translation artifacts, loss of cultural context, and grammatical inconsistencies. These issues can lead to models solving tasks based on linguistic cues (e.g., grammar) rather than reasoning, or failing entirely due to semantic distortion.
Specific Difficulty: WinoGrande is particularly hard to translate because it requires strict adherence to lexical overlap (for "twin" sentences), correct grammatical cases (crucial in agglutinative languages like Estonian), and cultural appropriateness to maintain the commonsense logic.

2. Methodology

The authors created a localized Estonian version of the WinoGrande test set (1,767 instances) and compared it against two machine-translated versions.

A. Human Translation and Adaptation (The Gold Standard)

Process: Translated by a translation studies master's student and revised by a professional translator.
Localization & Cultural Adaptation: 53 samples (3%) were culturally adapted (e.g., replacing "desert/cactus" with "island/juniper" to fit Estonian geography; changing "Mortal Kombat" to a locally known game).
Error Correction: 89 samples (5%) were identified as ambiguous, incorrectly labeled, or trivial in the original English dataset. These were manually corrected to ensure the task required genuine reasoning.
Linguistic Constraints: Special care was taken to maintain 70% lexical overlap in "twin" sentences and ensure answer options matched the grammatical case of the antecedent (a critical feature in Estonian's 14-case system).
Validation: Inter-annotator agreement was high (Cohen's $\kappa$ = 0.816), with human accuracy on the dataset at 95% and 92% for two independent annotators.

B. Machine Translation Experiments

Two versions of the dataset were generated using OpenAI models (GPT-4o and GPT-4.1):

Simple Prompt: A generic zero-shot translation prompt.
Detailed Prompt: A complex prompt engineered to address specific issues found in the simple version (e.g., enforcing 70% lexical overlap, correcting grammatical cases, providing a list of Estonian names with gender/inflection for localization).

C. Model Evaluation

The authors evaluated a diverse range of models on four dataset versions:

Original English (EN)
Human-translated Estonian (HT)
Machine-translated Estonian (Simple Prompt)
Machine-translated Estonian (Detailed Prompt)

Models Tested:

Proprietary: Gemini 2.5 Pro/Flash, Claude Sonnet 4.5, GPT-4.1, GPT-5.
Open Source (Large/Medium): Llama 3.3-70B, Llama 3.1-405B, Gemma 3-27B, Qwen 2.5-72B.
Open Source (Small): Llama 3.1-8B, Apertus-8B, EuroLLM-9B.

3. Key Results

A. Performance Comparison

Human vs. Original: Proprietary models performed similarly on English and Human-translated Estonian (avg. difference 0.4%). Open models performed slightly worse on Estonian (avg. -5.5%), except for Gemma 3-27B which improved slightly.
Human vs. Machine Translation: All models performed significantly worse on machine-translated data compared to human-translated data.
- Proprietary models dropped by ~6-7% on MT data.
- Open models dropped by ~6-10%.
Prompt Engineering Impact: The "Detailed Prompt" offered limited improvement over the "Simple Prompt." The performance gap between the two MT versions was negligible, suggesting that current LLMs cannot reliably translate WinoGrande even with sophisticated prompting.

B. Subset Analysis

Cultural Adaptation: Models performed ~6% better on culturally adapted samples in the HT dataset compared to non-adapted samples. However, the authors note this is likely due to the removal of cultural barriers rather than an inherent increase in reasoning capability.
Error Correction: The subset of 89 corrected samples showed a massive performance gap. Models scored near chance level on the original English errors but significantly higher on the corrected Estonian versions. This highlights that original dataset noise propagates and amplifies in MT.
Semantic Shift: Manual analysis revealed that 15.2% of samples (Simple Prompt) and 11.8% (Detailed Prompt) had their meaning altered by MT. Models performed significantly worse (4.7% to 7.2% drop) on these "altered meaning" samples compared to "retained meaning" samples.

C. Specific Linguistic Failures in MT

Grammatical Cues: MT often failed to align grammatical cases, allowing models to solve tasks based on syntax (e.g., singular/plural agreement) rather than semantics.
Lexical Divergence: Twin sentences in MT often lost the required 70% lexical overlap, breaking the benchmark's structural integrity.
Ambiguity: MT introduced new ambiguities or removed necessary disambiguation cues present in the original.

4. Key Contributions

Estonian WinoGrande Dataset: The first high-quality, culturally adapted, and error-corrected Estonian benchmark for commonsense reasoning, available for public use.
Empirical Evidence on MT Limitations: Demonstrates that current state-of-the-art LLMs cannot reliably translate complex reasoning benchmarks like WinoGrande, even with advanced prompt engineering.
Importance of Human Expertise: Proves that involving translation specialists is critical for dataset creation. Automated translation introduces artifacts that obscure the true reasoning capabilities of models.
Dataset Quality Audit: Identified and corrected 5% of the original English dataset as ambiguous or erroneous, providing a cleaner baseline for future research.

5. Significance and Conclusion

The paper concludes that while proprietary models show strong reasoning capabilities in Estonian when provided with high-quality human data, machine translation is currently infeasible for creating reliable reasoning benchmarks.

Reliability: Evaluations based on machine-translated datasets may yield misleading results, as models might be solving (or failing) due to translation errors rather than reasoning ability.
Prompt Engineering Limits: Simply refining prompts is insufficient to overcome the linguistic and cultural complexities of translating reasoning tasks.
Future Direction: The study emphasizes the necessity of human-in-the-loop processes for benchmark creation in low-resource languages to ensure that evaluations of LLMs are interpretable and truly reflective of language competency.