💬 NLP

GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek

This paper introduces GreekMMLU, a comprehensive, native-sourced benchmark comprising over 21,000 multiple-choice questions across 45 subjects, designed to address the lack of authentic evaluation tools for Greek language models and revealing significant performance gaps between frontier, open-weight, and Greek-adapted models.

Original authors: Yang Zhang, Mersin Konomi, Christos Xypolopoulos, Konstantinos Divriotis, Konstantinos Skianis, Giannis Nikolentzos, Giorgos Stamou, Guokan Shang, Michalis Vazirgiannis

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yang Zhang, Mersin Konomi, Christos Xypolopoulos, Konstantinos Divriotis, Konstantinos Skianis, Giannis Nikolentzos, Giorgos Stamou, Guokan Shang, Michalis Vazirgiannis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant library of knowledge, and inside it are thousands of books written in English. Now, imagine you want to test how smart a new AI robot is. You give it a quiz based on those English books. If the robot gets a high score, you think, "Wow, this robot is a genius!"

But here's the problem: What if you want to test that same robot on Greek culture, history, and language?

If you just take the English quiz and use a machine translator to turn it into Greek, the robot might still get a high score. But it's not because it understands Greek; it's just because it's good at guessing based on the English version it already knows. It's like asking someone to recite a poem in a language they don't speak, but they memorized the English translation and just read the words backwards. They sound like they know the language, but they don't really get the meaning, the jokes, or the cultural heart of it.

This is exactly the problem the authors of this paper wanted to fix.

The Problem: The "Translated" Trap

For a long time, to test AI on Greek, researchers just took English tests and translated them. This is like trying to judge a chef's ability to cook authentic Greek food by giving them a recipe translated from French. The ingredients might be there, but the flavor is wrong. The AI misses the nuance, the specific way Greeks think about their history, and the unique structure of their language.

The Solution: GreekMMLU (The "Native" Quiz)

The team created GreekMMLU, which is like building a brand-new, authentic Greek test from scratch.

Where did the questions come from? Instead of translating English, they went straight to the source: real Greek school exams, university tests, professional licensing exams (like for doctors or lawyers), and even driving tests.
What's inside? It's a massive collection of 21,805 questions covering 45 different subjects.
- STEM: Math, Physics, Biology.
- Humanities: Greek Mythology, History, Literature.
- Social Sciences: Economics, Law, Politics.
- Greek-Specific: Things that only a Greek speaker would know, like the rules of Greek driving or the details of Greek Orthodox traditions.
The "Secret Sauce": They split the questions into two piles. One pile is public (so anyone can test their AI), and the other is a "secret" pile kept in a vault. This prevents AI companies from accidentally memorizing the test answers before the test happens (a problem called "data contamination").

The Big Experiment: Who Passed the Test?

The authors took over 80 different AI models (both free, open-source ones and expensive, closed-source ones) and gave them this Greek quiz. Here is what they found, using some simple analogies:

The "Big Brains" vs. The "Small Brains":
- The biggest, most powerful AI models (like the ones from Google and OpenAI) did very well. They are like students who have read every book in the library.
- The smaller, open-source models struggled. Some of them scored barely better than random guessing. It's like giving a high school physics test to someone who hasn't finished middle school yet.
The "Specialist" vs. The "Generalist":
- Some AIs are "Generalists"—they know a little bit about everything in many languages.
- Some are "Specialists"—they were specifically trained to speak and understand Greek better.
- The Result: The Specialists won! Models that were specifically tuned for Greek (like Llama-Krikri or Meltemi) performed much better than general multilingual models of the same size. It's like hiring a local tour guide (Specialist) versus a guide who just learned the language from a phrasebook (Generalist). The local guide knows the shortcuts and the hidden gems.
The "Instruction" Boost:
- They found that if you "teach" the AI how to take the test (by giving it examples of how to answer), it gets smarter. This is called "instruction tuning." It's like giving a student a practice exam before the real one; they do much better because they understand the format.

Why Does This Matter?

This paper is a wake-up call. It shows that just because an AI speaks Greek, doesn't mean it understands Greek.

For Researchers: It gives them a fair, honest ruler to measure how good an AI really is at Greek.
For the Future: It encourages companies to stop just translating English tests and start building real, native-language tests for every culture. If we want AI to be truly helpful to Greeks, it needs to understand Greek culture, not just Greek words.

In short: The authors built a real, authentic Greek exam to stop AI from "faking it." They found that while the biggest AIs are smart, the ones specifically trained on Greek culture are the true champions of the language.

1. Problem Statement

Despite the inclusion of Greek in the multilingual training corpora of many Large Language Models (LLMs), reliable evaluation benchmarks for the language are scarce. Existing evaluations suffer from three critical limitations:

Reliance on Machine Translation: Most current benchmarks are machine-translated from English (e.g., MMLU). This introduces "translationese," semantic drift, altered difficulty calibration, and the preservation of English cultural priors, failing to capture the nuances of authentic Greek usage.
Lack of Cultural Grounding: Greek has unique linguistic features (rich morphology, flexible word order, complex negation) and knowledge domains (history, law, civics) tightly coupled to national curricula and cultural context. Translated benchmarks cannot accurately assess localized knowledge.
Data Contamination: Standard web-crawled datasets often overlap with model training data, making it difficult to distinguish between memorization and genuine reasoning.

2. Methodology: The GreekMMLU Dataset

The authors introduce GreekMMLU, the first large-scale, native-sourced benchmark designed to evaluate massive multitask language understanding in Greek.

Data Sourcing:
- Source: All 21,805 questions are sourced exclusively from authentic Greek academic, professional, and governmental examinations (e.g., university exams, driving tests, civil service recruitment, medical licensing).
- Format: Multiple-Choice Questions (MCQs) with 2–4 options and one correct answer.
- Ethical Compliance: Data was collected from open-access or educational reuse licenses to ensure legal compliance and minimize data contamination risks.
Taxonomy and Structure:
- Subjects: Organized into 45 subject areas across four high-level categories: STEM, Humanities, Social Sciences, and Other.
- Greek-Specific Subset: Includes 8 specialized domains requiring localized cultural knowledge (e.g., Greek History, Mythology, Literature, Traditions, Modern Greek Language).
- Difficulty Levels: Annotated with educational levels ranging from Primary School to Professional/University examinations.
Data Curation Pipeline:
- Extraction: Used PyMuPDF4LLM for structure-aware text extraction from PDFs and Tesseract OCR for scanned legacy documents.
- Correction: An LLM-assisted phase (using Claude 3.5) corrected extraction artifacts while preserving semantic intent.
- Validation: A team of five native Greek-speaking experts (graduate/professor level) manually reviewed the dataset, filtering out personal information and correcting errors. The estimated noise level is ~2%.
Dataset Split:
- Public Subset: 16,857 questions (released for research).
- Private Subset: 4,948 questions (reserved for a private leaderboard to prevent data contamination and ensure robust evaluation).

3. Key Contributions

Native-Sourced Benchmark: The creation of GreekMMLU, a fully native dataset that avoids the pitfalls of machine translation, providing a true measure of Greek language understanding.
Comprehensive Evaluation: A large-scale evaluation of 80+ open- and closed-source LLMs, including general-purpose models (Llama, Qwen, Gemma), Greek-adapted models (Meltemi, Krikri, Plutus), and frontier closed-source models (GPT-4o, Gemini).
Systematic Analysis: A deep dive into factors influencing performance, including model scale, instruction tuning, prompting strategies (zero-shot vs. five-shot), and domain-specific capabilities.
Public Resources: Release of the dataset, code, and an official leaderboard to foster future research in low-resource European languages.

4. Key Results and Findings

The evaluation of over 80 models yielded several significant insights:

Performance Gaps:
- Closed vs. Open: Frontier closed-source models significantly outperform open-weight alternatives. Gemini 3 Flash achieved the highest average accuracy (93.16%), followed by GPT-5.2 (87.75%) and GPT-4o (86.81%).
- Open-Weight Limits: The best open-weight models (e.g., Llama-3.3-70B-Instruct, Qwen2.5-72B-Instruct) peaked around 79.5%–79.7%, leaving a substantial gap (~10–13%) compared to the best closed models.
Impact of Adaptation:
- Greek-Adapted Models: Models explicitly trained on Greek data (e.g., Llama-Krikri, Meltemi, Plutus) consistently outperformed general multilingual baselines of similar size, particularly on Greek-specific subjects.
- Instruction Tuning: Instruction-tuned variants consistently outperformed their base counterparts across all model families, highlighting the importance of alignment with the MCQ format and Greek prompt conventions.
Scaling Laws:
- There is a strong positive correlation between model parameter count and performance. Models below ~~2B parameters performed near the random baseline (~~30%), while performance improved steadily with scale.
Domain and Cultural Challenges:
- Greek-Specific Subjects: General multilingual models struggled significantly with culturally grounded tasks (History, Mythology). Greek-centric models showed marked improvements in these areas.
- Educational Levels: Accuracy was generally higher on primary/secondary school questions and lower on university/professional tasks, reflecting increased complexity and specialized knowledge requirements.
Calibration:
- Greek-specialized models (e.g., Llama-Krikri) demonstrated superior calibration (high correlation between confidence and accuracy, $r \approx 0.93$ ) compared to generic multilingual models, which often exhibited miscalibration.

5. Significance

Paradigm Shift in Evaluation: GreekMMLU demonstrates that translated benchmarks are insufficient for evaluating low-resource languages with rich cultural contexts. It sets a new standard for "native-sourced" evaluation.
Guidance for Model Development: The results indicate that while scaling helps, targeted adaptation (Greek-specific pretraining and instruction tuning) is crucial for achieving state-of-the-art performance in Greek, especially for culturally specific domains.
Community Resource: By providing a contamination-resistant private leaderboard and a high-quality public dataset, the authors enable fair, reproducible, and rigorous research into Greek language capabilities, encouraging the development of more authentic and culturally grounded AI systems.

In conclusion, GreekMMLU reveals that while current frontier models are powerful, there is a distinct "Greek gap" in open-weight models that can only be bridged through dedicated, native-language training and adaptation.