Evaluating Monolingual and Multilingual Large Language Models for Greek Question Answering: The DemosQA Benchmark

1. Problem Statement

Despite the rapid advancement of Large Language Models (LLMs), research remains heavily biased toward high-resource languages like English. Multilingual models often suffer from:

• Training Data Imbalance: Under-resourced languages (like Greek) are marginalized in training corpora.
• Cultural Misrepresentation: Models may fail to capture social, cultural, and historical nuances specific to these languages.
• Evaluation Gaps: There is a scarcity of comprehensive, human-curated benchmarks for Greek Question Answering (QA), particularly those reflecting real-world community discourse.
• Hardware Constraints: Evaluating LLMs typically requires high-end GPUs, limiting reproducibility for researchers with limited resources.

The study specifically addresses the lack of systematic evaluation for Greek QA and the uncertainty regarding whether open-weight monolingual models can compete with proprietary multilingual models in this domain.

2. Methodology

A. Dataset Construction: DemosQA

The authors introduced DemosQA, a novel dataset designed to capture the "Greek social and cultural zeitgeist."

• Source: Data was scraped from the r/greece subreddit on Reddit.
• Curation Process:
  1. Collection: Used the PRAW Python library to collect posts categorized as "questions" (ερωτήσεις).
  2. Filtering: Retained posts with at least 5 upvotes and 5 answers. Removed duplicates, image-only posts, and adult content.
  3. Human Curation: Researchers manually reviewed the top 10 upvoted answers to remove hate speech, sarcasm, and irrelevant content.
  4. Formatting: Questions were constructed by concatenating titles and main text. Four candidate answers were selected manually (rather than relying solely on upvote counts) to ensure relevance, with the highest-upvoted answer designated as the ground truth.
  5. Bias Mitigation: Answer options were shuffled to prevent LLMs from selecting the first option by default.
• Final Stats: 600 curated QA pairs with diverse domains (everyday life, history, politics, science) and longer, more complex text compared to existing datasets.

B. Evaluation Framework

The authors developed a memory-efficient evaluation framework to run on commodity hardware (Intel i5, 64GB RAM, 12GB VRAM GPU).

• Quantization: Utilized 4-bit model quantization (via bitsandbytes) to reduce VRAM requirements (e.g., a 7B model requires ~3.5GB VRAM in 4-bit vs. ~14GB in 16-bit) with minimal accuracy loss.
• Models Evaluated: 11 LLMs (7B–12B parameters), including:
  • Monolingual (Greek-adapted): Llama Krikri 8B, Meltemi 7B.
  • Multilingual: Gemma 2 9B, Llama 3.1 8B, Mistral NeMo 12B, EuroLLM 9B, Aya Expanse 8B, Teuken 7B, etc.
  • Proprietary Baseline: GPT-4o mini (accessed via API).
• Prompting Strategies: Three distinct strategies were tested:
  1. Instruction Prompt: Direct instruction to select the best answer.
  2. Role Prompt: Assigning a specific role (e.g., "You are a Greek language model") before the instruction.
  3. Zero-Shot Chain-of-Thought (CoT): Instructing the model to reason step-by-step before answering.
• Metrics: Macro-accuracy on multiple-choice QA tasks.

C. Comparative Datasets

DemosQA was evaluated alongside five existing Greek QA datasets:

• Greek Medical MCQA (Medical exams)
• Greek Truthful QA (Misconception checking)
• BELEBELE (Reading comprehension, translated from English)
• INCLUDE (Educational exams)
• Greek ASEP MCQA (Civil service exams)

3. Key Contributions

1. DemosQA Dataset: The first community-reviewed, human-curated Greek QA dataset derived from social media, offering a benchmark for realistic, culturally grounded Greek text.
2. Efficient Evaluation Framework: A reproducible pipeline leveraging 4-bit quantization, enabling LLM evaluation on consumer-grade hardware without significant accuracy degradation.
3. Comprehensive Benchmarking: The first extensive comparison of 11 monolingual and multilingual LLMs across 6 diverse Greek QA datasets using three prompting strategies.
4. Open Resources: Public release of code and data to facilitate reproducibility and future research in Greek NLP.

4. Experimental Results

• Top Performers:
  • GPT-4o mini achieved the highest overall accuracy, particularly excelling in specialized domains (Medical, Civil Service, Education).
  • Llama Krikri 8B (Greek monolingual) and Gemma 2 9B (Multilingual) were the top open-weight performers.
  • Llama Krikri 8B matched or exceeded GPT-4o mini on DemosQA and Greek Truthful QA, demonstrating that monolingual adaptation is highly effective for general and community-driven tasks.
• Prompting Strategy Impact:
  • GPT-4o mini: Performed best with simple Instruction prompts.
  • Open-Weight Models: Showed significant improvement with Role prompts.
  • Chain-of-Thought (CoT): Generally improved performance for models with strong reasoning capabilities but degraded performance for most open-weight models, likely due to hallucinations during the reasoning process.
• Domain Specificity: Open-weight models struggled significantly in specialized domains (Medical, ASEP, INCLUDE) compared to GPT-4o mini, suggesting that prompt engineering cannot fully compensate for a lack of specialized domain knowledge.
• Multilingual vs. Monolingual: Monolingual Greek models (Llama Krikri, Meltemi) generally outperformed multilingual counterparts on Greek-specific tasks, validating the need for language-specific adaptation.

5. Significance and Conclusion

This study bridges a critical gap in Greek NLP by providing a robust benchmark (DemosQA) and demonstrating that open-weight monolingual models can rival proprietary models on culturally relevant, community-driven tasks.

• Cultural Relevance: The results highlight that models trained specifically on Greek data capture social nuances better than general multilingual models.
• Accessibility: The memory-efficient framework democratizes LLM evaluation, allowing researchers with limited hardware to contribute to the field.
• Future Directions: The authors advocate for training Greek LLMs from scratch on diverse corpora (including dialects and polytonic texts) and expanding evaluation to other NLP tasks like summarization.

The paper concludes that while proprietary models lead in specialized knowledge, open-weight monolingual models are highly competitive for general and social QA, provided the evaluation framework accounts for their specific strengths (e.g., role prompting).