Technical Summary: OMGEval

Problem Statement

While Large Language Models (LLMs) have demonstrated remarkable capabilities, current evaluation benchmarks are predominantly English-centric. Existing multilingual benchmarks largely focus on discriminative tasks (e.g., classification, paraphrase detection) rather than the open-ended generative tasks that define modern LLMs. Furthermore, direct translation of English datasets into other languages often fails to account for cultural nuances, leading to "English-centric bias" where questions reference culturally specific concepts (e.g., Thanksgiving, US states) that are irrelevant or confusing in other cultural contexts. This gap hinders the assessment of LLMs' true multilingual and cross-cultural capabilities.

Methodology

The authors introduce OMGEval (Open Multilingual Generative Evaluation), a benchmark designed to assess LLMs across five languages: Chinese (Zh), Russian (Ru), French (Fr), Spanish (Es), and Arabic (Ar). The construction and evaluation process involves three key stages:

1. Data Construction and Localization:

   • Base Data: The authors curated 805 open-ended questions from the English-language AlpacaEval dataset.
   • Localization: Unlike simple translation, OMGEval employs a rigorous localization process. Human annotators adapt culturally specific elements (names of people, places, festivals, foods, and historical periods) to fit the target culture while maintaining the original intent and logical structure. For example, a question about "Thanksgiving and turkey" is localized to "Dragon Boat Festival and Zongzi" for Chinese.
   • Verification: Each question undergoes manual verification by linguistics experts to ensure grammatical correctness, fluency, and logical coherence.
   • Categories: The dataset covers nine capability types: General Knowledge, Professional Knowledge, Generation and Creation, Language Comprehension, Code Skills, Logical Reasoning, Maths Competence, Chit Chat, and Harmlessness.

2. Evaluation Protocol:

   • Adjudicator: Following the AlpacaEval framework, GPT-4 is used as the automated adjudicator to score model outputs.
   • Metric: The evaluation compares a target model's output against a baseline model (initially GPT-text-davinci-003, later switched to GPT-3.5-turbo due to deprecation) to calculate a win rate.
   • Human Correlation: The authors validated the GPT-4 adjudicator against human annotators for Chinese and Spanish subsets, reporting high correlation (F1 scores of 0.88 and 0.91, respectively).

3. Experimental Setup:

   • The benchmark was used to evaluate ten representative models, including proprietary models (GPT-4, GPT-3.5-turbo) and open-source multilingual models (BLOOMZ, PolyLM, Okapi, Phoenix, Guanaco, Chimera).

Key Contributions

• First Open Multilingual Generative Benchmark: OMGEval is presented as the first open-source test set specifically designed to evaluate the generative capabilities of LLMs across multiple languages with a focus on cultural compatibility.
• Cultural Localization Framework: The paper provides a methodology for transforming English-centric benchmarks into culturally relevant datasets, ensuring that questions reflect local traditions, history, and social norms rather than mere linguistic translation.
• Comprehensive Dataset: The release includes 804 rigorously verified questions per language, covering a diverse range of cognitive and generative tasks.

Results

The evaluation of representative models on OMGEval reveals significant disparities in performance:

• Proprietary vs. Open-Source: GPT-4 is the only model to surpass a 50% average win rate (55.52%), demonstrating superior multilingual and cultural understanding. GPT-3.5-turbo serves as the baseline with a 50% win rate by design.
• Open-Source Performance: Open-source multilingual models (e.g., Guanaco-13b, Chimera, Phoenix) exhibit significantly lower win rates, generally ranging between 1% and 23% on the full test set.
• Localization Impact: While the ranking of models remained consistent between the full test set and the Localization Subset (questions heavily dependent on cultural knowledge), open-source models showed a slight decline in performance on the localization subset, highlighting their specific weakness in processing cultural nuances.
• Error Analysis: Case studies (e.g., questions regarding Zhang Yimou's debut film) show that open-source models frequently produce factual hallucinations or incorrect cultural references, whereas GPT-4 and GPT-3.5-turbo provide accurate answers.

Significance and Claims

The authors claim that OMGEval provides a valuable reference for the community to better understand and improve the multilingual capabilities of LLMs. The results underscore a critical gap in the field: current open-source models struggle significantly with cultural nuances and localization, indicating a broader issue of cultural bias in model training.

The paper modestly concludes that while OMGEval currently covers only five languages, the proposed methodology is extensible. The primary goal is to encourage the community to address cultural biases and enhance the global applicability of LLMs, moving beyond English-centric evaluation to ensure models benefit individuals from diverse cultural backgrounds. The authors emphasize that their work is intended for research purposes to advance the ethical and effective deployment of AI across different cultures.