Multi-lingual Functional Evaluation for Large Language Models

1. Problem Statement

Current evaluations of Large Language Models (LLMs) in multi-lingual settings rely heavily on static data benchmarks (e.g., M-MMLU, M-GSM, Belebele). These benchmarks typically consist of fixed datasets translated into various languages. The authors argue that these static evaluations fail to capture the practical performance and robustness of models when faced with dynamic, functional tasks.

Key limitations of existing static benchmarks identified in the paper include:

• Lack of Functional Robustness: They do not test if a model can consistently execute instructions or solve problems when prompt variables change (e.g., changing numbers in a math problem or constraints in an instruction).
• Data Contamination: Static benchmarks are often contaminated with training data, inflating performance scores.
• Cultural/Contextual Gaps: Direct translations often lack realistic cultural context or fail to account for linguistic nuances (e.g., case sensitivity in scripts without uppercase/lowercase distinctions).
• Misleading Rankings: Models often rank similarly across static benchmarks, masking significant disparities in their ability to handle specific instruction types or reasoning tasks across different languages.

2. Methodology

The authors propose a shift from static evaluation to Functional Evaluation, where model inputs are generated dynamically from fixed templates with modifiable variables.

A. New Benchmarks Created

The paper introduces two new multi-lingual functional benchmarks by translating existing English functional templates into five additional languages: French, Spanish, Hindi, Arabic, and Yoruba. These languages were selected to span a spectrum of NLP resource availability (High, Medium, and Low/Extremely Low resource).

1. Cross-Lingual Grade School Math Symbolic (CL-GSM Symbolic):

   • Source: Translated 100 GSM-Symbolic templates.
   • Mechanism: Uses a fixed template with variables (e.g., {n1}, {n2}) and distractors (e.g., {name}, {color}). The ground truth is calculated via a deterministic function $f(X)$, ensuring the answer is mathematically verifiable regardless of the specific numbers used.
   • Scale: 50 variants per template per language $\rightarrow$ 5,000 QA pairs per language.
   • Task: Multi-step mathematical reasoning.

2. Cross-Lingual Instruction-Following Eval (CL-IFEval):

   • Source: Translated 541 IFEval prompts containing verifiable instructions.
   • Mechanism: Prompts include specific constraints (e.g., "use only capital letters," "write at least 300 words").
   • Filtering: Prompts relying on script-specific features (like case distinction in Arabic or Hindi) were removed to ensure cross-lingual comparability, resulting in a final set of 371 prompts.
   • Task: Instruction following with strict/loose accuracy metrics.

B. Data Construction & Validation

• Translation Pipeline: Initial translations were generated using Google Translate.
• Native Speaker Validation: Native speakers (students in the US and a native speaker in Nigeria for Yoruba) validated all translations.
• Quality Control: Validators checked for fidelity to intent, constraint strength, and cultural appropriateness. They corrected literal word choices that did not alter functional constraints but flagged prompts where instructions were impossible to execute in the target language (e.g., case constraints in non-cased scripts).

C. Evaluation Setup

• Models: Evaluated on open-weight models including Aya (23-35B, Expanse-32B), Gemma-2-9B-it, Qwen3-8B, Mistral-7B, and Mixtral-8x7B.
• Metrics:
  • CL-GSMSym: Exact match on the final answer (8-shot setting).
  • CL-IFEval: Prompt-level and Instruction-level accuracy (Strict and Loose).
• Baseline Comparison: Results were compared against static benchmarks: M-MMLU, M-GSM, and Belebele.

3. Key Contributions

1. Introduction of Functional Benchmarks: The creation of CL-GSM Symbolic and CL-IFEval, which test the robustness of model reasoning and instruction following rather than static knowledge recall.
2. Multi-lingual Resource Spectrum: The first systematic functional evaluation covering a wide range of resource levels, including low-resource languages like Yoruba.
3. Revealing the "Static vs. Functional" Gap: Demonstrating that high performance on static benchmarks does not guarantee functional competence.
4. Granular Robustness Analysis: Identifying specific failure modes, such as the inability of models to generalize probabilistic reasoning across languages or handle specific instruction types in low-resource settings.

4. Key Results

A. The Performance Gap

There is a significant and uneven gap between static and functional evaluation performance:

• Math Reasoning: Performance drops from M-GSM (static) to CL-GSM Symbolic (functional) by 24% (English), 17% (French), and 18% (Spanish).
• Instruction Following: The drop from Belebele (static reading comprehension) to CL-IFEval (functional instruction following) ranges from 15% to 24% across languages.
• Knowledge Recall: The drop from M-MMLU to CL-IFEval is minimal (0.5% to 3%), suggesting static knowledge benchmarks align better with functional performance for recall tasks than for reasoning or instruction following.

B. Language Performance Gaps

• Widening Disparities: Functional evaluations reveal much larger performance gaps between high-resource and low-resource languages compared to static benchmarks.
  • Example: Aya-expanse-32B has a 5.89% gap between high-resource languages on static benchmarks, but a 25.88% gap on CL-IFEval.
  • Example: Qwen3-8B shows a 31% performance gap on CL-IFEval across high-resource languages, despite low gaps on static math benchmarks.
• Low-Resource Collapse: In low-resource languages (e.g., Yoruba), functional performance is drastically lower (e.g., 4%–14% on CL-GSM Symbolic) compared to static benchmarks, where scores might still be in the 30% range.

C. Model Ranking Shifts

• Static Benchmarks: Models like Qwen3-8B, Gemma-2-9B, and Aya-Expanse consistently rank high.
• Functional Benchmarks: The ranking shifts significantly. While Qwen3-8B remains strong, other models (like Aya-23-35B) drop relative to their static performance.
• Robustness Variance: Some models are stable in certain languages but brittle in others depending on the prompt structure. For instance, Yoruba showed complete failure in the "Start/End" instruction category for all models.

D. Specific Failure Modes

• Probabilistic Reasoning: Template 3 (probabilistic inference) consistently produced the lowest performance across all models and languages, indicating a systemic weakness in generalizing probabilistic logic.
• Instruction Constraints: Models struggle with constraints that rely on script-specific features (e.g., capitalization in Arabic/Hindi) or complex counting constraints in low-resource languages.

5. Significance and Conclusion

This paper fundamentally challenges the reliance on static, translated datasets for evaluating multi-lingual LLMs. The authors demonstrate that:

1. Static benchmarks are insufficient: They mask critical weaknesses in reasoning and instruction following, particularly for low-resource languages.
2. Functional evaluation is necessary: It provides a more rigorous test of a model's ability to generalize and execute tasks dynamically.
3. Resource disparity is real: The "multi-lingual" capability of current SOTA models is often an illusion; they perform robustly in high-resource languages but degrade significantly in low-resource settings when tested functionally.

The work calls for a paradigm shift in LLM evaluation towards verifiable, template-based functional benchmarks to ensure models are truly robust and safe for global deployment. The authors also highlight the need for better translation validation and the inclusion of proprietary models in future studies to ensure comprehensive assessment.