Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study

Problem Statement
While Large Language Models (LLMs) are increasingly utilized as automatic evaluators ("LLMs-as-a-Judge") for generated text, the majority of existing research and benchmarks are centered on English. Extending these evaluation capabilities to multilingual settings, particularly for low-resource languages, remains a significant challenge. Current gaps include a lack of robust multilingual evaluation datasets, uncertainty regarding optimal training strategies (e.g., monolingual vs. multilingual supervision, translation of instructions), and limited understanding of how model scale and data availability (in-domain vs. out-of-domain) influence evaluation reliability across languages with varying resource levels and typological structures.

Methodology
The authors conduct an empirical study focusing on three languages representing different resource levels and typological families: English (high-resource, Germanic), Spanish (mid-resource, Romance), and Basque (low-resource, language isolate with agglutinative morphology).

• Dataset Creation: Since no public judgment datasets existed for Basque or Spanish, the authors adapted existing English datasets. They translated the FeedBack Collection (100k instruction-response pairs) to create training data. For evaluation, they translated two benchmarks: RECON (treated as in-domain, following the same generation protocol as training data) and FLASK (treated as out-of-domain, based on human scoring of alignment dimensions).
• Experimental Scenarios: The study investigates two primary conditions:
  1. In-Domain Availability: Fine-tuning judges on task-specific data.
  2. Out-of-Domain Unavailability: Evaluating zero-shot or fine-tuning on related but out-of-domain data.
• Training Configurations: The authors compared:
  • Language Alignment: Full translation of all components (instructions, rubrics, outputs) vs. partial translation (keeping evaluation rubrics and system prompts in English while translating only instructions and outputs).
  • Supervision Type: Monolingual training (single language) vs. Multilingual training (combined parallel data).
  • Model Selection: Experiments utilized LLaMA family models (8B and 70B variants), including instruction-tuned models, reasoning-distilled models, and models already proficient in the target language (Latxa for Basque).
  • Scale Analysis: Comparing 8B models against 70B models and proprietary alternatives (GPT-5.2, Command-A).

Key Contributions

1. Resource Expansion: The work extends two existing meta-evaluation datasets (RECON and FLASK) to Spanish and Basque, providing a controlled testbed for multilingual evaluation research.
2. Strategic Insights for In-Domain Settings:
   • Instruction Translation: Keeping evaluation rubrics and instructions in English during training and testing (partial translation) yields more consistent judgments and higher correlation with gold labels than full translation, even for monolingually trained models.
   • Multilingual vs. Monolingual: Multilingual training improves performance for Spanish and Basque compared to strictly monolingual training, demonstrating effective cross-lingual transfer.
   • Model Proficiency: Adapting a model already proficient in the target language (e.g., Latxa for Basque) using English instructions is more effective than training a general-purpose model on translated target-language data.
   • Model Scale: When in-domain data is available, smaller 8B models fine-tuned on task-specific data can achieve performance comparable to much larger 70B models and proprietary systems.
3. Strategic Insights for Out-of-Domain Settings:
   • Fine-tuning Risks: In out-of-domain scenarios, fine-tuning on related data can degrade performance, particularly for larger models (70B), which may exhibit overconfidence and score polarization.
   • Zero-Shot Superiority: For out-of-domain evaluation, zero-shot inference with larger models often proves more robust and reliable than fine-tuning on limited or related data.

Results

• In-Domain Performance: Models trained with English evaluation rubrics (partial translation) achieved a Pearson correlation of 0.73 with gold labels, significantly outperforming fully translated setups (0.56). Multilingual training improved Pearson scores by 4.2 points for Basque and 1.4 points for Spanish compared to monolingual baselines.
• Scale Effects: While zero-shot performance generally improved with model size, fine-tuning reduced the marginal gains of scaling. Fine-tuned 8B models often matched or surpassed 70B models and proprietary alternatives in in-domain tasks.
• Out-of-Domain Performance: Fine-tuning 70B models on out-of-domain data led to a degradation in generalization, characterized by a tendency to assign extreme scores (polarization) and a drop in metrics related to factuality and metacognition. Conversely, 8B models benefited from fine-tuning but still underperformed compared to zero-shot large models.
• Translation Quality: Statistical analysis indicated that machine-translated benchmarks were sufficient for identifying effective training strategies, as models did not differ systematically in sensitivity to translation quality compared to human-verified subsets.

Significance
The paper provides practical guidance for constructing efficient and reliable multilingual evaluation pipelines. It challenges the assumption that larger models are always superior, demonstrating that targeted fine-tuning of smaller, open-weight models can be highly effective when in-domain data is available. Conversely, it warns against indiscriminate fine-tuning in data-scarce or out-of-domain scenarios, where zero-shot evaluation of larger models may be more robust. The findings suggest that full translation of evaluation prompts is often unnecessary, and leveraging English rubrics can improve cross-lingual consistency. These insights aim to support the development of more transparent, linguistically inclusive, and resource-efficient evaluation systems for diverse language communities.