Challenges and Recommendations for LLMs-as-a-Judge in Multilingual Settings and Low-Resource Languages

Problem Statement
The "LLM-as-a-Judge" paradigm has become the dominant evaluation method for natural language generation (NLG) tasks, largely due to the high correlation between LLM judgments and human assessments in English, as well as the cost and scalability advantages over human annotation. However, this paradigm is increasingly being extended to multilingual settings and low-resource languages (LRLs) without sufficient validation. The paper identifies a critical gap: while LLMs exhibit limited proficiency in LRLs, researchers often apply them as evaluators in these settings without verifying their reliability. This leads to risks such as inconsistent evaluation outcomes, overestimation of performance in LRLs, and a widespread reliance on single judge models without human validation or ensemble methods. The authors note that existing surveys on LLM-as-a-Judge focus predominantly on English, leaving a lack of systematic analysis regarding the reliability of these judges in multilingual and low-resource contexts.

Methodology
To address this gap, the authors conducted a systematic literature review of the ACL Anthology.

• Search Strategy: They implemented a reproducible pipeline to search metadata (titles and abstracts) from the ACL Anthology using three keyword groups: (1) LLMs, (2) Judge/Evaluation functionality, and (3) Low-resource/Multilingual contexts.
• Filtering: From an initial set of 650 papers mentioning LLM-as-a-Judge, the search yielded 49 candidates containing low-resource or multilingual keywords.
• Annotation: The authors manually reviewed these 49 papers, excluding those where LLMs were used for direct text labeling (annotator-oriented) rather than evaluating model outputs (evaluator-oriented), or where the focus was on non-natural languages (e.g., code).
• Final Corpus: This process resulted in a final corpus of 33 in-scope papers.
• Analysis Dimensions: The authors analyzed these 33 papers across five dimensions: judging paradigm (e.g., direct scoring, pairwise comparison), downstream NLP tasks, judge model families (closed vs. open-source), language coverage, and the validation protocols used to assess judge reliability.

Key Findings and Results
The analysis of the 33 papers reveals several structural weaknesses in current research practices:

1. Shallow Depth in Low-Resource Coverage: While many papers claim multilingual coverage, the focus on truly low-resource languages is shallow. Of the 33 papers, only 8 (24%) include low-resource languages as at least half of their evaluation set. Coverage is heavily skewed toward higher-resource languages, and specific regions (e.g., Indigenous American, Pacific Island, and most Sub-Saharan African languages) are largely absent.
2. Dominance of Closed-Source and Single Judges: The judge ecosystem is dominated by closed-source models, particularly the GPT family (used in 79% of papers). Furthermore, 48% of the studies rely on a single judge model family, and 33% use GPT as the sole judge. Cross-judge validation is rare.
3. Validation Deficiencies: Although 73% of papers report some form of human comparison, this validation is often conducted only on English or high-resource subsets. For instance, some studies validate a multilingual judge using only English benchmarks, leaving the reliability in the target LRLs unchecked. Several papers deploy LLM judges in LRLs (e.g., Nepali, Yorùbá) without any human or gold-label checks, assuming reliability based on the judge's multilingual capabilities rather than empirical evidence.
4. Inconsistency and Bias: Existing evidence cited in the paper indicates that human-LLM agreement drops significantly in LRLs. Issues such as verbosity bias, self-enhancement bias, and language-specific performance degradation are more pronounced in low-resource settings.

Recommendations
Based on these findings, the authors propose four specific recommendations for the NLP community:

1. Validate in Target Contexts: Researchers must investigate whether a judge is reliable for the specific target language, rather than assuming English-based validation transfers to LRLs.
2. Keep Humans in the Loop: Human evaluation should be conducted on at least a subset of data for LRLs, accompanied by statistical confidence estimates. Researchers should also document the demographics and expertise of human evaluators to ensure representativeness.
3. Re-evaluate Metric Necessity: Before adopting LLM-as-a-Judge, researchers should consider if non-LLM metrics (e.g., chrF, Exact Match) are equally feasible. LLM judges should only be used when no better existing metric is available and should be used in conjunction with validated traditional metrics to catch errors like language variety mismatches.
4. Consider Cultural and Domain Relevance: Evaluation should extend beyond linguistic competence to include cultural competence and domain-specific relevance (e.g., medical or legal contexts), as LLMs may amplify cultural stereotypes or fail in specialized domains.

Significance and Claims
The paper claims that while LLM-as-a-Judge offers efficiency, its application in multilingual and low-resource settings is currently unsound due to a lack of validation and over-reliance on unproven assumptions. The authors argue that without rigorous validation, the ease of using LLMs as judges may lead to a degradation of evaluation quality, creating a cycle where models are judged by other models that are themselves poorly validated in those languages. The study serves as a call to action for the *ACL community to move beyond superficial multilingual claims and establish robust, human-validated evaluation protocols for low-resource languages. The authors explicitly state that their work is a survey of existing literature rather than a set of new experiments, aiming to highlight shortcomings in current research trends.