Lost in Translation: Do LVLM Judges Generalize Across Languages?

1. Problem Statement

Large Vision-Language Models (LVLMs) increasingly rely on automated evaluators, such as reward models and "LLM-as-a-judge" frameworks, for alignment, training, and benchmarking. However, current evaluation benchmarks (e.g., VL-RewardBench, Multimodal RewardBench) are overwhelmingly English-centric.

• The Gap: There is a lack of large-scale benchmarks to evaluate how well these automated judges generalize across diverse languages and multimodal inputs.
• The Risk: Relying on English-only evaluations masks significant performance drops, biases, and hallucinations in non-English settings, leading to unreliable automated evaluation for global deployment.

2. Methodology: MM-JudgeBench

The authors introduce MM-JudgeBench, the first large-scale benchmark designed specifically for multilingual and multimodal judge model evaluation.

A. Dataset Construction

• Scale & Scope: The benchmark contains over 60,000 pairwise preference instances spanning 25 typologically diverse languages (including low-resource languages like Kazakh and Bengali).
• Subsets:
  1. M-VL-RewardBench: A multilingual extension of VL-RewardBench covering general vision-language preference judgments (e.g., hallucination detection, general instructions).
  2. M-OpenCQA: A chart-centric visual-text reasoning subset derived from OpenCQA, testing structured multimodal reasoning.
  3. M-MM-RewardBench (Training Set): A disjoint multilingual training set of ~100k instances derived from MM-RewardBench to support domain adaptation.
• Translation Pipeline:
  • Model Selection: The authors empirically selected Gemini-3-Pro as the translation engine after comparing it against Gemini-2.5-Pro, Flash, and Flash-Lite using LaBSE and CometKiwi metrics.
  • Quality Control: A rigorous filtering process was applied. Translations with LaBSE/Comet scores < 0.75 underwent human back-translation inspection. Poor translations were re-translated or discarded.
  • Efficiency: To minimize API costs, queries and candidate answers were translated into all 24 target languages in a single prompt, reducing API calls by 24x.

B. Evaluation Protocol

• Models Evaluated: 22 state-of-the-art LVLMs, including 15 open-source models (e.g., Qwen3-VL, InternVL, Gemma-3, LLaVA-Critic) and 7 proprietary models (e.g., GPT-5, Gemini-2.5, Claude-4.5).
• Metrics:
  • Pairwise Accuracy: Percentage of times the judge correctly identifies the preferred response.
  • Cross-lingual Variance: Standard deviation of accuracy across the 25 languages (lower is better).
  • Bias Analysis: Measurement of Position Bias (preference for the first/second answer) and Length Bias (preference for longer answers regardless of correctness).
  • Reasoning Impact: Comparison of judges that output a rationale vs. those that output only a label.

3. Key Contributions

1. MM-JudgeBench: The first unified benchmark for multilingual multimodal judge evaluation, covering 25 languages and 60k+ instances.
2. Large-Scale Empirical Study: Evaluation of 22 diverse LVLMs, revealing that English-only metrics fail to capture significant cross-lingual performance degradation.
3. Multilingual Training Resource: Release of a 100k-sample multilingual training set to facilitate domain-adaptive fine-tuning for open models.
4. Insight into Scaling & Bias: Demonstration that model size is not a reliable predictor of multilingual robustness and that high accuracy does not guarantee low bias.

4. Key Results & Findings

A. Performance Variance Across Languages

• English-Centric Bias: Most models perform best in English and significantly worse in low-resource languages (e.g., Kazakh).
• Efficiency Collapse: "Efficiency-optimized" variants (e.g., Gemini-2.5-Flash-Lite, Claude-4.5-Haiku) suffer severe performance collapse in non-English settings, dropping to ~40-50% accuracy, whereas their larger counterparts remain stable.
• Open-Source Leaders: The Qwen3-VL family (specifically the 32B variant) demonstrated the most consistent multilingual behavior among open models, often outperforming larger proprietary models and rivaling GPT-5 in stability.
• Specialized Models: Reward-specialized models like LLaVA-Critic-7B performed poorly (<50% accuracy), suggesting they do not generalize well to multilingual settings without specific training.

B. Bias and Robustness

• Position Bias: Significant positional bias was observed. For instance, Grok-4.1-Fast showed very high positional bias, while GPT-5-Mini and Qwen3-VL-8B were the most robust.
• Length Bias: Many models preferred longer answers even when incorrect, a bias that varied significantly by language.
• Conclusion: High accuracy alone is insufficient; a reliable judge must also be bias-resilient across languages.

C. Task-Level Analysis

• Structured vs. Open-Ended: Judges performed significantly better on structured tasks (Hallucination detection: 73%, Math reasoning: ~68%) compared to open-ended general multimodal instructions (49%).
• Reasoning Augmentation: Requiring the judge to generate a rationale alongside the decision improved accuracy by 2.2% to 4% for smaller models (Qwen3-VL-2B/8B).

D. Domain Adaptation

• Fine-tuning Qwen3-VL-8B on the provided multilingual training set (M-MM-RewardBench) resulted in a +14% improvement in accuracy on M-VL-RewardBench compared to zero-shot prompting, proving the utility of the released training data.

5. Significance and Future Work

• Exposing Limitations: The paper fundamentally challenges the assumption that English-centric benchmarks are sufficient for evaluating LVLM judges. It highlights that current "state-of-the-art" judges exhibit inconsistent and unreliable behavior in non-English contexts.
• Practical Implications: Developers deploying LVLMs globally must validate their reward models and judges across diverse languages to avoid alignment failures and safety risks.
• Future Directions: The authors suggest extending benchmarks to more languages, exploring reinforcement learning for multilingual alignment, and investigating the cultural nuances lost in automated translation.

Conclusion: MM-JudgeBench provides the necessary infrastructure to move beyond English-centric evaluation, revealing that while some models (like Qwen3-VL and GPT-5) show promise, the field currently lacks robust, universally reliable multilingual LVLM judges.