Technical Summary: Truth Knows No Language: Evaluating Truthfulness Beyond English

Problem Statement

The evaluation of truthfulness in Large Language Models (LLMs) has been predominantly confined to English, primarily utilizing the TruthfulQA benchmark. While TruthfulQA is the standard for assessing how LLMs mimic human falsehoods, its exclusive availability in English limits the understanding of whether LLMs maintain truthfulness across different languages, particularly low-resource ones. Existing attempts to translate the dataset have been informal (machine-translated) rather than professional, lacking a rigorous assessment of the dataset's utility for non-English languages. Furthermore, there is a need to determine if the cultural and temporal context inherent in the original English questions affects truthfulness evaluations when translated, and whether current automatic evaluation metrics (like multiple-choice) are sufficient for cross-lingual truthfulness assessment.

Methodology

This study introduces a professionally translated extension of the TruthfulQA benchmark into four languages: Basque, Catalan, Galician, and Spanish. The translation process involved professional native speakers and adhered to specific guidelines to maintain parallelism while handling cultural nuances (e.g., literal translation of proverbs, borrowing of acronyms, and generalization of culture-specific terms).

Experimental Setup

The authors evaluated 12 state-of-the-art open LLMs from the Llama 3/3.1 and Gemma 2 families, ranging from 7B to 70B parameters. Both base and instruction-tuned variants were tested.

Three evaluation methods were employed:

1. Human Evaluation: A manual assessment of 400 responses (100 questions × 4 models) for truthfulness and informativeness using binary labels.
2. Multiple-Choice (MC2): The standard automatic metric used in leaderboards, calculating the likelihood of correct answers against incorrect ones.
3. LLM-as-a-Judge: An automatic generation evaluation where a fine-tuned judge model (specifically Gemma 2 9B Instruct, trained on machine-translated data) evaluates the truthfulness of generated responses. This method was selected based on its high correlation with human judgments.

Additionally, the study compared the professionally translated dataset against a version generated via Machine Translation (MT) using Claude 3.5 Sonnet to assess the viability of MT for scaling truthfulness benchmarks. The dataset was also analyzed by splitting questions into universal (time/context-independent) and context/time-dependent categories.

Key Contributions

1. Multilingual TruthfulQA Dataset: The first professionally translated version of TruthfulQA covering Basque, Catalan, Galician, and Spanish, addressing the gap in low-resource language evaluation.
2. Evaluation Framework: A comprehensive comparison of evaluation methods, demonstrating that LLM-as-a-Judge correlates significantly better with human judgments than multiple-choice metrics (MC2) across all languages.
3. Informativeness Analysis: The identification that base models often produce uninformative responses (e.g., "I have no comment"), which artificially inflates truthfulness scores in standard metrics. The study highlights the necessity of evaluating informativeness separately, particularly for base models.
4. Translation Viability: Evidence suggesting that high-quality machine translation is a viable, scalable alternative to professional translation for extending truthfulness benchmarks to additional languages, provided the translation engine is state-of-the-art.

Results

• Language Disparities: LLMs generally perform best in English and worst in Basque (the lowest-resourced language). However, the overall discrepancy in truthfulness scores across languages is smaller than anticipated.
• Model Performance:
  • Instruction-tuned models consistently outperform base models in truthfulness.
  • Larger models (e.g., 70B) tend to outperform smaller counterparts within the same family, contrasting with some previous findings where model size did not correlate with truthfulness.
  • Gemma 2 models demonstrated superior truthfulness compared to Llama 3 models across languages.
• Evaluation Metrics: The LLM-as-a-Judge method showed higher agreement with human evaluators (Cohen's Kappa) than MC2. Notably, a judge trained on English data performed better than MC2 even when evaluating non-English languages, though a judge trained on multilingual data performed best.
• Universal vs. Contextual Questions: Models perform significantly better on universal knowledge questions (approaching 90% accuracy) compared to time- and context-dependent questions. This suggests that static benchmarks lacking contextual variety may lead to performance saturation and fail to reflect real-world misinformation challenges.
• Machine Translation: Statistical tests confirmed no significant difference in truthfulness scores between the professionally translated and machine-translated datasets, supporting the use of MT for dataset expansion.

Significance and Claims

The paper claims that while English remains the language where LLMs exhibit the highest truthfulness and coherence, the gap between English and other languages is not as vast as previously feared. The study challenges the assumption that multiple-choice metrics are sufficient for truthfulness evaluation, advocating for LLM-as-a-Judge approaches that better capture the nuances of generated text.

The authors emphasize that informativeness is a critical, often overlooked factor; without it, truthfulness metrics for base models can be misleadingly high due to non-committal responses. Furthermore, the results indicate that universal questions are increasingly easy for modern LLMs to answer correctly, suggesting that future truthfulness benchmarks must incorporate cultural and temporal variability to remain rigorous and relevant.

Finally, the paper posits that machine translation offers a scalable, cost-effective pathway for extending truthfulness benchmarks to a wider array of languages, potentially democratizing the evaluation of LLM safety and reliability beyond the English-speaking world. However, the authors modestly note limitations regarding the specific scope of Iberian languages and the static nature of the dataset, which may not fully capture dynamic, real-world misinformation.