💬 NLP

Beg to Differ: Understanding Reasoning-Answer Misalignment Across Languages

This paper introduces a human-validated framework revealing that large language models exhibit significantly higher reasoning-conclusion misalignment in non-Latin scripts compared to Latin ones, despite high task accuracy, thereby exposing critical blind spots in current multilingual evaluation practices.

Original authors: Anaelia Ovalle, Candace Ross, Sebastian Ruder, Adina Williams, Karen Ullrich, Mark Ibrahim, Levent Sagun

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anaelia Ovalle, Candace Ross, Sebastian Ruder, Adina Williams, Karen Ullrich, Mark Ibrahim, Levent Sagun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a student to take a history test. You give them the same exam in English, Spanish, Hindi, and Korean.

The Old Way of Checking:
In the past, if the student got the right answer on the Korean section, you would say, "Great job! They understand history." You only looked at the final bubble they filled in on the answer sheet.

The New Discovery:
This paper says: "Wait a minute. Let's look at how they got there."

The researchers built a system to read the student's "scratch work" (their reasoning) and check if the logic actually leads to the answer. They found a shocking blind spot: The student is often getting the right answers for the wrong reasons, especially when the test is in a language they aren't as fluent in.

Here is the breakdown using simple analogies:

1. The "Lucky Guess" vs. The "Real Logic"

Imagine a student taking a math test.

Scenario A (English): They write out the steps: "2 + 2 = 4. 4 + 4 = 8." Then they circle 8.
- Verdict: Perfect. The logic supports the answer.
Scenario B (Korean): They write: "I think the answer is 8 because... uh... numbers are cool? Also, 2+2 is 4." Then they circle 8.
- Verdict: The answer is right, but the logic is nonsense. They got lucky.

The paper found that for languages using non-Latin scripts (like Korean, Arabic, Hindi), the models were much more likely to be in "Scenario B." They were guessing the right answer but couldn't explain why in a way that made sense.

2. The "Script" Problem

The researchers noticed something weird about the shape of the letters.

Latin Scripts (English, Spanish): The models were like a reliable GPS. They gave the right turn-by-turn directions to get to the destination.
Non-Latin Scripts (Korean, Arabic, Hindi): The models were like a drunk friend giving directions. They might point you to the right building (the correct answer), but the path they described was full of holes, contradictions, or made-up facts.

In fact, the "drunk friend" behavior was twice as common in non-Latin languages compared to Latin ones. It wasn't just that the models knew less; it was that their thinking process broke down differently.

3. The "Magic 8-Ball" Effect

The paper calls this Reasoning-Answer Misalignment.
Think of the AI as a Magic 8-Ball.

If you shake it in English, it says "Outlook good" and gives you a logical reason why.
If you shake it in Korean, it might still say "Outlook good" (the correct answer), but if you look inside the ball, the reason it gives is "Because the stars aligned" or "Because I feel like it."

The AI is essentially hallucinating a justification after it has already guessed the answer. It's not thinking to the answer; it's guessing the answer and then inventing a story to make it look smart.

4. Why Does This Matter?

You might ask, "If the answer is right, who cares how they got there?"

The paper argues that trust is the issue.

If a doctor uses an AI to diagnose a disease, and the AI says "You have the flu" (Correct), but the reasoning is "Because you look tired and I like the flu," that's dangerous.
If the AI is used for legal or political decisions in different languages, and it's "guessing" in those languages while "thinking" in English, we are building a system that is unfair and unreliable for half the world.

The Takeaway

The researchers created a new "report card" for AI. Instead of just grading the final answer (A, B, C, or D), they grade the essay the AI writes to get there.

Their main finding: Current AI models are great at English reasoning, but when they switch to other languages, they often stop "thinking" and start "pattern matching." They get the right answer by luck or memorization, but their internal logic falls apart.

In short: Just because the AI gives you the right answer doesn't mean it actually understands the question. And right now, it understands English much better than it understands the rest of the world.

1. Problem Statement

While Large Language Models (LLMs) have demonstrated strong reasoning capabilities via Chain-of-Thought (CoT) prompting, current evaluation practices predominantly focus on answer accuracy. This creates a critical blind spot: models may arrive at correct answers through flawed reasoning (hallucination or memorization) or fail to support their conclusions logically, particularly in multilingual settings.

The authors identify a gap in understanding whether reasoning quality transfers across languages. Specifically, they investigate:

Does the logical coherence of reasoning traces degrade in low-resource languages (LRL) or non-Latin scripts compared to high-resource languages (HRL) and Latin scripts?
Can models produce correct answers based on reasoning that does not actually support those answers?
How do reasoning failures manifest differently across language scripts and resource levels?

2. Methodology

The authors propose a human-validated framework to evaluate Reasoning-Answer Alignment (the logical support a reasoning trace provides for a conclusion) without requiring gold-standard reasoning traces.

A. Evaluation Framework

Isolation of Reasoning: The final answer is removed from the model's output to create a "truncated trace."
Back-Translation: Non-English responses are back-translated to English using a high-performing translator (Command-R+) to ensure consistent evaluation by English-centric evaluators.
Evaluation Dimensions:
1. Correctness: Does the model's final answer match the ground truth?
2. Consistency (Trace Inconsistency): Does the reasoning trace logically support the stated conclusion?
Metric: The primary metric is the Trace Inconsistency Rate (TIR), defined as the proportion of cases where the reasoning fails to support the model's conclusion, regardless of whether the final answer is correct or incorrect.

B. Experimental Setup

Dataset: GlobalMMLU (7 fact-retrieval domains: global facts, history, politics, etc.).
Languages: 6 languages representing varying resource levels and scripts:
- High-Resource/Latin (HRL/LS): English, Spanish.
- High-Resource/Non-Latin (HRL/NLS): Hindi, Arabic.
- Low-Resource/Non-Latin (LRL/NLS): Ukrainian, Korean.
Models: 6 frontier instruction-tuned models (Llama-4-Scout, Llama-3.3-70B, Qwen3-32B/Thinking, Qwen2.5-72B/32B).
Annotation Protocol:
- Human Evaluation: A two-phase study involving expert annotators who assess truncated traces for logical coherence, information sufficiency, and inferred answers.
- Automated Evaluation: An LLM (Llama-4-Scout) is used as an automated judge to scale the evaluation to 65k+ examples, validated against human consensus.

C. Error Taxonomy

Through thematic analysis of human annotations, the authors developed a taxonomy of reasoning failures, including:

Unsupported Claims: Assertions made without evidence (e.g., "common knowledge").
Ambiguous Facts: Vague or underspecified claims.
Logical Contradictions: Internal inconsistencies within the trace.
Illogical Leaps: Conclusions that do not follow from premises.
Multiple Answers: The trace concludes with multiple options without preference.

3. Key Results

A. Reasoning-Answer Misalignment

Disproportionate Drop in Alignment: While answer accuracy drops moderately from High-Resource to Low-Resource languages (e.g., ~80% to ~70%), Reasoning-Answer Alignment drops by more than two-fold.
Script Bias: Non-Latin script languages (NLS) exhibit at least twice as much misalignment as Latin script languages (LS). This gap is larger than the gap observed between high and low-resource languages, suggesting script type is a stronger predictor of reasoning failure than resource availability.
Correct Answers, Wrong Reasons: A significant portion of correct answers in LRL/NLS settings are derived from reasoning traces that do not logically support the conclusion. Models are often "getting the right answer for the wrong reasons."

B. Error Distribution

Evidential Errors Dominate: The most common failure modes are evidential errors (Unsupported Claims and Ambiguous Facts), accounting for ~47% of errors. This indicates models struggle more with grounding reasoning in verifiable facts than maintaining internal logical consistency.
Thinking Models: "Thinking" models (e.g., Qwen3-32B-thinking) show improved consistency overall but still exhibit a significant jump in unsupported claims when operating in low-resource settings.

C. Automated vs. Human Evaluation

The automated evaluator (Llama-4-Scout) showed strong correlation with human judgment, particularly in Answer Selection (Cohen's $\kappa$ = 0.53, 62.5% agreement), validating the use of automated scaling for large-scale reasoning analysis.

4. Key Contributions

Evidence of Systematic Misalignment: Demonstrated that reasoning-answer misalignment is a systematic issue in multilingual LLMs, disproportionately affecting non-Latin scripts (2x higher than Latin scripts).
Human-Validated Framework: Introduced a scalable framework for evaluating reasoning coherence that isolates the reasoning process from the final answer.
Independence of Correctness: Proved that reasoning failures occur independently of answer correctness; models can be accurate but unfaithful in their reasoning.
Error Taxonomy: Characterized the specific failure modes (primarily evidential gaps) across language scripts and subjects.

5. Significance and Implications

Rethinking Evaluation: The paper argues that answer accuracy is an incomplete metric for multilingual capabilities. Relying solely on accuracy masks the fact that models may be memorizing patterns or hallucinating reasoning in non-dominant languages.
Trust and Reliability: For high-stakes applications, a model that provides a correct answer via flawed reasoning is unreliable. The findings highlight the need for reasoning-aware evaluation frameworks in multilingual settings.
Future Directions: The results suggest that current training paradigms may not be teaching uniform reasoning processes across languages. Future work must focus on improving the grounding of reasoning in factual knowledge, particularly for non-Latin scripts and low-resource languages, rather than just optimizing for final answer accuracy.

In conclusion, the paper reveals a critical "blind spot" in current LLM benchmarks: multilingual models often possess the ability to guess the right answer without the ability to justify it logically, a problem that is exacerbated in non-Latin scripts and low-resource languages.