💬 NLP

ADAGE: A Language-Agnostic Pipeline for Analogical Reasoning Evaluation

The paper introduces ADAGE, a language-agnostic pipeline that creates translation-free analogical reasoning benchmarks in Arabic, Amharic, and Japanese, revealing a significant performance gap where models proficient in English struggle with culturally grounded reasoning in these native languages.

Original authors: Ahmed Haj Ahmed, Alvin Grissom II

Published 2026-07-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ahmed Haj Ahmed, Alvin Grissom II

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand human jokes. You might think the best way is to take a book of famous jokes written in English, translate them into Spanish, Japanese, or Arabic, and ask the robot to explain them. But here's the catch: a joke often relies on the specific culture it was born in. If you translate a joke about a specific American holiday into a language where that holiday doesn't exist, the joke loses its magic. The robot might guess the answer because it sounds funny, not because it actually understands the meaning. This is the problem scientists face when testing "multilingual" AI. They often just translate English tests into other languages, which creates a fake illusion that the AI is smart in those languages.

To truly test if an AI can think like a human, we need to see if it can understand the deep, unspoken rules of a culture—like knowing that "don't count your chickens before they hatch" means you shouldn't be too confident until something actually happens. This kind of thinking is called analogical reasoning. It's the ability to take an old idea and apply it to a brand-new situation. If an AI can only do this in English, but fails when the same logic is presented in a different culture, it hasn't really learned to think; it's just memorized English patterns. This paper asks a simple but scary question: Are our AI models actually smart in other languages, or are they just really good at translating?

The researchers introduce a new tool called ADAGE (Analogical Difficulty-by-design Assessment for Grounded Evaluation). Think of ADAGE as a master chef who doesn't just translate recipes from a French cookbook; instead, they go into local markets in Arabic, Amharic, and Japanese to find the freshest, most authentic local ingredients (proverbs) and cook up brand-new dishes (scenarios) that test if the AI can taste the difference.

Here is how they cooked up their experiment:

The Ingredients: They gathered thousands of proverbs (short, wise sayings like "He who does not know the falcon will roast it") from native speakers in Arabic, Amharic, and Japanese.
The Recipe: Instead of just asking the AI to explain the proverb, they used a clever trick. They grouped proverbs by their "flavor" or theme (like "patience" or "caution"). Then, they asked an AI to write a modern story for each proverb.
The Trap: To make the test hard, they created a multiple-choice question. They gave the AI a story and four proverbs to choose from. One was the perfect match. The other three were "distractors":
- One was from the same theme group (a "fake-out" that sounded right but wasn't quite right).
- One was a different theme but sounded similar (a "near-miss").
- One was completely random (the "easy" wrong answer).
The Taste Test: They fed these tests to 14 different AI models, ranging from tiny ones to massive ones, and watched how they performed.

What They Found

The results were a bit of a shock, like finding out a student who aced the English test failed the math test in a different language.

The "Translation Trap" is Real: When the AI models were tested on English proverbs, they did very well, scoring around 80% or higher. But when the researchers tested them on the native Arabic, Amharic, and Japanese benchmarks, the scores plummeted. The best models dropped by 12 to 52 percentage points. For example, a model that got 83% right in English might only get 70% right in Arabic, and a mere 41% in Amharic.
The "Low-Resource" Reality: The Amharic test was the hardest. Even the biggest, smartest models barely scored better than random guessing (which would be 25% for a four-choice question). The AI models seemed to get stuck on the language itself, unable to even understand the words, let alone the deep meaning.
Size Isn't Everything: The researchers expected that bigger models (with more "brain power") would do better. While bigger models did improve on Arabic and Japanese, the improvement was small. On Amharic, making the model bigger didn't help at all. It suggests that just adding more data or size doesn't fix the problem if the AI doesn't have a deep cultural foundation to stand on.
The Distractor Trick Worked: The clever "distractor" strategy worked perfectly. The AI models made the most mistakes on the "near-miss" answers (the ones that sounded similar but were wrong), proving they were actually trying to reason and not just guessing randomly.

The Big Takeaway

The paper concludes that the current way we test AI—by simply translating English tests into other languages—is lying to us. It makes AI look smarter in other languages than it really is. The AI models are great at English cultural reasoning, but they struggle to apply that same logic to the rich, complex cultures of Arabic, Amharic, and Japanese.

The authors suggest that until we build tests that are native to each culture, using local proverbs and stories, we won't know if our AI is truly multilingual or just a very good translator. They have released their new "kitchen" (the ADAGE pipeline) and their "recipes" (the benchmarks) so other scientists can cook up their own tests for any language in the world, ensuring that future AI can truly understand us, no matter where we are from.

Technical Summary: ADAGE – A Language-Agnostic Pipeline for Analogical Reasoning Evaluation

1. Problem Statement

Current multilingual reasoning evaluation relies overwhelmingly on machine-translating English-centric benchmarks. The authors argue this practice creates an "illusion of multilingual competence" due to two fundamental flaws:

Linguistic Artifacts: Machine translation introduces unnatural phrasing ("translationese") and subtle errors that confound measurement, causing model failures to reflect poor translation rather than poor reasoning.
Cultural Displacement: Translation preserves the source culture's assumptions while discarding the target culture's reasoning patterns. Abstract analogical reasoning, particularly involving proverbs, relies on culturally shared metaphors and norms that are non-transferable. Consequently, translation-based benchmarks test a model's ability to apply English-centric analogies in another language, rather than its ability to reason within a different cultural framework.

Existing native-authored benchmarks (e.g., MultiNRC, Jawaher, ProverbEval) address this gap but suffer from being constructed ad hoc for single languages without a replicable pipeline, and most test proverb explanation (knowledge retrieval) rather than analogical application (mapping abstract meaning to novel scenarios).

2. Methodology: The ADAGE Pipeline

The authors introduce ADAGE (Analogical Difficulty-by-design Assessment for Grounded Evaluation), a language-agnostic, semi-synthetic pipeline designed to construct culturally grounded reasoning benchmarks without translation. The pipeline operates in seven stages:

Data Acquisition: Normalization and deduplication of a native proverb corpus.
Thematic Clustering:
- Pass 1: An LLM assigns concise English theme labels to each proverb based on native explanations. English is used as an intermediate target to leverage stronger instruction-following capabilities.
- Pass 2: Theme labels are embedded using a multilingual sentence-transformer and grouped via agglomerative clustering (Ward's linkage). The number of clusters ( $k$ ) is optimized to maximize the weighted silhouette coefficient.
- Pass 3: Clusters are assigned descriptive names.
- Design Rationale: Clustering on abstract theme labels rather than raw text avoids surface-level lexical similarities, ensuring groups are thematically coherent.
Scenario Generation: An LLM (GPT-OSS-120B) generates 10 candidate scenarios per proverb. These are short, modern narratives in the target language illustrating the proverb's meaning via analogy across 10 rotating social domains (e.g., workplace, family, politics).
Automated Filtering: Scenarios are filtered based on language identification, length, keyword overlap (to prevent lexical leakage), and embedding similarity to the proverb.
Human Quality Audit: A lightweight sanity check by native speakers confirms coherence and cultural appropriateness.
Distractor Selection & MCQ Assembly: A novel distractor strategy constructs a controlled difficulty gradient for each multiple-choice question (MCQ):
- Same-Cluster Distractor: A proverb from the same thematic cluster (thematic confusion trap).
- Cross-Theme Near-Match: The most embedding-similar proverb from a different cluster (traps models relying on surface similarity).
- Random-Cluster Distractor: A proverb from a distant cluster (baseline plausibility).
- Candidates with high embedding similarity (>0.85) are excluded to prevent near-synonyms.
Multi-Agent Validation: Three open-weight LLM judges evaluate items on faithfulness, naturalness, and distractor plausibility using a 1–5 Likert scale. Items are flagged if scores fall below thresholds or inter-judge agreement is low.
Dataset Curation: Final train/dev/test splits are created at the proverb level to prevent memorization of proverb-theme associations.

3. Key Contributions

ADAGE Pipeline: A replicable, language-agnostic methodology for constructing culturally grounded reasoning benchmarks, validated across three typologically diverse languages.
Three New Benchmarks: The release of CAPR-ar (Arabic), CAPR-am (Amharic), and CAPR-ja (Japanese). These are the first benchmarks targeting abstract analogical proverb reasoning in these specific languages.
Empirical Evidence of a Cultural Reasoning Gap: A demonstration that translation-based evaluation significantly overestimates multilingual reasoning capabilities.

4. Experimental Results

The pipeline was validated by constructing benchmarks for Arabic, Amharic, and Japanese, and evaluating 14 open-weight models (ranging from 270M to 35B parameters) under zero-shot conditions.

Performance Drop: Models performing well on English proverb reasoning (ePiC benchmark) struggled substantially on native benchmarks. Accuracy dropped by 12–52 percentage points relative to English.
- Arabic: Best model (Gemma 3 27B) achieved 70.1% (vs. 83.0% on English).
- Amharic: Best model achieved only 41.3% (vs. 25% chance), indicating a genuine low-resource failure.
- Japanese: Best model achieved 93.0%, though the authors note this may be influenced by high pretraining coverage and cross-lingual cognates.
Scaling Behavior:
- Arabic: Showed log-linear scaling gains with model size.
- Japanese: Showed diminishing returns; scaling yielded gains but plateaued.
- Amharic: Scaling effects largely vanished; models clustered near chance regardless of size, suggesting a language competence bottleneck rather than a reasoning capacity issue.
Distractor Analysis: The distractor strategy functioned as designed. On Arabic and Japanese, "cross-theme near-match" distractors attracted the highest error rates, creating a clear difficulty gradient. On Amharic, the distribution was flat, consistent with random guessing.
Position Bias: Small models (e.g., Gemma 3 270M) exhibited severe positional bias (selecting option D in 89.6% of cases), while larger models (>4B) showed uniform distributions.

5. Significance and Claims

The paper claims that ADAGE provides the first methodology enabling native speaker communities to construct rigorous, difficulty-controlled reasoning benchmarks without translation infrastructure.

The primary significance of the work is the revelation of a consistent cultural reasoning gap. The authors argue that "multilingual" performance is not a single capability but a composite of language-specific competences that vary by orders of magnitude. Specifically:

Translation-based evaluation obscures the fact that models often lack the language competence to engage with culturally grounded tasks in low-resource settings (evidenced by the Amharic results).
Even in high-resource settings (Arabic), models fail to transfer reasoning capabilities from English to native contexts, suggesting that current models rely on English-centric heuristics rather than robust, culture-independent reasoning.
The gap does not close with parameter scaling alone; training data composition (e.g., Aya Expanse's targeted multilingual training) appears as critical as model size.

The authors conclude that evaluation-by-translation substantially overestimates multilingual reasoning capabilities and that native, translation-free benchmarks are necessary to accurately assess a model's ability to reason within different cultural schemas.