💬 NLP

Reasoning or a Semblance of it? A Diagnostic Study of Transitive Reasoning in LLMs

This paper investigates whether LLaMA 2 and Flan-T5 exhibit genuine transitive reasoning or rely on superficial cues by controlling for word overlaps, pre-trained knowledge, and named entities, revealing that while both models leverage lexical overlaps, Flan-T5 demonstrates greater resilience to knowledge-based manipulations, suggesting a potential role for fine-tuning in developing logical understanding.

Original authors: Houman Mehrafarin, Arash Eshghi, Ioannis Konstas

Published 2026-09-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Houman Mehrafarin, Arash Eshghi, Ioannis Konstas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving field of artificial intelligence, a specific type of computer program known as a Large Language Model has captured the world's attention. These systems, trained on vast amounts of text from the internet, can write stories, translate languages, and answer complex questions. A central question for scientists studying these machines is whether they truly understand logic or if they are simply mimicking the patterns they have seen before. To test this, researchers look at a fundamental skill called transitive reasoning. In everyday terms, this is the ability to connect two separate pieces of information to reach a new conclusion. For instance, if a person knows that a cat is a type of animal, and that all animals need food, they can logically deduce that a cat needs food without being explicitly told that specific fact. This process requires more than just memory; it demands the ability to weave facts together. The question driving recent research is whether these powerful computer programs are actually performing this mental weaving, or if they are taking a shortcut by spotting familiar words and guessing the answer.

A team of researchers from Heriot-Watt University and the University of Edinburgh set out to investigate this very issue. They focused on two distinct types of language models, one called LLaMA 2 and another called Flan-T5, to see how they handled questions that required connecting two facts. The scientists used two existing collections of questions designed to test this kind of thinking. One collection, known as QASC, presents multiple-choice questions about science that require combining two statements to find the right answer. The other collection, called Bamboogle, contains questions that are tricky enough that a standard internet search engine might get them wrong, forcing the model to rely on its own processing. The researchers wanted to know if the models were genuinely reasoning through the steps or if they were just latching onto specific words, like dates or names, that appeared in both the question and the answer.

To find the truth, the team designed a series of clever experiments where they systematically altered the information given to the models. They started by checking if the models could solve the problems when given the facts and a clear example of how to connect them. Both models performed well under these conditions, but the researchers suspected this might be too easy. They then removed the example of how to connect the facts, leaving the models to figure it out on their own. The results were revealing. The Flan-T5 model, which had been specifically trained on similar reasoning tasks, continued to perform very well even without the example. The LLaMA 2 model, however, struggled significantly without that guidance, suggesting it relied heavily on seeing the pattern of reasoning before it could follow it.

The researchers then took a more drastic approach to see if the models were truly understanding the meaning of the words. They scrambled the order of the words within the facts, turning clear sentences into jumbled, nonsensical strings of text. If the models were truly reasoning about the meaning, this chaos should have made the answers impossible to find. Surprisingly, the models' performance barely dropped. They were still able to pick the correct answer even when the sentences made no grammatical sense. This suggested that the models were not reading the sentences as coherent thoughts. Instead, they were likely scanning for specific keywords that matched the answer choices. When the researchers removed those matching keywords entirely, the models' accuracy plummeted, confirming that they were often just matching words rather than deducing logic.

To rule out the possibility that the models were simply memorizing the answers from their training data, the team turned to the Bamboogle dataset, which contained questions the models had never seen before. Here, they introduced a final, rigorous test. They took the names of people, places, and dates—the specific entities that often appear in the answers—and replaced them with nonsense words. They also scrambled the order of the words in the facts. When the models were faced with these gibberish versions, the LLaMA 2 model failed almost completely. It could not find the answer without the familiar names and dates to guide it. The Flan-T5 model, however, showed a different kind of resilience. While its performance did drop, it remained significantly better than the other model. This suggests that because Flan-T5 was explicitly trained on reasoning tasks, it had developed a more robust ability to follow the logical chain, even when the familiar clues were stripped away.

The study concludes that while large language models can appear to reason, their success often depends on the specific way they were trained and the cues available in the text. For models that have not been specifically fine-tuned on reasoning tasks, the ability to answer complex questions seems to rely heavily on recognizing familiar words and patterns rather than genuine logical deduction. Even when they seem to be thinking step-by-step, they may simply be spotting the right keywords. However, the research suggests that when a model is carefully trained on datasets designed to teach it how to connect facts, it can develop a more genuine capacity for transitive reasoning. This capacity allows it to handle questions even when the usual shortcuts, like recognizable names or dates, are removed. The findings serve as a reminder that high scores on reasoning tests do not always prove a machine understands logic; sometimes, it is just very good at finding the right words.

Technical Summary: Reasoning or a Semblance of it? A Diagnostic Study of Transitive Reasoning in LLMs

Problem Statement

While Large Language Models (LLMs) demonstrate high performance on reasoning benchmarks, it remains unclear whether they engage in genuine logical reasoning or merely rely on implicit cues, such as word overlaps, named entities, or prior knowledge, to generate answers. Specifically, the mechanisms underlying transitive reasoning—the ability to deduce a conclusion ( $A \to C$ ) from two premises ( $A \to B$ and $B \to C$ )—are not well understood. Existing benchmarks may not accurately reflect a model's capacity to reason, as models might exploit superficial patterns rather than performing compositional deduction.

Methodology

The authors conducted a diagnostic study using two distinct LLM architectures: LLaMA 2 (decoder-only, 7B and 13B parameters) and Flan-T5 XXL (encoder-decoder, 11B parameters). The study utilized two compositional datasets:

QASC: A multiple-choice dataset requiring the integration of two facts to deduce an answer.
Bamboogle: A dataset of questions that cannot be answered by standard search engines, decomposed into two facts to enforce transitive reasoning. Crucially, this dataset was released after Flan-T5's pre-training, ensuring the models had no prior exposure to the specific questions.

The researchers designed a series of diagnostic experiments involving automatic and manual re-annotations to control for specific sources of information exploitation:

Prompt Variations: Comparing performance across prompts including full demonstrations with deductions, questions with no facts (QA), questions with facts but no deduction steps (QAF), and prompts with shuffled word orders.
Word/Phrase Overlap Ablation: Systematically removing connecting words between the question and facts, or between the two facts themselves, to test reliance on lexical overlap.
Keyword Removal: Removing answer keywords from the facts to determine if models rely on matching tokens between facts and answer choices.
Named Entity Manipulation: Transforming proper names, dates, and numbers into "gibberish" (randomized characters) to control for the models' reliance on specific named entities (e.g., looking for dates when the question asks "When").
Word Order Shuffling: Randomly shuffling words within facts to test if models can internally restore syntactic structure or if they rely on the sequence of tokens.

Key Results

1. Performance on QASC (Fine-tuned vs. Non-Fine-tuned)

Flan-T5 (fine-tuned on QASC) significantly outperformed LLaMA 2 across all prompts.
LLaMA 2 models struggled when demonstrations lacked explicit deduction steps (QAF prompt), whereas Flan-T5 maintained high accuracy, suggesting fine-tuning on reasoning datasets may induce reasoning-like behavior.
Word Order Insensitivity: Shuffling the word order within facts had minimal impact on performance for both models. However, follow-up experiments showed that models could not internally restore the word order of shuffled sentences when asked to do so as a standalone task. This suggests models are not syntactically parsing the facts but are extracting patterns from the token sequences.
Keyword Reliance: Removing answer keywords from the facts caused a significant drop in performance (up to 34% for LLaMA 2-7b), indicating a heavy reliance on matching answer choices to keywords in the facts rather than pure deduction.

2. Performance on Bamboogle (Controlling for Prior Knowledge)

Prior Knowledge: Low scores on the "QA" prompt (questions without facts) confirmed that neither model had seen these specific questions during pre-training or fine-tuning.
Entity Dependence: When named entities (dates, names) were converted to gibberish:
- LLaMA 2-13b performance dropped significantly (25% drop in Rouge-1), indicating a strong reliance on recognizing specific named entities to answer questions.
- Flan-T5 experienced an even larger performance decline (38% drop), suggesting that despite its higher baseline accuracy, it is also heavily dependent on named entities, though it retained higher absolute accuracy than LLaMA 2 in the original setting.
Shuffling: When facts were shuffled, Flan-T5's performance dropped notably from 96% (Full) to 77% (Both Facts Shuffled), indicating that while it is more robust than LLaMA 2, it still relies on the structural integrity of the facts to some degree.

Key Contributions

Diagnostic Framework: The paper introduces a rigorous set of manipulations (ablation of connecting words, keyword removal, entity gibbering, and word shuffling) to dissect the specific cues LLMs exploit in transitive reasoning tasks.
Differentiation of Mechanisms: The study distinguishes between models that rely on lexical cues (overlapping words, answer keywords, named entities) and those that may exhibit transitive reasoning capabilities.
Role of Fine-tuning: The results suggest that while LLMs generally rely on surface-level cues, fine-tuning on reasoning-specific datasets (as with Flan-T5 on QASC) can lead to more resilient reasoning behaviors that are less dependent on specific named entities or exact word overlaps compared to non-fine-tuned models, though they are not immune to such manipulations.
Limitations of Benchmarks: The findings challenge the validity of current reasoning benchmarks, showing that high accuracy often stems from pattern matching (e.g., finding a date in the text when the question asks "When") rather than logical deduction.

Significance and Claims

The authors claim that current LLMs often exhibit a "semblance" of reasoning rather than genuine logical deduction. Their performance on transitive reasoning tasks is frequently driven by:

Implicit Cues: Exploiting word overlaps between questions, facts, and answers.
Named Entity Bias: Relying on specific entities (dates, names) to answer questions.
Prior Knowledge: Recalling answers from pre-training data rather than deducing them.

However, the paper posits that fine-tuning on relevant reasoning datasets (like Flan-T5 on QASC) appears to foster a more robust form of transitive reasoning that is less susceptible to the removal of named entities than non-fine-tuned models, though it still relies on some structural cues. The authors conclude that while models can solve compositional questions, the mechanisms they use are often distinct from human-like logical reasoning, and future work must investigate how specific training objectives influence these capabilities. The study does not claim that models have achieved true reasoning but rather identifies conditions under which they approximate it more closely.