💬 NLP

Exploring the Reversal Curse and Other Deductive Logical Reasoning in BERT and GPT-Based Large Language Models

This study reveals that while bidirectional BERT models are immune to the "Reversal Curse" that hinders auto-regressive GPT models in symmetric logical deduction, both architectures struggle with complex multi-set deductive reasoning, suggesting that model selection should be tailored to the specific logical demands of tasks like biomedical knowledge graph construction.

Original authors: Da Wu, Jingye Yang, Kai Wang

Published 2026-07-15

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Da Wu, Jingye Yang, Kai Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Brainy Bots and the Mirror Maze

Imagine you are teaching a robot to understand the world. You show it a picture of a cat and say, "This is a cat." Later, you show it the word "cat" and ask, "What picture goes with this?" A human brain handles this switch effortlessly; we know that if "Cat" equals "Picture," then "Picture" equals "Cat." But for a long time, the most advanced computer brains, known as Large Language Models (LLMs), struggled with this simple flip. They were like students who memorized a list of answers but couldn't figure out the question if you asked it backwards. This specific glitch is called the "Reversal Curse."

To understand why this matters, think of these AI models as two different types of readers. One type, the Decoder (like the famous GPT series), reads a story from left to right, predicting the next word based only on what came before. It's great at writing stories or finishing sentences, but it can't peek ahead to see the whole picture. The other type, the Encoder (like BERT), reads a sentence all at once, looking at every word in relation to every other word simultaneously. It's like reading a sentence with a magnifying glass that shows you the context of every single word at the same time. Scientists have been wondering: If we teach these robots to do basic math with groups of things—like combining lists of friends or finding the friends they have in common—can they actually think through the logic, or are they just guessing based on patterns they've seen before? This question is crucial because if we want to use these robots to build complex maps of knowledge (like for medicine or science), they need to be able to reason, not just repeat.

The Paper's Story: Who Can Turn the Tables?

In this study, researchers Da Wu, Jingye Yang, and Kai Wang decided to put these different robot brains to the test. They wanted to see if the "Reversal Curse" was a permanent bug or just a quirk of how certain models are built, and whether these models could actually handle more complicated logic puzzles involving sets of items.

The Mirror Test: The Reversal Curse
First, they tackled the "Reversal Curse." Imagine you teach a robot that "Jimmy Carter is the 39th President." If you ask the robot, "Who is the 39th President?", a standard robot (like GPT-3) often freezes. It knows the fact in one direction but can't flip it around to answer the reverse question. The researchers found that the Decoder models (like GPT and Llama) really do suffer from this curse; they struggle to deduce "B is A" from "A is B."

However, the Encoder model (BERT) was a different story. Because BERT reads both ways at once, it didn't get confused. When the researchers asked BERT the reverse question, it answered correctly about 99% of the time. It turns out, if you want a robot to understand that a relationship works both ways, you need a brain that can look at the whole picture, not just the past.

The Logic Puzzle: Mixing and Matching Groups
Next, the team moved on to a harder challenge: set operations. They taught the robots to perform "Union" (combining two lists of friends into one big list) and "Intersection" (finding the friends that appear on both lists).

The Easy Level (Two Sets): When the robots were asked to combine or compare just two groups of names (like "Superheroes" and "Dinosaurs"), both the Decoder models (Llama 2 and 3) and the Encoder model (BERT) did a fantastic job. They got the answers right almost every time. It looked like they had mastered the logic.
The Hard Level (Three Sets): Then, the researchers made the puzzle harder. They asked the robots to mix three groups at once, or to do a mix of operations, like "Find the friends common to Group A and Group B, then add Group C."

Here is where the magic trick fell apart. While the robots were great at the simple two-group tasks, they stumbled badly when three groups were involved.

The BERT Problem: The BERT model seemed to be cheating. It wasn't actually doing the math; it was just looking for words it recognized. If a sentence had a new name it hadn't seen before, it would immediately say "No, that's wrong." But if the wrong answer used names it did know, it would get confused and say "Yes, that's right." It was memorizing the vocabulary, not the logic.
The Llama Problem: The Llama models (the Decoder ones) also struggled. When asked to find the common friends in three groups, they often just listed everyone they knew, ignoring the specific rules of the puzzle. They seemed to be recalling a list of names they had seen during training rather than calculating the correct intersection.

The GPT-4 Wildcard
The researchers also tested the very latest, closed-source model, GPT-4. This robot was much better than the others. It could handle most of the three-set logic puzzles correctly, often breaking the problem down into smaller steps to solve it. However, even GPT-4 started to make mistakes when the number of groups got too high (like seven or eight sets), suggesting that even the smartest current models have limits when it comes to complex, multi-step deduction.

The Big Takeaway

The main lesson from this paper is that not all AI brains are built the same, and no single robot is perfect for every job. If you need a model to understand a relationship in reverse (like flipping a fact), a bidirectional model like BERT is the clear winner. But if you need a model to write a story or predict the next word in a sentence, the unidirectional models like GPT are still the kings.

Most importantly, the study suggests that while these models are amazing at spotting patterns and mimicking human language, they aren't necessarily "thinking" in the way humans do. When the logic gets too complex, they tend to fall back on memorizing words rather than understanding the rules. So, if you are building a system to create a medical knowledge graph or solve a tough math proof, you can't just trust the robot to figure it out on its own; you have to be very careful about how you ask the questions and what kind of robot you choose to do the work. The robots are getting smarter, but they still have a lot to learn about true logical reasoning.

Technical Summary: Exploring the Reversal Curse and Deductive Logical Reasoning in BERT and GPT-Based LLMs

Problem Statement

The study addresses the "Reversal Curse," a phenomenon observed in autoregressive decoder-based Large Language Models (LLMs) where the model fails to deduce the inverse relationship "B is A" despite being trained on the forward relationship "A is B." This limitation hinders the reliability of models like GPT-3 and Llama-1 in tasks requiring bidirectional logical inference, such as constructing knowledge graphs. Furthermore, the paper investigates whether these models, along with bidirectional encoder models like BERT, possess the capacity for more complex deductive reasoning, specifically the ability to perform set intersection ( $\cap$ ) and union ( $\cup$ ) operations on multiple sets without explicit training on those specific combinations.

Methodology

The authors conducted a comparative analysis between bidirectional encoder models (BERT) and autoregressive decoder models (GPT-4, Llama 2, and Llama 3) across two primary experimental domains:

Reversal Curse Evaluation:
- Dataset: The study utilized the dataset from the original Reversal Curse paper, adapted for text classification. It included synthetic pairs of names and descriptions (e.g., "Jimmy Carter" and "39th president").
- Models: BERT (fine-tuned for binary classification) was compared against GPT-3 (175B) and Llama 3.
- Task: The models were tested on their ability to infer the reverse direction (Description $\to$ Name) after training on the forward direction (Name $\to$ Description).
Set Operation Reasoning (Intersection and Union):
- Data Generation: Synthetic datasets were created using four categories of entities (Superheroes, Dinosaurs, Mammals, Birds).
- Training: Models were fine-tuned on "simple" tasks involving the intersection and union of two sets ( $A \cap B$ $A \cap B$ and $A \cup B$ $A \cup B$ ).
  - BERT: Trained on positive and negative examples for binary classification.
  - Llama 2/3: Trained exclusively on positive examples for next-token prediction (autoregressive generation).
- Testing: Models were evaluated on:
  - Simple Cases: Unseen pairs of two sets.
  - Complex Cases: Operations involving three sets, including triple intersections ( $A \cap B \cap C$ ), triple unions ( $A \cup B \cup C$ ), and mixed operations ( $(A \cap B) \cup C$ and $(A \cup B) \cap C$ ).
- Metrics: BERT was evaluated using Recall, Precision, and F1 scores. Llama models were evaluated using Exact Match and 80% Match rates. GPT-4 was evaluated heuristically as a proprietary model.

Key Results

1. Immunity to the Reversal Curse

BERT: Demonstrated high proficiency (approx. 99% accuracy) in inferring "B is A" from "A is B," confirming that bidirectional encoder models do not suffer from the Reversal Curse.
Decoder Models (GPT-3, Llama 3): Consistent with prior literature, these models showed near-zero accuracy in the reverse direction when relying solely on pre-training or fine-tuning on forward-direction data.

2. Performance on Two-Set Operations

Both BERT and Llama models performed robustly on simple intersection and union tasks involving two sets, achieving high accuracy rates (BERT ~99%, Llama slightly lower but still high).
The authors note that while performance was high, it remains unclear if the models genuinely understood the logic or merely recognized patterns in the prompt keywords.

3. Failure on Complex Three-Set Operations

General Struggle: When tested on operations involving three sets without prior specific training on those combinations, both model types exhibited significant limitations.
BERT's Bias: While BERT maintained high accuracy on positive cases, it displayed a strong bias toward vocabulary. It correctly identified negative samples containing new names (outside the training sets) but struggled when negative samples contained known names arranged incorrectly (e.g., subsets or supersets). This suggests BERT relies on term recurrence rather than semantic logical structure.
Llama's Pattern Matching: Llama 2 and 3 performed well only on the triple union task ( $A \cup B \cup C$ ), achieving up to 92% accuracy in predicting at least 80% of the target set. However, they failed significantly on intersection and mixed operations, often hallucinating or recalling all previously seen entities rather than applying the logical operator.
GPT-4: Showed better capability than Llama 2/3 in handling three-set operations, often decomposing the problem into steps. However, error rates increased with the number of sets (e.g., seven or eight sets), and performance was not perfect.

Key Contributions

Empirical Validation of Reversal Curse: Confirmed that the Reversal Curse is an architectural limitation of autoregressive decoders, while bidirectional encoders (BERT) are immune to it.
Logical Reasoning Benchmark: Demonstrated that fine-tuning LLMs on simple set operations (two sets) does not generalize to complex logical compositions (three sets).
Architectural Comparison: Provided a direct comparison showing that while decoders excel at sequence prediction, encoders may be superior for tasks requiring bidirectional context and classification of logical relationships, provided the task can be framed as such.
Bias Analysis: Identified that LLMs often rely on surface-level pattern matching (recalling seen tokens) rather than abstract logical deduction when faced with novel combinations of known entities.

Significance and Claims

The paper concludes that the selection of an LLM architecture (Encoder vs. Decoder) should be driven by the specific logical requirements of the task.

For Knowledge Graphs: The authors caution against relying solely on autoregressive decoder models for constructing medical knowledge graphs or performing rigorous logical deductions (e.g., set intersections/unions of ontology terms). The inability of these models to reliably infer reverse relationships or generalize complex set operations suggests they may act more as "writing assistants" replicating learned patterns rather than "logical machines."
For Logical Tasks: Tasks requiring reverse deduction or classification of bidirectional relationships are better suited for bidirectional encoders like BERT.
Limitations: The study highlights that current LLMs, even when fine-tuned, lack the autonomous ability to deduce complex logical sequences (like $A \cap (B \cup C)$ ) without explicit exposure to similar structures. The authors suggest that achieving reliable logical reasoning in high-stakes environments (e.g., medical diagnosis, mathematical proofs) may require more than just scaling data; it may necessitate advanced algorithms designed to enhance logical structure comprehension.