💬 NLP

Memorization, Emergence, and Explaining Reversal Failures: A Controlled Study of Relational Semantics in LLMs

This study demonstrates that while logical relational semantics can emerge in shallow autoregressive LLMs with sufficient supervision, their characteristic reversal failures are primarily caused by left-to-right order bias rather than a lack of understanding of inversion logic.

Original authors: Yihua Zhu, Qianying Liu, Jiaxin Wang, Fei Cheng, Chaoran Liu, Akiko Aizawa, Sadao Kurohashi, Hidetoshi Shimodaira

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yihua Zhu, Qianying Liu, Jiaxin Wang, Fei Cheng, Chaoran Liu, Akiko Aizawa, Sadao Kurohashi, Hidetoshi Shimodaira

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart, but very literal, robot to understand the world. You want to know: Does this robot actually understand how relationships work, or is it just memorizing patterns like a parrot?

This paper is a controlled experiment to answer that question. The researchers built a "mini-universe" made entirely of synthetic facts (like "Alice is the father of Bob") to train a small AI from scratch. They then tested the AI to see if it could figure out the rules of relationships (like "If Alice is Bob's father, then Bob is Alice's son") or if it just got confused when the sentence order changed.

Here is the breakdown of their findings using simple analogies:

1. The Setup: Building a "Toy Universe"

Instead of feeding the AI the entire internet (which is messy and full of confusing examples), the researchers built a clean, controlled playground.

The Data: They created thousands of tiny stories about made-up people and jobs.
The Rules: They included specific logical rules:
- Symmetry: If Alice is Bob's friend, Bob is Alice's friend.
- Inversion: If Alice is Bob's father, Bob is Alice's son.
The Test: They trained the AI on these stories and then asked it questions about people it had never seen before, or asked questions in a different order than it was taught.

2. Finding #1: The "Lightbulb Moment" (Emergence)

The Analogy: Imagine teaching a child to solve a math puzzle. At first, they just guess. But suddenly, after seeing enough examples, their eyes light up, and they get it. They don't just guess anymore; they understand the rule.

The Result: The AI showed a similar "lightbulb moment."

When the researchers gave the AI a small amount of data, it failed. It was just guessing or memorizing specific names.
But once they crossed a certain threshold of training data, the AI suddenly "woke up." It started understanding the logic of relationships.
Surprise: This happened even in very small, "shallow" AI models (like a 2 or 3-layer brain). You don't need a massive supercomputer to understand logic; you just need enough good examples.

3. Finding #2: The "Left-to-Right" Blindness (The Reversal Curse)

The Analogy: Imagine you teach a robot to read a sentence from left to right. You show it: "The cat chased the dog." The robot learns that "cat" comes before "dog."
Now, you ask it: "Who did the dog get chased by?"
Even though the robot knows the relationship, it gets stuck. It's like a person who can only read a sentence if it starts with the subject. If you flip the sentence, the robot panics because it's used to the "left-to-right" flow.

The Result:

The researchers found that the AI did understand the logic (it knew father/son were opposites).
However, it failed miserably when asked to answer a question in the "reverse" order (e.g., asking "Who is the son of X?" instead of "X is the father of whom?").
The Cause: This wasn't because the AI was "dumb" or didn't understand the words. It was because of Order Bias. The AI is trained to predict the next word based on the previous words. When the order flips, the AI's internal "predictive engine" gets confused, even though it knows the facts.

4. Finding #3: The Magic Fix (Bidirectional Training)

The Analogy: If you only teach a driver to drive on the right side of the road, they will crash if you suddenly put them on the left side. But if you teach them to drive on both sides, they become a master driver who can handle any direction.

The Result:

When the researchers trained the AI on the same facts but in both directions (e.g., showing it "Alice is Bob's father" AND "Bob is Alice's son" during training), the "reversal curse" disappeared.
The AI could answer questions in any order perfectly.
They also tested a different type of AI (a "Diffusion" model, which looks at the whole sentence at once rather than word-by-word). This model didn't have the problem at all, proving that the issue is specific to the "left-to-right" reading style of standard AI.

5. The "Layer" Detective Work

The researchers also looked inside the AI's "brain" (its layers) to see where the thinking happened.

Success: In models that learned the logic, the "understanding" happened in the middle layers and stayed stable all the way to the end.
Failure: In models that failed, the middle layers started to figure it out, but the final layer messed it up. It was like a student who understood the math in the middle of the test but got so nervous at the end that they wrote the wrong answer.

Summary: What Does This Mean?

AI can learn logic: Even small AIs can learn the rules of relationships if they are given enough clear examples.
It's not "dumb," it's "biased": When AI fails to answer a reversed question (like "Who is the son?"), it's usually not because it doesn't understand the concept of "son." It's because it's too used to reading sentences in a specific order.
Training matters: To fix this, we don't necessarily need bigger AI models; we just need to train them on data that shows relationships in multiple directions.

In a nutshell: The AI isn't broken; it's just a very literal reader who needs to be taught that sentences can be read backwards, too!

1. Problem Statement

Autoregressive Large Language Models (LLMs) excel at relational tasks (e.g., "A is the father of B") but it remains unclear whether they truly internalize the logical semantics of these relations (such as symmetry and inversion) or merely memorize surface-level co-occurrence patterns. Two critical questions arise:

Emergence: Under what conditions do LLMs learn relational logic, and can they generalize this to unseen entities?
Reversal Curse: Why do models trained on "A is the father of B" often fail to answer "B is the son of A"? Is this due to a lack of inversion semantics or an order bias inherent in left-to-right autoregressive (AR) decoding?

Standard web-scale pretraining makes these questions difficult to answer due to data contamination, polysemy, and uncontrolled variables.

2. Methodology

The authors propose a fully controlled experimental framework using synthetic corpora derived from Knowledge Graphs (KGs) to isolate specific variables.

Synthetic Data Generation:
- KG Construction: Triples are generated with specific relation types: Inversion pairs (e.g., father/son), Symmetric relations (e.g., friend/brother), and Non-logic attributes (person/job).
- Graph Types:
  - Train Graphs: Contain full bidirectional facts.
  - Evaluation Graphs: Contain only one direction of a relation (e.g., "A is father of B" but not "B is son of A") to test logical inference.
  - ICL Graphs: Contain entirely unseen entities to test generalization.
- Verbalization: Triples are converted into natural language paragraphs using randomized templates and shuffled sentence orders to prevent rote memorization of specific phrasing.
Model Training:
- Architecture: GPT-2 style decoder-only models trained from scratch.
- Stages:
  1. Pre-training: Next-token prediction on the synthetic corpus.
  2. Supervised Fine-Tuning (SFT): Question-Answering (QA) pairs generated only from the training content (never the evaluation content) to teach the model to answer questions.
Evaluation Metrics:
- Memorize QA: Tests recall of explicit facts.
- Logic QA: Tests inference of missing inverse/symmetric facts based on observed data.
- In-Context Learning (ICL): Tests generalization to unseen entities via completion (Q_ICL-C) and QA (Q_ICL-Q).
- Forward/Reverse Tests: Compares performance when the query order matches the training order vs. when it is reversed.
- Diffusion Baseline: A bidirectional diffusion model is trained on the same data to isolate the effect of AR order bias.

3. Key Contributions & Results

A. Emergence of Relational Semantics

The study identifies a sharp phase transition in the emergence of relational logic:

Threshold Effect: Relational semantics do not emerge gradually. Instead, once the number of logic-bearing training samples ( $N$ ) exceeds a specific threshold (e.g., $N > 2,000$ for completion, $N > 20,000$ for QA), performance jumps from near-zero to near-perfect (>95%).
Shallow Models: This emergence occurs even in very shallow models (2–3 layers), suggesting that depth is not the primary driver for learning these specific logical rules.
Template Diversity: Sufficient template diversity ( $K$ ) is required alongside data scale to induce generalization.
Layer-wise Dynamics: Successful generalization correlates with stable, logic-relevant representations in intermediate layers (layers 5–8). Models that fail exhibit unstable signals in intermediate layers that degrade at the final layer.

B. The Reversal Curse: Order Bias vs. Semantics

The paper definitively answers the cause of the "Reversal Curse":

Primary Cause: Reversal failures are driven by autoregressive order bias (left-to-right decoding), not a lack of inversion semantics.
- Evidence 1: Symmetric relations (which have no logical inversion) show the same performance gap between forward and reverse queries as inverse relations.
- Evidence 2: When the model is trained on bidirectional data (both "A is father of B" and "B is son of A"), the forward/reverse gap disappears.
- Evidence 3: Diffusion models (which are bidirectional and non-autoregressive) show no reversal curse, performing equally well in both directions.
Conclusion: The model does learn the logical semantics, but the AR decoding process struggles to utilize them when the query order contradicts the training order.

C. Generalization to Pre-trained Models

The phase-transition phenomenon was replicated on a pre-trained GPT-2 small model continued on the synthetic data, confirming that the emergence of relational logic is a general property of AR models given sufficient rule-consistent supervision, not just an artifact of random initialization.

4. Significance and Implications

Mechanism of Logic Learning: The study challenges the view that LLMs only learn surface patterns. It demonstrates that with sufficient data density, AR models can internalize abstract logical rules (symmetry/inversion) and generalize them to unseen entities.
Diagnosis of Reversal Failures: It shifts the blame for the Reversal Curse from "semantic deficiency" to "architectural bias." This suggests that mitigation strategies should focus on bidirectional training exposure or architectural changes (like diffusion) rather than trying to teach the model "inversion" as a new concept.
Interpretability Insights: The layer-wise analysis provides a new diagnostic tool: stable intermediate-layer signals are a precursor to successful logical generalization, while final-layer degradation indicates a failure to integrate these rules.
Methodological Contribution: The framework of using controlled KG-synthetic data offers a robust testbed for future research into LLM reasoning, free from the confounds of real-world pretraining data.

5. Limitations

Relation Scope: The study focuses on symmetric and inverse kinship relations; more complex relations (transitivity, multi-hop) may have different thresholds.
Template Bias: While diverse, the data is template-generated and may not capture the full nuance of natural language.
Diffusion Maturity: The diffusion baseline results are qualitative regarding the reversal curse; a full quantitative comparison across all tasks is future work.

In summary, the paper proves that relational logic emerges abruptly in AR models with sufficient data, but their ability to apply this logic is severely constrained by the left-to-right order bias of the autoregressive generation process.