💬 NLP

Exemplars in Disguise: Pure Exemplar Models Mimic Abstraction-First Learning

This paper challenges the claim that large language models learn abstract knowledge before item-specific details by demonstrating that pure memorization models can mimic such learning patterns depending on input distribution, suggesting that the distinction between item-specific and abstract knowledge may be ill-defined for distributed representations.

Original authors: Zachary Nicholas Houghton, Vsevolod Kapatsinski

Published 2026-08-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zachary Nicholas Houghton, Vsevolod Kapatsinski

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to speak. You have two big theories about how it learns. The first theory, let's call it the "Rulebook Theory," says the robot first figures out the big, abstract rules of language—like how all verbs that mean "giving" work together—before it memorizes specific words. The second theory, the "Memory Book Theory," says the robot just memorizes every single sentence it hears, one by one, and only later does it notice patterns by comparing its massive pile of memories.

For a long time, scientists argued about which one was right. But recently, a new study looked at giant AI models (like the ones that write essays or chat with you) and claimed to find proof that the "Rulebook Theory" is correct. They said these AIs learn the big rules before they learn the specific quirks of individual words. This paper, however, is like a detective stepping in to say, "Wait a minute. Let's look closer at the clues." The authors ran their own experiments using simple computer simulations to see if the evidence really proves the robot is thinking in rules, or if it is a result of how the robot pays attention to new information.

The Great Heist: Rules vs. Memories

In the world of language learning, there's a classic tug-of-war. On one side, you have Abstraction. This is like learning the concept of "fruit." Once you know what a fruit is, you can guess that a new, weird-looking thing you've never seen before is probably a fruit, even if you've never tasted it. On the other side, you have Exemplars (or specific memories). This is like remembering every single apple, banana, and orange you've ever eaten. You don't need a rule for "fruit"; you just compare the new thing to your mental pile of apples and oranges. If it looks like an apple, you call it a fruit.

The big question is: Which comes first? Do we learn the general rule first, or do we pile up specific memories first?

Recently, researchers looked at a famous AI model (GPT-2) and saw something interesting. They watched the AI learn over time. They noticed that the AI started grouping similar verbs together (like "give" and "sell") before it started noticing the tiny differences between them (like how "give" is used slightly differently than "sell"). They called this "Abstraction-First" learning. It seemed like the AI was building a rulebook before it filled its memory book.

The Plot Twist: The "Sensitivity" Switch

The authors of this paper, Zachary Nicholas Houghton and Vsevolod Kapatsinski, decided to test if this "Abstraction-First" pattern was a real sign of a rulebook, or just a side effect of how the AI pays attention.

To do this, they built two simple, fake learners in a computer simulation. These weren't fancy AIs with hidden rulebooks. They were pure memorizers. They had no way to create abstract rules; they could only remember what they saw.

The "Zero-Sensitivity" Learner: Imagine this learner is a super-smooth, calm robot. When it sees a new word, it doesn't get excited or jittery. It blends the new information slowly and perfectly into its memory, ignoring any weird, one-off mistakes. It's like a calm painter blending colors on a canvas without splashing.
The "Variable-Sensitivity" Learner: This learner is a bit jittery. It reacts strongly to every single new word it sees. If it sees a weird word early on, it gets very excited about it. How jittery it is depends on a setting called k (think of it as a "jitter knob"). If k is low, the learner is super sensitive to every little detail. If k is high, the learner is very calm and ignores the little details, smoothing them out.

The Big Discovery

Here is the magic trick the paper reveals: The order in which these learners "learn" depends entirely on how jittery they are, not on whether they have rules.

When the learner is jittery (Low k): The "Memory Book" wins. The learner gets distracted by the first few weird words it sees. It starts noticing the differences between specific words (like "give" vs. "sell") before it notices the big group they belong to. This looks like "Exemplar-First" learning.
When the learner is calm (High k): The "Rulebook" wins. Because the learner ignores the early, noisy details and waits for a pattern to emerge, it starts grouping the words together before it notices their tiny differences. This looks exactly like "Abstraction-First" learning.

The paper shows that even though these learners are pure memorizers with zero ability to create abstract rules, they can still look like they are learning rules first if they are just calm enough (high k).

What This Means for AI

The authors argue that the previous study claiming AI learns rules first might have been influenced by this factor. They suggest that the AI model (GPT-2) they analyzed was probably just acting like a "calm memorizer" (a high k learner). It wasn't necessarily building a secret rulebook; it was just smoothing out the noise of its early training data so well that it appeared to learn the big patterns first.

In fact, the paper suggests that for these kinds of distributed systems, the line between "remembering a specific word" and "knowing a general rule" might be blurry. The "rule" might just be the average of all the memories, and the "memory" might just be a specific point in that average. You can't really separate them.

The Verdict

So, does this paper prove that AI learns rules first? No. In fact, it suggests that the specific test used to prove it (looking at the order of when patterns appear) isn't a reliable test. A pure memorizer can pass that test just by being calm enough. The paper does not conclude that the pattern is definitely an artifact; rather, it concludes that the evidence provided by the previous study is insufficient to distinguish between true abstraction-first learning and a high-sensitivity exemplar model. The authors state that whether transformer models are truly abstraction-first learners remains an open question, but answering it will require new methods beyond just looking at the order of onset. Until then, the "Rulebook vs. Memory Book" debate is still wide open, and the evidence we thought we had might be a result of how the data is processed.

Technical Summary: Exemplars in Disguise: Pure Exemplar Models Mimic Abstraction-First Learning

Problem Statement

A central question in language learning and cognitive science concerns the temporal ordering of knowledge acquisition: do learners first acquire abstract, class-level generalizations (e.g., verb argument structures) or do they first accumulate item-specific, idiosyncratic knowledge (e.g., specific verb-token collocations)? Recent work by Jian and Manning (2026) analyzed learning trajectories in GPT-2 and concluded that abstract knowledge emerges before item-specific knowledge ("abstraction-first"). They operationalized this by measuring the onset of between-class differentiation ( $o_b$ ) versus within-class divergence ( $o_w$ ), finding that $o_b < o_w$ for certain verb classes.

However, this conclusion relies on the assumption that the observed ordering is diagnostic of an inductive bias toward abstraction. The authors of this paper argue that this ordering may instead be an artifact of the learner's sensitivity to individual observations and the distributional properties of the input, rather than evidence of a distinct abstract learning mechanism. Specifically, they question whether the distinction between "item-specific" and "abstract" knowledge is even well-defined for distributed representations, where class-level properties may be inextricably embedded within item-specific distributions.

Methodology

The authors conduct controlled simulations using two idealized learner models that share an identical data-generation process but differ in how they estimate verb distributions from observations. Crucially, both models are pure memorizers (exemplar models) with no information flow between verbs and no explicit class-level representations.

Data Generation:
- Two verb classes (35 and 36 items) are simulated with a vocabulary of 1,000 tokens.
- Each verb's true distribution ( $P_v$ $P_{v}$ ) is constructed from three token types:
  - Cross tokens: Shared by all verbs (background).
  - Within-class tokens: Shared only by verbs in the same class (encoding class-level preferences).
  - Idiosyncratic tokens: Unique to each verb (encoding item-specific preferences).
- Parameters control the overlap between classes and the variance of token weights ( $\mu, \sigma$ ).
Learner Models:
- Zero-Sensitivity Learner (ZSL): A noiseless analytical baseline. The learner's estimate is a linear interpolation between a uniform distribution and the true distribution ( $P_v$ ). It is unaffected by the specific identity of individual tokens, representing a limit where observation-level noise is removed.
- Variable-Sensitivity Learner (VSL): A Bayesian Dirichlet-Multinomial learner formalizing add- $k$ $k$ smoothing. The parameter $k$ $k$ controls sensitivity:
  - Low $k$ : The learner is highly sensitive to early, potentially unrepresentative observations (idiosyncratic noise dominates).
  - High $k$ : The learner is conservative; the prior suppresses early noise, allowing stable class-level patterns to accumulate gradually.
Onset Criteria:
- Between-class onset ( $o_b$ ): Declared when a verb's distribution is significantly more similar to same-class verbs than to opposite-class verbs (Mann-Whitney U-test).
- Within-class onset ( $o_w$ ): Declared when the mean Jensen-Shannon divergence (DJS) within a class exceeds a fixed threshold (0.01).
- The study traces the learning trajectory to determine which onset occurs first ( $o_b < o_w$ or $o_w < o_b$ ).

Key Contributions

Demonstration of Sensitivity-Dependent Ordering: The authors show that the ordering of $o_b$ and $o_w$ is not a fixed property of "abstract-first" vs. "exemplar-first" learning strategies. Instead, it depends on the interaction between learner sensitivity (controlled by $k$ ) and the input distribution's structure.
Noiseless Baseline: The introduction of the Zero-Sensitivity Learner demonstrates that when observation-level noise is removed, $o_b < o_w$ holds universally, even in a pure memorizer with no abstraction.
Replication of "Abstraction-First" via Smoothing: The Variable-Sensitivity Learner replicates the "exemplar-first" ordering ( $o_w < o_b$ ) at low $k$ (high sensitivity) but transitions to the "abstraction-first" ordering ( $o_b < o_w$ ) at high $k$ (low sensitivity). This suggests that the "abstraction-first" pattern observed in previous studies may result from high smoothing (low sensitivity) rather than an inductive bias.

Results

Zero-Sensitivity Learner: Produced $o_b < o_w$ in 85% of seeds overall. The ordering was universal for low noise ( $\sigma=0.5$ ) and held across most parameter combinations, failing only when class signals were extremely weak relative to noise.
Variable-Sensitivity Learner:
- At low $k$ ( $k \le 0.1$ ), $o_w < o_b$ occurred in ~99% of seeds. Early idiosyncratic observations caused immediate within-class divergence before class-level patterns could stabilize.
- At high $k$ ( $k \ge 1.0$ ), $o_b < o_w$ predominated. The strong prior suppressed early noise, allowing the systematic between-class differences to emerge before the within-class idiosyncrasies accumulated enough to cross the absolute threshold.
- The transition point occurred around $k=0.5$ , where both orderings were equally likely.
Robustness: The results were robust across variations in vocabulary size, verb frequency distributions (Zipfian extension), and the specific thresholds used for onset detection.

Significance and Claims

The paper claims that the "abstraction-first" ordering ( $o_b < o_w$ ) is not diagnostic of an abstraction-first learning strategy. Instead, it is a predictable consequence of low sensitivity to individual observations.

Reinterpretation of LLM Trajectories: The findings suggest that the results of Jian and Manning (2026) do not prove that GPT-2 learns abstract rules first. A pure exemplar memorizer with sufficient smoothing (high $k$ ) produces the same apparent ordering. The authors note that transformers may naturally behave like high- $k$ learners due to adaptive partial pooling, which suppresses idiosyncratic noise, but this mechanism does not necessarily imply the existence of discrete, separable abstract representations.
Ill-Defined Dichotomy: The authors argue that for distributed representations, the distinction between item-specific and abstract knowledge may be ill-defined. Class-level properties are embedded within the item-specific distribution; learning the distribution of a specific verb inherently encodes its class membership. Therefore, treating between-class and within-class divergence as separable learning events may impose a conceptual distinction that the representations themselves do not support.
Methodological Limitation: The current methodology of ordering onset criteria is insufficient to distinguish between abstraction-first learning and high- $k$ exemplar learning. Answering whether transformers are truly abstraction-first learners requires methodologies that go beyond the ordering of onset criteria.