🔬 condensed matter

Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining

This paper introduces "natural ungrokking," a phenomenon where language models spontaneously learn and then abruptly forget specific rules during pretraining because the training corpus's frequency of rule-supporting evidence determines a rule's survival, a process characterized by asymmetric control where destroying a rule is easy but restoring it is impossible.

Original authors: Juliana Li, Diya Sreedhar

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Juliana Li, Diya Sreedhar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Learning a Rule, Then Forgetting It

Imagine you are teaching a very young child (the AI model) how to speak by reading them a massive library of books (the training data).

In the middle of this process, the child suddenly figures out a specific grammar rule: "If a sentence mentions a girl's name (like 'Sue'), the next pronoun should be 'she'." The child gets really good at this. If you ask them, "Sue cried because...", they confidently say "she."

But then, something strange happens. As the child keeps reading more books, they completely forget this rule. By the end of the training, if you ask the same question, they guess "he" (which is what the general crowd usually says, even if it's wrong for a girl).

The paper calls this "Natural Ungrokking." It's the opposite of "Grokking" (where a model suddenly understands something). Here, the model learns something, uses it for a while, and then silently drops it, even though the books it is reading still contain the rule.

The Mystery: Why Did It Forget?

Usually, when a computer model forgets something, it's because the data changed or the data disappeared. But in this experiment, the data never changed. The rule was still in the books the whole time.

The researchers discovered that the deciding factor is frequency (how often something happens).

The Rule: "Sue" $\rightarrow$ "she."
The Competitor: A general habit in the language where people often use "he" as a default guess.

In the specific library the model was reading, the "he" habit appeared much more often than the "Sue $\rightarrow$ she" rule. Even though the rule was there, the "he" habit was so loud and frequent that it drowned out the rule. The model didn't "erase" the rule; it just stopped listening to it because the "he" habit was winning the competition.

The Experiment: A One-Way Street

The researchers wanted to see if they could control this. They treated the training data like a dial they could turn.

1. The "Kill" Switch (Easy to Destroy)
They took a library where the rule was strong and started flipping the data. They changed every instance of "Sue... she" into "Sue... he."

Result: The more they flipped the data, the faster the model forgot the rule. It was a smooth, predictable slide from "perfect" to "zero."
Analogy: If you keep telling a student, "No, the answer is X," even when the book says "Y," the student will eventually stop believing the book and start believing you.

2. The "Rescue" Switch (Impossible to Restore)
Then, they tried the reverse. They took a model that had already forgotten the rule (because it was reading the "he" heavy library) and tried to "rescue" it. They injected massive amounts of "Sue... she" examples back into the data—300 times more than what was needed to keep the rule alive in the first place.

Result: Nothing happened. The model still forgot the rule.
The Analogy: Imagine a student who has already learned that "2+2=5" because their teacher told them so for a year. If you suddenly hand them a stack of 1,000 textbooks saying "2+2=4," they might get confused, but they won't unlearn the "2+2=5" habit. The damage is done; the "pathway" in their brain has been paved over.

The "Why": It's a Displacement, Not an Erasure

The researchers looked inside the model's "brain" (its internal math) to see what was happening.

They found that the model didn't lose the ability to understand the sentence structure. It still knew how to process the words.
Instead, the internal confidence shifted. The model's internal "vote" for "she" was overpowered by a stronger "vote" for "he."
It's like a courtroom. The "she" evidence was still there, but the "he" evidence became so loud that the judge (the model) stopped listening to the "she" witness.

The Takeaway

This paper teaches us three main things about how AI learns:

Mid-training skills are fragile: Just because a model learns a rule halfway through training doesn't mean it will keep it.
Frequency rules all: If a rule is rare in the data, it will likely get crushed by a more common, competing pattern, even if the rule is still present.
You can break it easily, but you can't fix it easily: Once a model unlearns a rule because of the data mix, simply adding the rule back in later often doesn't work. The "forgetting" is permanent for that specific training run.

In short: AI models are like students who listen to the loudest voice in the room. If the "wrong" voice is louder than the "right" rule, the student learns the wrong thing. And once they've learned the wrong thing, shouting the right rule at them later often doesn't help.

Technical Summary: Natural Ungrokking

Problem Statement

This paper investigates a phenomenon termed natural ungrokking: the within-run reversal of a learned capability in a language model during pretraining. Unlike "grokking" (delayed generalization) or "ungrokking" caused by dataset shrinkage between runs, natural ungrokking occurs on stationary data where a model acquires a specific linguistic rule, applies it successfully, and then spontaneously abandons it despite the rule's evidence remaining present in the training stream.

The focal case study involves a 11.5M-parameter decoder-only transformer trained on web text. The model learns a pronoun-gender rule (e.g., resolving "Sue cried because" to "she" rather than the corpus-prior "he"). By training step 925, the model achieves 0.94 accuracy on held-out conflict probes. However, by step 3,500, accuracy collapses to near zero, even though the training data distribution remains unchanged and the loss curve shows no trace of this behavioral shift. The model retains the grammatical construction but ceases to apply the specific rule when it conflicts with surface-level priors.

Methodology

Experimental Setup

Models: A 4-layer decoder-only transformer ( $d_{model}=256$ , 2 attention heads, 11.5M parameters) trained for 4,400 steps.
Corpora: Two datasets were used:
1. TinyStories: A synthetic corpus where the pronoun-gender rule is densely supported.
2. Filtered Web Corpus: Derived from ClimbMix, where the rule's support frequency is low relative to competing surface patterns.
Variables: The study varies the support frequency ( $f$ ) of the rule and the data-to-parameter ratio ( $D/N$ ).
Evaluation: A frozen battery of templated probes measures performance on "conflict" items (rule vs. prior disagree) and "agree" controls (rule and prior align). A classifier categorizes trajectories as Recovered, Displaced, Partial, Never, or Unstable.
Mechanism Metric: A Contrast Margin (CM) tracks the log-probability difference between the rule-conforming continuation and the prior-conforming competitor.
Rigor: All confirmatory metrics, thresholds, and directional predictions were pre-registered before data inspection. A blind replication seed set was quarantined.

Causal Interventions

To establish causality, the authors performed "in-place" edits to the training corpora:

Kill (Removal): Flipping rule-supporting tokens to counter-evidence (e.g., changing "she" to "he" after female names) at specific rates ( $p$ ) within a single corpus.
Rescue (Restoration): Injecting rule-supporting evidence back into a collapsed web corpus at doses up to $3\times$ the density required for survival in TinyStories.
Second Rule: Replicating the kill ladder on an unrelated rule ($a/an$ allomorphy) to test generalizability.

Key Results

1. Support Frequency Determines Fate

The survival of a rule is predictable from a single corpus statistic: support frequency (how often the training stream shows the rule winning).

Survival: In TinyStories (high support), the rule survives at ceiling accuracy across all data budgets.
Collapse: In the web corpus (low support), the rule emerges early but collapses to near-zero accuracy by the end of training, regardless of the $D/N$ ratio.
Modulation: The $D/N$ ratio modulates the depth of the collapse (how low the accuracy falls) but never flips the survival verdict. High data volume cannot save a rule with low support frequency.

2. Asymmetric Control (The "Kill/Restore" Asymmetry)

The paper demonstrates a stark asymmetry in controlling rule survival:

Destruction is Easy and Dose-Dependent: Flipping support to counter-evidence destroys the rule with strictly monotone dose-response. At full flip ( $p=1.0$ ), the rule dies in all seeds.
Restoration is Impossible: Injecting support back into a collapsed corpus, even at 450 $\times$ the natural sustaining density (and up to $3\times$ $3 \times$ the TinyStories density), fails to restore the behavior.
- While the internal mechanism (contrast margin) shifts slightly toward positive values at matched doses, the behavioral accuracy remains collapsed.
- This holds true even when the injection is concentrated before the collapse step, ruling out timing as the sole cause of failure.

3. Mechanism: Displacement, Not Erasure

The loss of the rule is a displacement by a competing surface pattern, not an erasure of the construction.

Evidence: The "agree" control condition (where rule and prior align) remains solved at ceiling accuracy throughout the collapse.
Contrast Margin: The CM crosses zero exactly when behavioral collapse occurs (within 100 steps). In collapsed runs, the contextual pathway (attention/MLP contributions) carries a negative margin, overriding the rule.
Representation: Cue-gender decodability in the final layer decays to chance levels on web data, indicating the representation of the rule is displaced upstream of the logits.

4. Generalization to Public Checkpoints

The phenomenon is not an artifact of the specific 11.5M model. Public checkpoints of Pythia (70M–1.4B) and OLMo-1B exhibit the same "emerge-then-collapse" signature. The depth of the collapse is ordered by model scale (smaller models collapse deeper), consistent with predictions.

Significance and Claims

The paper claims three primary contributions:

Predictability of Capability Loss: A rule's end-of-training fate can be predicted by a measurable corpus statistic (support frequency) before training ends. This challenges the assumption that loss curves are sufficient for monitoring capability retention.
Asymmetric Causality: The paper establishes that support frequency is causally sufficient for destruction but insufficient for restoration. Once a rule is displaced by a stronger prior, re-introducing evidence (even in massive quantities) cannot recover the capability.
Mechanistic Insight: The loss is identified as a physical displacement where a competing surface pattern out-competes the learned rule, measurable via a contrast margin crossing zero.

Practical Implications:

Mid-training Evaluation: Capabilities observed at mid-training checkpoints are not guaranteed to exist in the final model.
Data Filtering: Thinning the support of a rare rule can doom it without deleting a single example of it.
Continual Pretraining: Shifting data mixes can silently "ungrok" capabilities a base model had already learned.

The authors explicitly state that predicting the exact boundary on the frequency axis where survival flips remains an open problem for future work, though the qualitative law (support frequency dictates fate) is established. The work is framed as a constructive contribution to understanding training dynamics, memory, and generalization, with a specific focus on the fragility of learned rules in the presence of competing priors.