🧬 biology

Forgetting Is Not a Fix: Path Dependence in Sequential Engram Editing

This paper falsifies the Compositional Memory States Hypothesis of the AI Engram framework by demonstrating that sequential engram editing exhibits path dependence and non-commutative behavior, thereby proving that memory erasure is not a stable fix under cumulative sequential loads.

Original authors: Ferdinand M. Schessl

Published 2026-07-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ferdinand M. Schessl

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you have a giant, magical library where every book is written by a super-smart robot. This robot doesn't just read books; it learns from them, storing facts and ideas in its own "brain" made of numbers. Scientists have recently figured out how to find these specific memories inside the robot's brain and erase them, like deleting a single file from a computer. This process is called "machine unlearning," and it's a big deal because if a robot learns something it shouldn't (like a secret or a harmful fact), we need to be able to make it forget that specific thing without breaking the rest of its brain.

Think of the robot's brain like a giant, complex web of strings. When the robot learns something, it tightens certain strings. To make it forget, scientists found a way to cut just the right strings to loosen that specific memory. A recent study suggested that if you want the robot to forget two different things, it doesn't matter which one you cut first; the final result would be the same, like mixing blue and yellow paint to get green, whether you pour blue first or yellow first. They called this a "commutative manifold," which is a fancy way of saying the order of operations doesn't change the outcome. But this paper asks a very important question: What happens if you keep cutting strings over and over again? Does the brain get tired? Does the order actually matter?

This paper is like a materials scientist testing a bridge. Instead of just checking if the bridge holds one car, they drive a heavy truck over it, then another, then another, to see if the metal starts to fatigue or crack. The researchers took the exact method from the previous study and tested it on three different robot brains (called models) of varying sizes. They didn't just cut one memory; they cut a sequence of memories, one after another, and watched what happened to the ones they didn't cut.

Here is what they found, and it turns out the "order doesn't matter" idea is actually wrong when you do it repeatedly.

First, they discovered that the robot's brain is path-dependent. This means the history of what you did matters. If you cut memory A then memory B, the robot ends up in a completely different state than if you cut B then A. In fact, the difference was huge, reaching about 81% to 100% of a full composition's magnitude for closely related memories, and about 47% to 68% for more distant ones. It's like trying to fold a piece of paper: if you fold it left-then-right, it ends up in a different shape than if you fold it right-then-left. The robot's brain remembers the sequence of cuts, and the final result changes based on the order.

Second, they found that overlap makes the order even more critical. If the two things you want the robot to forget are related (like "Eiffel Tower" and "Louvre," both famous Paris landmarks), the order of cutting them changes the result almost as much as the cut itself. In one specific test, cutting the Louvre before the Eiffel Tower caused the robot to forget a third, totally unrelated thing (the Colosseum in Rome) completely. But if they cut the Eiffel Tower first, the Colosseum memory stayed safe. The order of the cuts literally decided whether a third, untouched memory survived or died.

Third, the paper shows that the robot's brain accumulates fatigue. Every time they cut a memory, the parts of the brain that weren't touched started to shift and warp. The researchers measured this using the robot's own internal "strain gauges" (mathematical tools that track how the brain's data is organized). They saw that with every new cut, these untouched parts drifted further and further away from their original state. It's like bending a paperclip back and forth; even if you don't break the part you're holding, the metal gets weaker and changes shape over time.

Finally, and perhaps most surprisingly, they found that forgetting isn't permanent in the way we thought. When they cut a second or third memory, the robot sometimes started to "remember" the first thing they tried to erase. In one case, a memory that was almost completely gone (with a score of 8.91) came back to life significantly (dropping to 4.85) just because they cut a different, unrelated memory later. The act of editing the brain to forget one thing actually pushed some of the erased knowledge back into the robot's mind.

So, what does this mean? The idea that you can erase memories in any order and get a perfect, stable result is false. If you use this method to make a robot forget something today, and then you edit it again tomorrow to forget something else, the robot might accidentally remember the first thing again. The "erasure certificate" you get today might not be valid tomorrow. The robot's brain is not a static object where you can just delete files; it's a living, shifting system where the history of your edits changes the future. The scientists didn't break the robot, but they proved that the "magic eraser" has a limit: it works great for one cut, but if you keep using it, the order matters, the brain gets tired, and the forgotten things might just come back.

Technical Summary: Forgetting Is Not a Fix: Path Dependence in Sequential Engram Editing

Problem Statement
The paper addresses a critical gap in the recent work by Kwon et al. (2026) regarding "AI Engram," a method that formalizes memory localization in neural networks as a constrained inverse problem. While Kwon et al. successfully demonstrate that single and paired memory edits can be treated as linear arithmetic operations on a "commutative manifold" (Appendix F), their evidence is limited to single-cycle tests. They explicitly defer the question of "temporal dynamics" in continual learning to future work.

The core problem investigated here is whether the hypothesis of a commutative manifold—where the integration of edits A and B reaches a consistent equilibrium regardless of sequence—holds under sequential load. The authors argue that single-cycle tests are structurally incapable of revealing fatigue accumulation, path dependence, or load-history effects, analogous to how a single stress cycle in materials science cannot reveal fatigue. The paper tests if the "forgotten state" remains a fixed point when a model undergoes repeated, sequential edits.

Methodology
The study employs a "materials testing" framing, treating the neural network weights as a specimen subjected to a fatigue protocol.

Implementation: The authors use the original authors' reference implementation (ai-engram) and the exact edit regime selected for the TOFU LLM benchmark (fixed $\alpha = 0.6$ ).
Experimental Protocol: Two distinct arms are compared across three model "charges" (Qwen3-0.6B, TinyLlama-1.1B-Chat, and Qwen2-0.5B-Instruct):
- Arm 1 (Zero-Shot Composition): All engrams are extracted from the virgin model ( $M_0$ ) and applied sequentially as linear arithmetic. This represents the theoretical "commutative manifold" operation.
- Arm 2 (Re-calibrated Sequence): After each cut, the next engram is extracted from the current model state. This represents the reality of sequential deployment (e.g., compliance-driven deletion requests arriving over time).
Concepts: Six concepts with graded semantic overlap are used (e.g., Eiffel Tower/Louvre for high overlap; Insulin/Python for distant domains).
Metrics:
- Path Dependence: Weight distance (relative Frobenius) between Arm 1 and Arm 2.
- Non-Commutativity: Weight distance between cut orders (A→B vs. B→A).
- Survivor Drift: Monotonic growth in the layer-input covariances of uncut concepts (the method's own sufficient statistics), serving as "strain gauges."
- Collateral Damage: Negative Log-Likelihood (NLL) shifts in uncut concepts.

Key Contributions

Sequential Protocol: A pre-registered, two-arm protocol testing the limits of the engram composition hypothesis using the original authors' code and parameters.
Cross-Architecture Replication: Validation across three models spanning two vendors and two architecture families (0.5B–1.1B parameters).
Falsification of the Commutative Manifold Hypothesis: The paper demonstrates that the hypothesis in Appendix F of Kwon et al. (2026) does not hold for sequential editing.
Boundary Conditions: Identification of capacity limits where collateral damage structures break down and the discovery that erased knowledge can partially return.

Results
Four primary findings replicate across all three model charges:

Path Dependence (V1): Arm 1 and Arm 2 diverge significantly. The final weight states differ by 61–71% of the total edit magnitude. The state certified by zero-shot composition is not the state of a sequentially edited artifact.
Non-Commutativity (V2): Cut order is not interchangeable. The effect is overlap-ordered: high-overlap concept pairs (e.g., Eiffel/Louvre) produce order differences up to the scale of an entire edit (weight-distance ratio 0.81–1.00), whereas distant pairs show smaller effects (0.47–0.68). In one instance, the order of cutting two Paris landmarks determined whether an uninvolved third concept (Colosseum) survived (NLL 0.85 vs. 6.50 nats).
Survivor Drift (V4): The layer-input covariances of surviving (uncut) concepts drift monotonically with every subsequent cut. This "fatigue accumulation" is recorded by the method's own sufficient statistics, proving that load history accumulates even in untouched parts of the network.
Collateral Damage and Capacity (V3): The proximity structure of collateral damage (where nearby concepts suffer more) holds only when capacity reserve exists. At higher relative edit magnitudes (specifically in the smallest 0.5B model), collateral damage becomes "broadband," affecting distant concepts indiscriminately.

Bonus Finding: The Forgotten State is Not a Fixed Point
In two of the three charges, knowledge that was successfully erased (high NLL) partially returned after subsequent, unrelated cuts. For example, in the Qwen2-0.5B charge, the NLL of an erased concept dropped from 8.91 to 4.85 nats after two subsequent cuts, moving closer to the baseline. This indicates that the "forgotten state" is not stable; redistribution of weights during new edits can inadvertently restore erased associations.

Significance and Claims
The paper concludes that the Compositional Memory States Hypothesis (Appendix F of Kwon et al., 2026) is falsified for sequential editing. While the single-edit results and the closed-form estimator of the original paper remain untouched, the claim that edits form a commutative manifold regardless of sequence is incorrect.

The practical significance lies in unlearning-as-compliance (e.g., Right to be Forgotten). The findings demonstrate that an erasure certificate issued today does not certify the artifact's state after its next edit. A model that passes a deletion audit today may partially restore the deleted association after a routine, unrelated edit. Therefore, verification must be re-run after every subsequent edit, or the editing process must be guaranteed to be stable under composition—a guarantee the current method does not provide.

The authors explicitly state they do not contest the single-edit results, the engram identification framework, or the closed-form estimator of the original work; they only falsify the specific hypothesis regarding the composition of edits in sequential regimes.

Technical Summary: Forgetting Is Not a Fix: Path Dependence in Sequential Engram Editing

More like this