🤖 machine learning

Consolidator: Learning Persistent Routed Memory Across Context Boundaries

The paper introduces Consolidator, a lightweight shared operator that transforms short-term memory into persistent long-term memory without token replay, demonstrating that learned consolidation combined with hierarchical routing significantly improves the retrieval of updated mappings across context boundaries compared to identity accumulation.

Original authors: Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung

Published 2026-08-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your brain as a super-fast, super-organized library where new books (memories) are constantly being added to a "Just Arrived" cart. This cart is great for quick access, but it's tiny and gets cleared out the moment you walk away from the desk. In the world of artificial intelligence, specifically a type of computer brain called a Transformer, this "Just Arrived" cart is known as Short-Term Memory (STM). It holds the most recent words and ideas so the computer can understand a sentence right now. However, once the sentence is done, the computer usually forgets it to make room for the next one.

To remember things for the long haul, computers have a Long-Term Memory (LTM), like a massive, slow-moving archive in the basement. The big question scientists have been asking is: How do we move a book from the "Just Arrived" cart to the basement archive without losing the story? A simple idea is to just copy the book and put it on the shelf. But there's a catch: if you just copy a book that says "The sky is blue" and later try to update it to "The sky is red," a simple copy might just pile the new information on top of the old, creating a messy, confusing mix. The real challenge is figuring out how to learn how to update that memory correctly, so the computer knows exactly where to look and how to combine the old info with the new info, all without re-reading the original story.

This paper introduces a clever new tool called the Consolidator to solve this puzzle. Think of the Consolidator as a smart librarian who doesn't just copy books; they rewrite the summary on the fly before shelving it. The researchers tested this on a computer model called PMNet, which uses a special kind of memory that works like a set of rotating dials. They created a game where the computer had to learn a simple math rule (like "add 3 to the number") in the first round, and then change that rule to something new (like "add 7") in the second round, all while using the exact same "address" in its memory.

Here is the magic trick: After the first round, the computer's "Just Arrived" cart was completely emptied. Usually, this would mean the computer forgets everything. But because the Consolidator had already moved the rule into the long-term archive, the computer could still remember it. Even cooler, when the second round came and the rule changed, the Consolidator didn't just dump the new rule on top of the old one. It learned how to accumulate the new information with the old, effectively transforming the stored state so the "add 3" was updated to reflect "add 7" in its place.

The researchers found that when they let the Consolidator learn how to do this, the computer got the updated rule right 87.02% of the time. But when they forced the computer to just blindly copy the old memory without learning how to revise it (an "identity" approach), it only got it right 18.32% of the time. This proves that simply copying memory isn't enough; you need a smart, learned process to update it correctly.

Even more fascinating, the researchers discovered that this long-term memory does double duty. It's not just a place where facts are stored; it also acts like a GPS signal that helps the computer decide where to look next. When they turned off the part of the system that uses the long-term memory to guide these decisions, the computer's performance dropped from 87.02% down to 44.38%, even though the memory itself was still there. This means the Consolidator didn't just save the answer; it taught the computer how to find the right spot in its brain to write and read the new information.

The best part? The researchers didn't have to retrain the whole giant computer brain to make this work. They froze 99.96% of the model and only trained a tiny, specialized part—the Consolidator—which has only 12,350 adjustable settings (compared to the model's nearly 30 million total). This tiny piece of code was enough to turn a forgetful computer into one that could learn, update, and remember new rules across different "sessions" without needing to re-read the old instructions. It's a small step toward giving computers the ability to truly learn and adapt over time, just like we do.

Technical Summary: Consolidator: Learning Persistent Routed Memory Across Context Boundaries

Problem Statement

Transformer architectures typically treat context as an append-only record of token and key-value (KV) history. While writable memory systems offer an alternative by compressing experience into bounded state that can be updated and retrieved after context is discarded, a critical gap remains in how state is preserved and utilized across context boundaries.

Simply copying short-term memory (STM) into a slower long-term memory (LTM) buffer ensures persistence but does not guarantee that the retained state influences subsequent memory access or resolves conflicts when the same memory address is reused. Two mechanistic questions are central to this work:

Does a fixed, pretrained memory interface require a learned transition to revise conflicting content, or is raw accumulation sufficient?
Does retained state merely supply values to the read path, or does it also guide the router to select specific memory slots for subsequent writes and reads?

Methodology

The authors investigate these questions using PMNet (Phasor Memory Network), an architecture where tokens write phase-valued state through a hierarchical router, and memory is separable from the local attention cache. The proposed mechanism, Consolidator, operates at the boundary between context segments.

Core Mechanism:

Consolidator: A shared, slot-local operator (a gated MLP) that transforms routed STM before accumulating it into LTM. It does not replay source tokens; it operates solely on the routed latent state and an occupancy mask.
Process: After each context segment, the Consolidator applies a phase transform to occupied STM slots. The result is accumulated into LTM (modulo $2\pi$ ), after which the KV cache and STM are cleared.
Routing Integration: In subsequent segments, the retained LTM state is fed directly into the hierarchical router. This conditions the router's slot selection scores, allowing the retained state to act as an "access state" that influences which memory slots new inputs access, in addition to serving as retrievable content.

Experimental Task:
The evaluation uses a controlled two-segment modulo-10 mapping task:

Segment 1: The model learns a specific rule (e.g., $y = (x+k) \mod 10$ ) and writes it to a specific memory address.
Reset: The KV cache and STM are cleared; LTM is retained.
Segment 2: The model is presented with an updated rule for the same memory address.
Query: A held-out query targets the updated mapping. The model must recover the new rule from LTM without access to the original demonstration tokens or STM.

Interventions and Controls:

Parameter Isolation: In the primary experiment, 99.959% of the model (backbone, router, read/write projections) is frozen. Only the 12.35K parameters of the Consolidator are trainable (0.041% of the total 29.95M model).
Identity vs. Learned: Comparing learned consolidation against "forced identity" accumulation (raw copy-and-accumulate) to test if a learned transformation is necessary for revision.
Routing Ablation: Comparing conditions where LTM is fed directly to the router ("routing on") versus conditions where it is only available via the read path ("routing off").
Mismatched/Fresh Controls: Replacing the correct LTM with mismatched experiences or fresh memory to verify dependence on specific content.

Key Contributions

Shared Slot-Local Transform: A mechanism to consolidate routed latent STM into persistent LTM without replaying source tokens or requiring topology-dependent parameter growth.
Direct LTM-Conditioned Routing: An architectural inductive bias where retained LTM state directly guides subsequent slot selection through a frozen router, functioning as both retrievable content and an access state.
Sequential Same-Address Update Task: A controlled task that isolates four distinct memory functions: carrying state across a reset, revising existing memory, retrieving retained content, and using LTM to guide slot selection.

Results

The study reports results based on five paired runs from the same STM-pretraining checkpoint:

Learned Consolidation vs. Identity: On the updated-mapping recall task (after the second consolidation and reset), the learned Consolidator achieved 87.02 ± 1.76% accuracy. In contrast, the forced identity accumulation (raw copy) achieved only 18.32 ± 0.04%. This represents a gain of 68.70 ± 1.76 percentage points, demonstrating that a learned boundary transform is essential for revising conflicting content at the same address.
Routing Ablation: When direct LTM conditioning to the router was removed (routing off), the updated-mapping recall dropped to 44.38 ± 1.94%. With routing enabled, it rose to 87.02 ± 1.76%. This 42.64 ± 1.10 percentage point increase confirms that consolidated LTM significantly improves performance by guiding slot selection, not just by supplying content to the read path.
Immediate STM Recall: Immediate recall of the second mapping (before consolidation) remained high (89.90%) in both routing conditions, indicating that the routing intervention specifically affects post-reset retrieval and slot selection, not immediate context processing.
Dual Objective: A separate experiment showed that a single parameter set could support both immediate STM recall and post-reset LTM recall (95.58% and 95.76% respectively) when trained with a dual-objective loss, though the primary contribution focuses on the post-reset capability.

Significance and Claims

The paper positions these results as a mechanism-level proof of concept rather than a claim regarding natural language long-context understanding, continual learning in general, or systems efficiency.

The authors claim that:

Consolidated LTM serves a dual role: it is both retrievable content and an access state that shapes subsequent slot selection.
A fixed pretrained memory interface can effectively utilize a compact, learned adaptation substrate (the Consolidator) to revise conflicting state without changing the backbone parameters.
The mechanism distinguishes itself from continual fine-tuning or test-time gradient descent by adapting through mutable non-parametric state rather than parameter updates.

The authors explicitly note limitations: the task is synthetic (modular arithmetic), the context segments are short, and the training retains gradients across boundaries (not detached/truncated). They do not claim the system solves general long-term memory problems or scales to natural language without further investigation.