💬 NLP

MeMo: Towards Language Models with Associative Memory Mechanisms

This paper introduces MeMo, a novel language modeling architecture that explicitly memorizes token sequences in layered associative memories to enable transparency, model editing, and the ability to forget specific texts.

Original authors: Fabio Massimo Zanzotto, Elena Sofia Ruzzetti, Giancarlo A. Xompero, Leonardo Ranaldi, Davide Venditti, Federico Ranaldi, Cristina Giannone, Andrea Favalli, Raniero Romagnoli

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fabio Massimo Zanzotto, Elena Sofia Ruzzetti, Giancarlo A. Xompero, Leonardo Ranaldi, Davide Venditti, Federico Ranaldi, Cristina Giannone, Andrea Favalli, Raniero Romagnoli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to speak. The current way we do this (using "Transformers") is like forcing the robot to read a library of books and then trying to guess the next word based on a vague, fuzzy feeling of what it might have seen before. It learns by adjusting invisible knobs inside its brain until it gets the answer right. This is powerful, but it's a "black box"—we don't really know where the robot is storing a specific fact, and if we want it to forget something, we have to retrain the whole brain.

The paper "MeMo" proposes a completely different approach. Instead of guessing based on fuzzy feelings, MeMo builds a literal, organized filing cabinet for the robot's memory.

Here is how MeMo works, broken down into simple concepts:

1. The Core Idea: "Memorize First, Learn Later"

Current models try to learn patterns while they memorize. MeMo flips this. It says: "Let's just put the facts in a drawer first."

Think of it like a Library vs. a Gut Feeling.

Current Models: Like a person who has read so many books that they can guess the ending of a story because it "feels right." They might get it wrong, or they might remember a detail from a different book mixed in.
MeMo: Like a librarian who has a specific card for every sentence. If you ask, "What comes after 'The cat sat on the...'", the librarian doesn't guess; they pull the exact card that says "mat" because they wrote it down there.

2. The Magic Tool: "Correlation Matrix Memories" (The Sticky Note System)

To make this filing cabinet work, MeMo uses a mathematical trick called Correlation Matrix Memories (CMM).

Imagine you have a giant whiteboard.

The Input (The Key): You take a sentence (like "The cat sat on the") and turn it into a unique, random pattern of dots (a vector).
The Output (The Value): You take the next word (like "mat") and turn it into a different pattern of dots.
The Storage: You draw a line connecting the "Sentence Pattern" to the "Word Pattern" on the whiteboard.

If you have a million sentences, you just draw a million lines on the same whiteboard. Because the patterns are designed to be unique, the lines don't get tangled up. When you want to remember, you just point to the "Sentence Pattern," and the whiteboard lights up the connected "Word Pattern."

Why is this cool?

Transparency: You can look at the whiteboard and see exactly which sentence is connected to which word. There are no hidden secrets.
Editing: If you want the robot to forget a specific sentence, you don't need to retrain the whole brain. You just take an eraser and wipe that one line off the whiteboard.

3. The Problem with Short Memories (The Single-Layer Limit)

In the first experiment, the researchers built a single whiteboard. It worked great for short sentences (like 4 words long). But if you tried to feed it a long paragraph, the lines got too crowded, and the robot started getting confused. It was like trying to write a whole novel on a single sticky note.

4. The Solution: The Multi-Layer Tower (The MeMo Architecture)

To handle long texts, MeMo stacks these whiteboards on top of each other, like a multi-story library.

Floor 1: Remembers pairs of words (e.g., "cat" + "sat").
Floor 2: Remembers groups of four words (e.g., "The cat sat on").
Floor 3: Remembers even longer chunks.

When you ask a question, the robot checks the bottom floor, then the next floor up, and so on, combining the clues from each level to find the answer. This allows it to remember much longer sequences without getting confused, just like a human uses short-term memory to build up to a full sentence.

5. What the Experiments Showed

The researchers tested this system with random sentences and words:

Capacity: They found that the more "space" (parameters) they gave the whiteboards, the more sentences it could hold. It's a direct, linear relationship: bigger board = more memory.
Complexity: When they made the sentences harder (adding "decoys" or confusing similar-looking words), they found that more floors (layers) were necessary. A one-story library couldn't handle the confusion, but a three-story library could.
Accuracy: With enough layers and space, the system could memorize over 250,000 sequences with very high accuracy, even when the sentences were tricky.

Summary

MeMo is a new way to build language models that treats memory like a transparent, editable filing system rather than a black-box guessing game.

It stores text directly using mathematical connections.
It retrieves text by looking up those connections.
It forgets text by simply erasing the connection.

The paper claims this makes language models more honest (you can see what they know), easier to fix (you can delete specific bad memories), and capable of handling long texts by stacking these memory layers. The authors note that because this is a new architecture, it doesn't yet fit perfectly into the existing software tools used by most AI developers, but the memory capabilities themselves work very well.

Technical Summary: MeMo – Towards Language Models with Associative Memory Mechanisms

Problem Statement

Current Transformer-based Large Language Models (LLMs) achieve high performance by learning complex sequential dependencies through extensive training, where memorization is an emergent byproduct of weight optimization. While this approach excels at generalization, it often lacks transparency and explicit control over stored information. The authors argue that the prevailing paradigm treats learning as the primary mechanism, with memorization as a secondary outcome. However, for tasks requiring exact factual recall, question answering, and information retrieval, a model's ability to explicitly memorize data is crucial. Existing methods to edit or control LLMs often rely on post-hoc interventions, whereas the authors propose a fundamental shift: designing architectures where memorization precedes learning. The goal is to create a language model that explicitly stores token sequences in a transparent, editable manner, allowing for controlled forgetting and direct manipulation of stored knowledge.

Methodology

The paper introduces MeMo, a novel architecture for language modeling that replaces the standard weight-based learning of Transformers with layered Correlation Matrix Memories (CMMs). The architecture is built upon three core theoretical pillars:

Multivariate Gaussian Random Vectors: Tokens and sequences are represented as random vectors in a high-dimensional space. Leveraging the Johnson-Lindenstrauss Lemma, these vectors are mapped into a lower-dimensional space ( $R^d$ ) while preserving their distances. This ensures that distinct tokens are nearly orthogonal ( $a^T b \approx 0$ if $a \neq b$ ) and identical tokens are highly correlated ( $a^T b \approx 1$ if $a = b$ ).
Correlation Matrix Memories (CMMs): Instead of learning weights, MeMo stores associations as the sum of outer products of key-value pairs. A CMM $C$ is defined as $C = \sum k_i v_i^T$ , where $k_i$ represents a sequence (key) and $v_i$ represents the next token (value). Retrieval is performed by multiplying the input sequence vector with the CMM matrix.
Johnson-Lindenstrauss Transform (JLT): This transform is used to embed sequences of tokens into the target space $R^d$ , ensuring that both input sequences and output vectors reside in the same dimensional space, facilitating the associative retrieval process.

Architecture Variants

The paper details two configurations:

Single-Layer MeMo: Uses a single CMM to memorize fixed-length windows of tokens ( $h$ ) and predict the next token. It demonstrates that a single matrix can generalize over partial sequences based on the overlap of tokens.
Multi-Layer MeMo: To handle longer sequences, the architecture stacks multiple CMM layers.
- Memorization Phase: Lower layers encode sequences of elements into unique representations. These representations are stored as keys in the CMM, with values being the encoded sequence as a whole. The final layer stores the relationship between long sequences and the next token.
- Retrieval Phase: Intermediate layers retrieve encodings of growing sequence lengths, which are summed up to query the final layer for the next token prediction.
- Forgetting: A unique feature of MeMo is the ability to explicitly "forget" stored texts. This is achieved by subtracting the outer product of the sequence and token from the CMM matrix, effectively erasing the specific association without retraining.

Key Contributions

Paradigm Shift: The paper proposes a shift from "learning to memorize" to "memorizing to learn," designing an architecture where memory is explicit and structural rather than implicit in weights.
MeMo Architecture: The introduction of a language model based on layered CMMs and random vector representations, offering a mathematically transparent mechanism for sequence prediction.
Editability and Transparency: The design inherently supports model editing, including the specific capability to forget texts, providing a mechanism to mitigate data leaks and social biases by removing specific stored associations.
Explicit Knowledge Modeling: The architecture opens the possibility of integrating explicit knowledge modeling (e.g., knowledge graphs, linguistic ontologies) directly into neural language models, potentially reducing data hunger.

Experimental Results

The authors conducted experiments to evaluate the memorization capacity of both single-layer and multi-layer configurations:

Single-Layer Capacity: Experiments with random sequences of symbols showed a linear relationship between the total number of parameters in the CMM and the number of sequences that can be stored with high accuracy (defined as average accuracy > 0.9). This aligns with the linear scaling laws observed in traditional Transformers but applies to explicit memorization.
Multi-Layer Capacity:
- Dimensionality: Increasing the inner dimension $d$ significantly improved memorization performance. A dimension of $d=4096$ was required to maintain high accuracy (>0.97) on simple sequences, while $d=8192$ was necessary for complex sequences with high "decoy" rates (repeated tokens requiring different contexts).
- Layer Depth: Increasing the number of layers ( $l$ ) enhanced the ability to memorize longer sequences and handle complex contexts. For instance, with 40 decoys, at least three layers were required to achieve a storage capacity above 0.88 for 250,000 sequences.
- Generalization: The multi-layer approach successfully expanded the memorization capacity beyond what a single layer could achieve, validating the stacked CMM approach for longer contexts.

Significance and Claims

The paper claims that MeMo represents a significant step toward transparent and editable language models. By decoupling memorization from the opaque weight updates of standard Transformers, MeMo offers:

Control: Direct control over what is memorized, how it is stored, and how it is accessed.
Safety: The ability to mitigate data leaks and social biases through explicit forgetting mechanisms.
Integration: The potential to leverage traditional linguistic studies and explicit knowledge structures within the era of large language models.

The authors acknowledge limitations, noting that the current implementation has compatibility issues with the existing Hugging Face ecosystem, preventing direct comparison with state-of-the-art Transformers using standard evaluation suites. However, they assert that this does not diminish the demonstrated memorization capabilities of the architecture. The work aims to return control to knowledge experts and linguists, offering a path to reduce the data hunger of LLMs by making memory mechanisms explicit and manageable.