🤖 AI

Predictive Associative Memory: Retrieval Beyond Similarity Through Temporal Co-occurrence

This paper introduces Predictive Associative Memory (PAM), a neural architecture that utilizes temporal co-occurrence via a dual Inward and Outward JEPA framework to retrieve associative memories based on experienced temporal relationships rather than mere representational similarity, achieving high precision in recalling non-similar but co-occurring states.

Original authors: Jason Dury

Published 2026-03-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jason Dury

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Why Your Brain Doesn't Just "Google" Itself

Imagine you are walking down a slightly wet staircase. Suddenly, you feel a tiny shiver of fear. You haven't slipped in years, and these stairs look nothing like the ones you slipped on back in 2015. The lighting is different, your clothes are different, and the decade is different.

Why did you remember that old slip?

Current computer memory systems (like the ones powering search engines and AI chatbots) work like a Library. If you ask for a book, they look for books that look similar to your request. If you ask about "wet stairs," they find other books about "stairs" or "wetness." They rely on similarity.

But your brain works differently. Your brain is more like a Time-Traveling Detective. It doesn't just look for things that look alike; it looks for things that happened together. Even though the wet stairs and the old slip don't look alike, they happened in the same "story" (a moment of danger).

This paper proposes a new way to build computer memory that mimics this detective style. They call it Predictive Associative Memory (PAM).

The Two Types of Memory: The "Outward" vs. The "Inward"

The authors suggest that to have a truly human-like memory, a computer needs two different tools working together:

1. The Outward Eye (Similarity)

What it does: This looks at the world and says, "That looks like a drill." It groups things by how they look or function.
The Analogy: Think of a Filing Cabinet. You put all the "Drills" in one folder and all the "Hammer" in another. If you ask for a drill, it gives you a drill.
The Problem: It can't tell you which specific drill stripped the screw on last Tuesday. It just gives you "a drill."

2. The Inward Eye (Association)

What it does: This looks at the flow of time. It says, "I saw this drill and that screw at the same time last Tuesday." It links things based on when they happened, not how they look.
The Analogy: Think of a Movie Reel. Even if a scene with a drill and a scene with a screw look totally different, if they were filmed in the same sequence, the movie reel knows they belong together.
The Magic: This is the "Inward" part of the paper. It learns to predict: "If I am in this situation right now, what other moments from my past are linked to this?"

How the New System Works

The researchers built a computer brain that learns by watching a stream of experiences (like a video of a robot walking around a house).

The Teacher: The system is trained on Time. It learns that "State A" and "State B" are friends because they appeared next to each other in the video.
The Prediction: When the robot is in a new room (State A), the system doesn't just look for similar rooms. It asks, "What other moments in my history were right next to this moment?"
The Result: It can jump across huge gaps. It can look at a wet staircase and instantly recall a slip that happened in a totally different building, because both were linked to the feeling of danger in the timeline of its life.

The "Staircase" Test: Did it Work?

To prove this works, they created a fake world (a video game) with 20 different rooms.

The Challenge: They made the rooms look very different from each other (like a kitchen vs. a garage).
The Test: They asked the system to recall things that happened in different rooms.
The Old Way (Similarity): If you asked a standard AI, "What is associated with the garage?" it would look for things that look like a garage. It failed completely when the answer was in the kitchen.
The New Way (PAM): The new system successfully jumped from the garage to the kitchen 42% of the time, even though the rooms looked nothing alike. It found the link because the robot had walked from the garage to the kitchen in its "memory."

The Scorecard:

Standard AI: 0% success at linking different rooms.
New System: 97% accuracy on the very first guess. It knew exactly what to remember.

Why This Matters: "Hallucination" vs. "Recall"

The paper makes a very important point about what "learning" means.

Standard AI (Retrieval): Tries to guess the answer to questions it has never seen before. It's like a student who memorizes the textbook but tries to guess the answers to a test on a subject they haven't studied.
New System (Recall): Tries to remember exactly what it experienced. It's like a student who says, "I remember exactly what happened in the lab on Tuesday."

The authors argue that for a robot to have real memories, it shouldn't try to guess things it hasn't seen. It should be a perfect recorder of its own life. If it "remembers" something it never saw, that's a hallucination, not a memory.

The "Creative" Part: Connecting the Dots

The paper also hints at something cool called Creative Bridging.

Imagine you saw A go to B on Monday.
Then you saw B go to C on Tuesday.
You never saw A go to C.
But because your memory system knows the chain (A→B and B→C), it can figure out that A is connected to C.

This is how humans get creative ideas! We connect two things we've never seen together because we know the middle step. The computer system can do this too, but only if the "middle step" (the object or person) stays the same across different days.

The Bottom Line

This paper introduces a new way for computers to remember things. Instead of just finding things that look alike (like a search engine), it learns to find things that happened together (like a human memory).

Old Way: "This looks like a drill."
New Way: "This is the exact drill that stripped the screw on Tuesday."

By teaching computers to value time and experience over just appearance, we might finally build AI that doesn't just know facts, but actually remembers its life.

1. Problem Statement

Current neural memory systems, including Retrieval-Augmented Generation (RAG) and Memory Networks, rely primarily on similarity-based retrieval. These systems assume that useful memories are those that are representationally similar to the query (e.g., finding a document about "stairs" when asked about "stairs").

The paper argues this paradigm fails to capture a fundamental property of biological episodic memory: association through temporal co-occurrence. Biological memory often links events that share no visual or semantic similarity but occurred near each other in time (e.g., the feeling of slipping on damp stairs evokes a memory of a fall years ago on different stairs). Existing approaches either:

Fail to learn associative structure from raw experience (requiring task supervision or hand-built graphs).
Treat memory as isolated attractors (Hopfield networks) without inter-memory paths.
Rely on forward-looking predictive models (Successor Representations) that do not capture lateral, episodic associations.

The core problem is the lack of a differentiable, unsupervised mechanism that learns associative structure directly from the temporal stream of experience to enable faithful episodic recall.

2. Methodology: Predictive Associative Memory (PAM)

The authors propose Predictive Associative Memory (PAM), an architecture based on Joint-Embedding Predictive Architectures (JEPA).

Core Architecture

PAM utilizes two complementary predictors operating in a shared embedding space:

Outward JEPA (Standard): Predicts future sensory states from the current state. This learns semantic similarity and world models (e.g., "drills look like impact drivers").
Inward JEPA (Novel): Given a current state $s(t)$ , it predicts the region of the embedding space containing temporally co-occurring past states from the agent's history. This learns episodic association (e.g., "this specific drill was used on this specific screw on Tuesday").

Training Mechanism

Input: A composite state $s(t)$ encoding multimodal inputs (visual, auditory, proprioceptive, context).
Target: States within a temporal window $\tau$ ( $N_\tau(t)$ ).
Objective: The predictor $g_\phi$ minimizes the distance between the predicted target representation and the actual embeddings of temporally adjacent states, while maximizing distance from non-adjacent states.
Loss Function: Uses a contrastive objective (InfoNCE) to pull temporally linked states together in the latent space, regardless of their representational similarity.
Key Distinction: Unlike standard JEPAs that predict the next observation, the Inward predictor navigates the associative structure of the memory bank.

Retrieval Process

Retrieval is a forward pass through the Inward predictor:

Input current state $s_q$ .
Predictor outputs a target region $\hat{z}$ in the embedding space.
Retrieve stored states falling within this region (via nearest-neighbor lookup).
Transitive Bridging: Multi-hop retrieval allows the system to traverse chains (A→B, B→C) to reach C from A, enabling creative recombination grounded in experience.

3. Key Contributions

Inward JEPA: Introduces a predictor specifically designed to navigate associative memory via temporal co-occurrence, distinct from standard forward-prediction world models.
Unification of Memory Functions: Proposes that associative recall, imagination, and creative recombination are all instances of the same computation: predicting a target region in meaning space given a source state.
Episodic Specificity Hypothesis: Argues that true episodic specificity ("that drill on Tuesday") emerges from the intersection of Similarity (Outward) and Association (Inward). The paper validates the Inward channel in isolation.
Evaluation Paradigm Shift: Distinguishes between Retrieval (generalizing to unseen relevant items) and Recall (faithfully re-activating specific experienced associations). The paper evaluates PAM as a recall system.

4. Experimental Results

The authors evaluated PAM on a synthetic benchmark (20 rooms, 50 objects, 50k states) designed to test cross-boundary recall.

Key Metrics & Findings:

Association Precision@1: The predictor's top retrieval is a true temporal associate 97% of the time (AP@1 = 0.970).
Cross-Boundary Recall: In scenarios where query and target are in different "rooms" (embedding space clusters) and cosine similarity is zero, PAM achieves Cross-Boundary Recall@20 = 0.421. Cosine similarity baselines achieve 0.000.
Discrimination (AUC): PAM separates "experienced together" from "never experienced together" with AUC = 0.916 overall. Crucially, on cross-room pairs (where similarity fails), PAM maintains AUC = 0.849, while cosine drops to chance (0.503).
Episodic Specificity: PAM discriminates true associates from "similar-but-not-associated" distractors (same room/category) with AUC = 0.848, significantly outperforming cosine (0.732).
Ablation Controls:
- Temporal Shuffle: Randomly shuffling temporal order within trajectories collapsed Cross-Boundary Recall by 90% (0.421 $\to$ 0.044), proving the signal is genuine temporal structure, not embedding geometry.
- Held-Out Queries: When tested on query states never seen as training anchors, recall dropped to zero, confirming the system performs anchor-specific episodic recall rather than hallucinating general associations.
- Bilinear Baseline: A linear learned similarity model failed on cross-room pairs (AUC $\approx$ 0.51), confirming that non-linear transformation is required to learn cross-boundary associations.

Negative Result (Boundary Condition):
The system failed at "Creative Bridging" (A→B→C across disjoint trajectories) because the synthetic benchmark lacked entity persistence (the same object appearing in multiple trajectories). This identifies a necessary condition for transitive recombination.

5. Significance and Implications

Beyond Similarity: Demonstrates that similarity-based retrieval (the foundation of current RAG systems) is fundamentally incapable of recovering associations that lack representational overlap. PAM provides a mechanism to recover these "cross-modal" or "cross-context" memories.
Biological Plausibility: The architecture aligns with Complementary Learning Systems theory, where the hippocampus (Inward) encodes specific conjunctions of episodes, while the neocortex (Outward) learns statistical regularities.
Foundation for Embodied AI: For agents operating in continuous environments, PAM offers a way to build a memory system that learns the structure of experience without manual annotation or task-specific supervision.
Future Directions: The paper sets the stage for integrating Inward and Outward predictors to achieve full episodic specificity and suggests that future systems should incorporate entity persistence and salience signals (emotion/arousal) to refine the associative structure.

In summary, the paper validates that temporal co-occurrence is a sufficient primitive for learning a robust, differentiable associative memory that outperforms similarity-based methods in retrieving non-similar but experientially linked events.