💻 computer science

Prism: An Evolutionary Memory Substrate for Multi-Agent Open-Ended Discovery

The paper introduces Prism, an evolutionary memory substrate that unifies layered persistence, vector and graph-based memory, and multi-agent search under a decision-theoretic framework to significantly enhance open-ended discovery performance through entropy-gated stratification, causal memory graphs, and replicator-decay dynamics.

Original authors: Suyash Mishra

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Suyash Mishra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a massive, never-ending puzzle. You have a team of brilliant detectives (AI agents) working together. In the past, these detectives had two big problems:

They forgot things: They couldn't remember what they learned yesterday, or they remembered too much useless junk.
They worked in silos: Detective A learned something, but Detective B never knew about it, so they kept making the same mistakes.

Prism is a new "brain" for these AI teams. It's not just a filing cabinet; it's a living, breathing ecosystem that helps the team learn, evolve, and get smarter together.

Here is how Prism works, explained through simple analogies:

1. The "Three-Drawer" Filing System (Entropy Stratification)

Imagine your brain has three specific drawers for your notes. Prism forces every piece of information into one of these three drawers based on how "important" and "complex" it is:

Drawer 1: The "Skills" (Low Entropy): These are the easy, repeatable things you do every day, like "how to tie a shoe" or "how to bake a cake." Prism keeps these always open and ready to use. They are short, simple, and high-frequency.
Drawer 2: The "Notes" (Medium Entropy): These are interesting observations, like "It rained yesterday" or "The client liked the blue color." You don't need these all the time, but you need to find them quickly when the situation calls for it. Prism stores these in a smart, searchable index.
Drawer 3: The "Attempts" (High Entropy): These are the messy, complicated, failed experiments. "We tried 500 ways to fix the engine and it exploded." You usually don't want to read these, but if you need to know why something failed, Prism can dig them up.

The Magic: Prism automatically sorts every new thought into the right drawer using a math formula (Shannon Entropy). It prevents the "Skills" drawer from getting clogged with junk, ensuring the AI always has the most useful tools ready.

2. The "Who Said What?" Map (Causal Graph)

In a normal team, if someone says, "The project failed," you don't know who said it or why. Prism builds a family tree for ideas.

It draws a map where every piece of knowledge is connected to:

Who discovered it (Agent A or Agent B).
Why it happened (Cause and Effect).

If Agent A tries a strategy and fails, the map records: "Agent A tried X, which caused Y to fail." This allows the whole team to learn from one person's mistake without that person having to explain it in a meeting. It turns individual experience into shared wisdom.

3. The "Survival of the Fittest" Memory (Evolutionary Dynamics)

This is the coolest part. Prism treats memories like living organisms.

Fitness: If a memory (a fact or a strategy) helps the AI solve a problem, it gets a "fitness score" up.
Decay: If a memory is never used, its score slowly drops, like a plant without water.
Evolution: Over time, the "fittest" memories (the ones that actually work) survive and multiply. The useless ones die off.

The paper proves mathematically that this system will eventually settle into a "Perfect Memory Set" where only the best, most useful knowledge remains. It's like natural selection, but for facts and strategies.

4. The "Heartbeat" Manager

Imagine a team leader who watches a clock. Prism has a Heartbeat Controller that checks in on the team regularly.

Reflection: Every hour, it forces the agents to pause and write down what they just learned.
Consolidation: If the "Notes" drawer gets too full, it automatically turns those notes into "Skills" (simplifying them).
Redirection: If the team gets stuck (stagnation), the Heartbeat detects it. Instead of letting them spin their wheels, it says, "Hey, you've been trying this for too long. Let's pivot to a new strategy!"

5. The "Teamwork" Boost

The paper tested Prism with a team of 4 agents versus a single agent.

The Result: The team of 4 was 2.8 times faster at solving complex problems.
Why? Because they shared a "hive mind." When one agent found a solution, the others immediately learned from it. The paper calls this Knowledge Reuse. It's like having four chefs in a kitchen where they all share the same recipe book, rather than four chefs each trying to invent a new recipe from scratch.

The Bottom Line

Prism is a system that stops AI from being a "forgetful genius." It gives AI agents:

A smart filing system to organize what they know.
A map to see how ideas connect.
A survival mechanism to keep only the best ideas.
A team manager to keep them moving forward.

By combining these, Prism allows AI teams to tackle open-ended, difficult problems (like designing new drugs or optimizing complex systems) much faster and more effectively than before. It turns a collection of computers into a true, evolving team of explorers.

1. Problem Statement

The paper addresses the challenge of open-ended discovery in autonomous AI systems: the sustained, iterative search for novel solutions where progress is long-horizon and non-obvious. While recent work shows Large Language Model (LLM) agents can achieve superhuman results in specific domains (e.g., math conjectures, kernel optimization), current memory architectures fail to support this at scale due to three main gaps:

Lack of Persistence: Existing systems often lack persistent memory across sessions or rely on rigid, single-agent structures.
Inefficient Collaboration: Multi-agent systems often lack shared knowledge structures that allow agents to build upon each other's discoveries.
Absence of Evolutionary Dynamics: Current memory systems treat data as static or linearly updated, lacking mechanisms to "evolve" memories based on utility, fitness, or evolutionary selection.

Existing solutions (e.g., Claude Code's file persistence, Mem0's vector graphs, Coral's multi-agent coordination) address these issues partially but in isolation. There is no unified framework that combines information-theoretic stratification, causal reasoning, and evolutionary game theory into a single memory substrate.

2. Methodology: The Prism Architecture

Prism (Probabilistic Retrieval with Information-Stratified Memory) unifies four paradigms (layered persistence, vector memory, graph memory, and evolutionary search) into a single decision-theoretic framework comprising eight interconnected subsystems:

A. Core Memory Hub (S1 & S2)

Extraction Engine (S1): Processes agent turns and applies a four-operation update protocol (Add, Update, Delete, Noop) to a shared store. Crucially, every memory is tagged with agent provenance ( $\alpha(m)$ ) to trace knowledge origin.
Entropy-Gated Tri-Partite Hub (S2): Memories are stratified into three tiers based on Shannon Entropy ( $H$ $H$ ) and frequency:
1. Skills (Tier 1): Low entropy, high frequency. Always loaded.
2. Notes (Tier 2): Medium entropy. Retrieved via Approximate Nearest Neighbor (ANN).
3. Attempts (Tier 3): High entropy (raw logs). Accessed only when Value-of-Information (VoI) analysis demands it.
- Theoretical Guarantee: Theorem 3.1 proves that this stratification bounds context-window utilization costs.

B. Causal & Evolutionary Dynamics (S3 & S4)

Causal Memory Graph (S3): A graph $G=(V, E_r, E_c)$ where edges include relational links and causal edges annotated with do-calculus operators ( $do(\cdot)$ ). This enables multi-agent knowledge tracing and measures Exploration Divergence ( $D_{KL}$ ) between agents to ensure diverse search.
Replicator-Decay Dynamics (S4): The paper's deepest theoretical contribution. It reinterprets Bayesian confidence as evolutionary fitness.
- Fitness is defined by the success rate of a memory in downstream tasks.
- Confidence evolves via a differential equation combining replicator selection (high fitness grows), temporal decay (aging), and mutation (exploration).
- Theoretical Guarantee: Theorem 3.3 proves the system converges to an Evolutionary Stable Memory Set (ESMS), where no new memory can invade and improve the population's mean fitness.

C. Retrieval & Control (S5, S6, S7)

Evolutionary VoI Retrieval (S5): Uses a bandit-based strategy selector to learn optimal retrieval configurations (budget, cost sensitivity, diversity). It maximizes Value-of-Information while minimizing cost.
AutoDream Consolidation (S6): A five-phase pipeline (fork, distillation, conflict resolution, pruning, sync) that converts "Notes" into "Skills" by eliminating redundancy using mutual information.
Heartbeat Controller (S7): Monitors system stagnation using Optimal Stopping Theory. If confidence plateaus, it triggers:
1. Reflection: Structured observation recording.
2. Consolidation: Distilling notes into skills.
3. Redirection: Pivoting agents to unexplored regions of the solution space (maximizing Risk of Inaction).

3. Key Contributions

The paper makes five primary contributions:

Entropy-Gated Stratification: A formal mechanism assigning memories to a tri-partite hub (Skills/Notes/Attempts) based on information content, with proven bounds on context-window usage.
Provenance-Attributed Causal Graph: A graph structure with interventional edges and agent attribution, enabling the measurement of exploration divergence and multi-agent knowledge tracing.
Self-Evolving Retrieval: A Value-of-Information (VoI) retrieval policy with a bandit-based strategy selector that learns optimal retrieval strategies, with provable regret bounds.
Heartbeat-Driven Consolidation: A stagnation detection system based on optimal stopping theory that dynamically triggers reflection, consolidation, or strategic redirection.
Replicator-Decay Dynamics: A framework interpreting memory confidence as evolutionary fitness, mathematically proving convergence to an Evolutionary Stable Memory Set (ESMS).

4. Experimental Results

Prism was evaluated on two benchmark suites:

A. Conversational Memory (LOCOMO Benchmark)

Performance: Prism achieved an LLM-as-a-Judge score of 88.1, a 31.2% improvement over the previous state-of-the-art (Mem0, 67.2).
Key Drivers: The largest gains came from temporal queries (+26.6 over Mem0g), attributed to the causal graph's interventional edges.
Efficiency: Prism reduced inference latency (p95) to 0.89s while maintaining high accuracy, outperforming full-context baselines.

B. Evolutionary Optimization (Multi-Agent Tasks)

Tasks: 100-city TSP, Circle Packing, and Kernel Engineering.
Performance: A 4-agent Prism system achieved a 2.8× higher improvement rate compared to single-agent baselines.
Knowledge Reuse: There was a strong correlation ( $r=0.89$ ) between the Knowledge Reuse Rate (fraction of memories drawn from other agents) and improvement rates, validating that multi-agent gains stem from shared evolutionary learning.
Ablation: Removing the causal graph or evolutionary VoI retrieval caused significant performance drops, confirming their necessity for open-ended discovery.

5. Significance and Impact

Theoretical Unification: Prism bridges Information Theory (Shannon entropy), Bayesian Inference (confidence decay), and Evolutionary Game Theory (replicator dynamics) into a single formalism for AI memory.
From Passive to Active Memory: It shifts the paradigm of AI memory from a passive data store to an evolving knowledge ecosystem that grows, adapts, and self-improves.
Scalable Multi-Agent Discovery: By providing a formal substrate for shared memory, Prism enables agents to collaborate effectively, avoiding redundant exploration and accelerating convergence on complex problems.
Practical Applications: The architecture is particularly relevant for high-stakes domains like pharmaceutical decision-making, where causal reasoning (drug-indication-mechanism) and strategic pivoting (redirection upon stagnation) are critical.

In conclusion, Prism represents a foundational step toward autonomous AI agents capable of sustained, open-ended discovery, proving that evolutionary principles applied to memory management can significantly outperform static or purely statistical memory architectures.