🤖 AI

MemForest: An Efficient Agent Memory System with Hierarchical Temporal Indexing

MemForest is an efficient agent memory system that addresses scalability and latency issues in long-context LLMs by introducing parallel chunk extraction and a hierarchical temporal index called MemTree, which replaces costly full-state rewrites with localized updates to achieve superior performance and throughput on memory benchmarks.

Original authors: Han Chen, Zining Zhang, Wenqi Pei, Bingsheng He, Ming Wu, Jason Zeng, Michael Heinrich, Wei Wu, Hongbao Zhang

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Han Chen, Zining Zhang, Wenqi Pei, Bingsheng He, Ming Wu, Jason Zeng, Michael Heinrich, Wei Wu, Hongbao Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are talking to a very smart friend (an AI agent) who is trying to remember everything you've ever said to them over months or years. The goal is for this friend to remember not just what you said, but when you said it, so they can answer questions like, "Where did you live before you moved to New York?" rather than just "Where do you live now?"

The paper introduces a new system called MemForest to solve a major problem: current AI memory systems are slow, clumsy, and expensive to update.

Here is how MemForest works, explained through simple analogies:

The Problem: The "All-or-Nothing" Mess

Current memory systems work like a person trying to update a single, massive diary entry every time you say something new.

The Bottleneck: Every time you add a new fact, the system has to read the entire diary, rewrite the whole thing to include the new fact, and then save it.
The Result: As your conversation history grows, this process gets slower and slower. It's like trying to edit a 1,000-page book every time you want to add a single sentence. It creates a huge delay (latency) and costs a lot of computing power.
The Confusion: Because these systems often just summarize the "latest" state, they forget the history. If you moved from Boston to New York, and then to San Francisco, a simple summary might just say "Lives in San Francisco." If you ask, "Where did you live before San Francisco?", the system might guess wrong or say "Boston," forgetting the middle step (New York).

The Solution: MemForest

MemForest changes the game by treating memory not as a single diary, but as a growing forest of trees.

1. Parallel Picking (The Extraction)

Instead of one person reading the whole conversation to find facts, MemForest sends out a team of workers.

Analogy: Imagine a library where, instead of one librarian reading every book cover-to-cover to find a quote, they slice the books into small chapters and have ten librarians read those chapters at the same time.
Benefit: This makes the initial "picking" of information incredibly fast because everyone works at once.

2. The "Canonical Fact" (The Standardized Brick)

Once the workers find a piece of information, they don't just throw it into a pile. They turn it into a standardized "brick" called a Canonical Fact.

Analogy: Instead of getting a messy pile of raw clay, the system turns every memory into a uniform Lego brick. This ensures that whether the fact came from 5 minutes ago or 5 months ago, it fits perfectly into the structure.

3. The MemTree (The Organized Forest)

This is the core innovation. MemForest organizes these Lego bricks into MemTrees.

How it works:
- Leaves: The bottom of the tree holds the specific, raw facts (e.g., "Bob moved to Miami in July 2024").
- Branches: The middle of the tree holds summaries of groups of facts (e.g., "Bob's moves in 2024").
- Root: The top of the tree holds a high-level summary (e.g., "Bob's residence history").
The Magic of Updates: When a new fact arrives, MemForest doesn't rewrite the whole tree. It simply adds a new leaf and updates the branch directly above it.
- Analogy: Imagine a family tree. If a new baby is born, you don't redraw the entire family tree from scratch. You just add the baby's name to the bottom and update the parents' branch. The rest of the tree (grandparents, cousins) stays exactly the same.
Benefit: This makes updates lightning fast, regardless of how big the memory gets.

4. The "Time-Travel" Search

When you ask a question, MemForest doesn't just look for keywords. It navigates the tree.

Analogy: If you ask, "Where was Bob before Miami?", the system doesn't just scan a list. It climbs down the "Bob" tree, looks at the "Miami" branch, and then looks at the branch immediately before it to find "Davis."
Benefit: It preserves the timeline perfectly, allowing the AI to answer complex questions about the past, not just the present.

The Results

The authors tested MemForest on two difficult memory tests (LongMemEval-S and LoCoMo).

Speed: MemForest was about 6 times faster at updating its memory than the best existing systems.
Accuracy: It achieved the highest accuracy (79.8%) among systems that try to remember long-term context.
Efficiency: It solved the "slow update" problem without losing the ability to remember the details of the past.

Summary

MemForest is like upgrading from a clumsy, slow-moving scribe who rewrites a whole book every time you add a sentence, to a modern, organized library where new books are instantly shelved in the right spot, and the librarian can instantly find the exact page you need, even if it was written years ago. It makes AI agents faster, cheaper to run, and much better at remembering the full story of your life.

Technical Summary: MemForest

1. Problem Formulation

The paper addresses the inefficiency of existing Long-Context Large Language Model (LLM) agent memory systems. While prior work has improved retrieval quality and storage organization, these systems suffer from significant maintenance overhead due to two structural bottlenecks:

Sequential Extraction and Maintenance: Existing architectures often embed the LLM directly into the write critical path. New dialogue chunks must be synchronously extracted, summarized, and reconciled against existing memory, forcing a serial execution that creates latency bottlenecks as interaction frequency increases.
Coarse-Grained State Management: Current systems frequently rely on full-state rewrites of compact "hot" states (e.g., user profiles or global summaries). Even when only minor new evidence arrives, the system must reread and rewrite the entire memory object. This imposes a maintenance cost that scales with the total accumulated memory size rather than the size of the new evidence.

Furthermore, the paper argues that long-context memory is inherently temporal. User states evolve, facts are revised, and historical transitions (e.g., moving from City A to City B) are necessary for complex reasoning. Existing systems often fail to preserve these temporal trajectories, either by losing intermediate states during compression or by retrieving evidence from the wrong time point due to a lack of temporal ordering in vector-based retrieval.

The core challenge is defined as a write-efficient temporal data management problem: creating a system that minimizes serial delays and state-size-dependent costs while rigorously preserving historical state evolution for long-context reasoning.

2. Methodology: MemForest Architecture

MemForest reframes agent memory as a write-efficient temporal data management problem, drawing inspiration from write-optimized indexing in database systems (e.g., LSM-trees). The architecture consists of three primary components:

A. Shared Memory Substrate

MemForest decouples persistent state from derived access artifacts:

Persistent State: The source of truth, containing canonical facts. These are stable, temporally anchored units of memory (normalized surface forms, source references, entity mentions) extracted from raw dialogue.
Derived Artifacts: Summaries, node embeddings, and root-index rows generated from the persistent state. These can be selectively regenerated after local edits, avoiding the need to rewrite the entire memory object.

B. Parallel Chunk Extraction

To break the sequential bottleneck, MemForest partitions incoming sessions into fixed-size extraction chunks (e.g., two-turn chunks). These chunks are processed concurrently to extract local memory candidates. The outputs are then canonicalized into stable facts. This approach avoids a single serialized LLM pass over the entire session.

C. MemTree: Hierarchical Temporal Index

Instead of flat global summaries or unordered fact stores, MemForest organizes memory as MemTrees—hierarchical, time-ordered trees scoped to specific entities, scenes, or sessions.

Structure: Leaves store time-local evidence (canonical facts or source dialogue cells). Internal nodes summarize contiguous time intervals. The root provides a coarse representation for forest-level recall.
Write Path: New evidence is inserted as time-ordered leaves. Maintenance is localized: only the ancestor paths of the inserted leaves are marked "dirty" and refreshed. This reduces the maintenance cost from $O(N)$ (full rewrite) to $O(\log N)$ (affected tree paths).
Retrieval: MemForest employs a coarse-to-fine retrieval strategy. It first recalls relevant trees based on root summaries (forest-level recall) and then browses down the temporal hierarchy from interval summaries to specific leaf evidence. This ensures that queries requiring historical context (e.g., "Where did the user live before moving to X?") can navigate the temporal structure rather than relying solely on semantic similarity.

3. Key Contributions

The paper identifies three main contributions:

Identification of Structural Limitations: The authors pinpoint serial LLM-in-the-loop extraction and state-size-dependent latency of full-state maintenance rewrites as the dominant limitations in current long-context agent memory systems.
MemForest Architecture: A novel system combining parallel extraction with hierarchical temporal indexing (MemTree). This enables localized updates, variable-granularity retrieval, and a persistent, queryable memory substrate that evolves temporally under continuous writes.
Empirical Validation: The authors demonstrate that this architectural shift improves the speed–accuracy trade-off. MemForest achieves superior performance on LongMemEval-S compared to stateful baselines while significantly reducing write-path costs.

4. Experimental Results

The system was evaluated on two benchmarks: LongMemEval-S and LoCoMo.

LongMemEval-S:
- Accuracy: MemForest achieved the best overall pass@1 accuracy among stateful baselines, reaching 79.8% (using a Qwen3-30B backbone).
- Efficiency: It sustained a memory construction throughput approximately 6× higher than EverMemOS (the strongest stateful baseline) and achieved a 13.7× speedup in wall-clock write time compared to MemoryOS.
- Trade-off: The system maintains high accuracy while drastically reducing the latency of turning new dialogue into queryable memory.
LoCoMo:
- MemForest remained competitive, particularly on temporally structured long-context question answering. While EverMemOS achieved slightly higher overall accuracy on this more complex benchmark, MemForest performed strongly on open-ended and temporally grounded queries, with substantially reduced write-path costs.
Ablation Studies:
- Lazy Maintenance: Batch "dirty-path" refresh significantly reduced redundant LLM calls compared to eager per-insert regeneration.
- Tree Structure: A moderate branching factor ( $k \approx 16$ ) balanced summary capacity and retrieval accuracy.
- Retrieval Modes: Planner-guided LLM browsing yielded the highest accuracy (90.0% pass@8 on a subset), while embedding-only browsing offered a low-latency alternative (85.0%) with minimal accuracy loss.

5. Significance and Claims

The paper claims that MemForest represents a shift in how persistent agent memory should be designed. By treating memory as a write-efficient temporal data management problem, the system overcomes the traditional trade-off between write efficiency and faithful temporal representation.

Scalability: The system avoids the "maintenance floor" where latency scales with total memory size. Instead, maintenance costs scale with the size of the new evidence and the affected tree paths.
Temporal Fidelity: By materializing memory as time-ordered trees rather than flat summaries, MemForest preserves state transitions and historical evidence necessary for complex reasoning, which is often lost in profile-based systems.
Lifecycle Efficiency: The separation of persistent state from derived artifacts enables efficient migration and merging of memory states without replaying raw session streams, a capability not found in comparable baselines.

The authors conclude that scalable long-context memory systems must be designed around write efficiency, explicit temporal preservation, and localized maintainability under continuous updates. MemForest demonstrates that these goals are achievable without sacrificing answer quality.