HTM-EAR: Importance-Preserving Tiered Memory with Hybrid Routing under Saturation

HTM-EAR is a hierarchical tiered memory system that combines HNSW-based working memory with archival storage, importance-aware eviction, and hybrid routing to effectively preserve essential information and maintain high retrieval precision under sustained saturation, significantly outperforming traditional LRU approaches while approaching the performance of unbounded oracle memory.

The Gist

Imagine you are the brain of a super-smart robot assistant that has been working non-stop for years. It has read millions of books, watched thousands of videos, and talked to millions of people. But here's the problem: its brain has a size limit. It can't remember everything forever.

If it tries to remember everything, it crashes. If it forgets the wrong things (like "the building is on fire" or "the boss hates Mondays"), it becomes useless.

This paper introduces a new way to manage that brain, called HTM-EAR. Think of it as a smart, two-layered filing system designed to keep the robot sane and helpful.

The Two-Layer Brain

Imagine your brain has two distinct zones:

1. The "Hot Desk" (L1 Memory): This is a small, super-fast workspace right in front of you. It holds only 500 items (like sticky notes). Because it's so small, you can grab anything from here instantly.
2. The "Warehouse" (L2 Memory): This is a massive, slightly slower storage room down the hall. It holds 5,000 items (like heavy boxes on shelves). It's big, but you have to walk over to get things.

The Problem: Running Out of Space

In the old days, if your "Hot Desk" got full, you would just throw away the oldest sticky note to make room for a new one. This is called LRU (Least Recently Used).

The Flaw: Imagine you have a sticky note that says "The CEO's birthday is tomorrow." It's old (you wrote it last month), so the old system throws it away. But tomorrow, when the CEO asks, "Do you remember my birthday?", your robot says, "Nope, I forgot!" That's a disaster.

The HTM-EAR Solution: The Smart Bouncer

HTM-EAR changes the rules. Instead of just looking at when you wrote a note, it looks at how important the note is.

1. The "Importance Score" (The Bouncer)

Every time a new fact enters the "Hot Desk," the system gives it a score based on two things:

• Importance: Does this fact contain keywords like "panic," "emergency," or "CEO"? (High score).
• Usage: How often do you look at this note? (High score).

When the "Hot Desk" is full, the system doesn't just kick out the oldest note. It kicks out the note with the lowest score.

• Result: The "CEO's birthday" stays on the desk because it's important, even if it's old. The note "I had coffee at 8:00 AM" gets thrown out because it's boring and rarely used.

2. The "Hybrid Router" (The Smart Receptionist)

When the robot gets a question (a query), it doesn't just blindly search the "Hot Desk." It acts like a smart receptionist:

• Step 1: It checks the "Hot Desk" first.
• Step 2: It asks, "Is the answer here? Is it a perfect match? Does it mention the right people?"
• Step 3: If the answer on the desk is weak or missing key details, the receptionist says, "Okay, I'll go to the Warehouse (L2) to find the rest."

This ensures that if the robot can't find the answer in the fast zone, it doesn't give up; it digs deeper.

3. The "Cross-Encoder" (The Editor)

Once the robot finds a few potential answers from the desk or the warehouse, it runs them through a final "Editor." This editor reads the question and the answer together to make sure they actually fit. It's like a spell-checker that understands context, ensuring the final answer is precise.

What Happened in the Experiments?

The researchers tested this system with a massive amount of data (15,000 facts) to see what happens when the brain is completely overloaded.

• The Old Way (LRU): It was super fast, but it forgot the important stuff. It threw away 2,400 critical facts. When asked about history, it got almost everything wrong.
• The New Way (HTM-EAR): It was slightly slower (because it had to check the Warehouse and run the Editor), but it never forgot the critical facts.
  • When asked about recent events, it was perfect (100% accuracy).
  • When asked about old history, it remembered the important parts, even though it forgot the boring details.
  • On real-world data (like server logs), it performed almost as well as a robot with an infinite brain, while the old system failed miserably.

The Big Takeaway

HTM-EAR is like a smart librarian who knows that you don't need to remember every grocery list you ever made, but you absolutely must remember the location of the fire extinguisher.

It teaches us that in a world of limited memory, forgetting is necessary, but forgetting the wrong things is fatal. By using importance scores and smart routing, we can build AI agents that stay helpful and safe, even after years of running non-stop.

Technical Summary

Here is a detailed technical summary of the paper "HTM-EAR: Importance-Preserving Tiered Memory with Hybrid Routing under Saturation".

1. Problem Statement

Autonomous agents operating over extended periods accumulate vast amounts of information, but they are constrained by bounded computational memory. The core challenge is managing a streaming flow of facts where:

• Memory Saturation: The working memory (L1) fills up quickly, necessitating the eviction of older data.
• Critical Information Loss: Naive eviction policies (like Least Recently Used - LRU) discard facts based solely on access time, often deleting "essential" information (e.g., safety-critical facts or high-importance events) while retaining trivial data.
• Retrieval Efficiency: Existing solutions often assume unlimited memory or static corpora. There is a gap in maintaining high retrieval precision for recent queries while preserving essential historical facts under strict capacity limits.

2. Methodology: HTM-EAR Architecture

The authors propose HTM-EAR, a tiered memory substrate designed to balance speed, capacity, and information preservation. The system consists of three main components:

A. Tiered Memory Structure

• L1 (Working Memory): A small, fast HNSW (Hierarchical Navigable Small World) index with a capacity of 500 items. It stores the most recent and active facts.
• L2 (Archival Storage): A larger HNSW index with a capacity of 5,000 items. It stores evicted items from L1.
• Permanent Deletion: If L2 also reaches capacity, items are permanently deleted. However, if an item's importance score exceeds a threshold (0.85), its deletion is tracked as "essential loss."

B. Importance-Aware Eviction Policy

When L1 reaches capacity, items are not evicted based on recency alone. Instead, a weighted score determines eviction priority:
$$S_{evict} = \alpha \cdot \text{importance} + \beta \cdot \min\left(\frac{\text{usage}}{10}, 1\right)$$

• Parameters: $\alpha = 0.75$ (weight for importance), $\beta = 0.25$ (weight for usage frequency).
• Mechanism: Items with the lowest scores are evicted to L2. This ensures that high-importance facts are retained even if they are not frequently accessed recently.

C. Hybrid Routing and Retrieval

Queries are processed through a multi-stage pipeline:

1. Initial Search (L1): The query is encoded (using E5-large bi-encoder) and searched against L1 ($k=100$).
2. Adaptive Gating: A router evaluates the top result from L1 based on two conditions:
   • Similarity Threshold: Is the similarity score $\ge 0.84$?
   • Entity Coverage: Do all query entities appear in the retrieved item's entity set?
   • If either condition fails, the query falls back to search L2 ($k=200$).
3. Candidate Scoring & Re-ranking: Retrieved candidates are scored using a combination of similarity, entity overlap, and importance:
   $$S_{retrieve} = \text{sim}^3 + \lambda \cdot \text{overlap} + \gamma \cdot \text{importance}$$
   • The cubic transformation ($\text{sim}^3$) heavily emphasizes high-similarity matches.
   • The top 20 candidates are re-ranked by a Cross-Encoder (trained on MS MARCO) to produce the final high-precision list.

3. Experimental Setup

• Datasets:
  • Synthetic Data: 15,000 facts with queries. Facts containing keywords like "panic" are assigned high importance (0.95); others are 0.5.
  • Real-World Data: 2,000 entries from the BGL (Berkeley Gateway Logs) dataset.
• Baselines & Ablations:
  • Full System: The complete HTM-EAR architecture.
  • Oracle Unbounded: A single memory with no eviction (upper bound).
  • No CE: Full system without Cross-Encoder re-ranking.
  • No Gate: Full system without the hybrid routing (searches only L1).
  • LRU: Both tiers use standard Least Recently Used eviction.
• Metrics: Mean Reciprocal Rank (MRR) for "Active" (recent) and "History" (old) facts, Latency, and count of "Essential Facts Lost."

4. Key Results

Performance under Saturation (Synthetic Data)

• Active Query Precision: The Full System achieves a perfect MRR of 1.000 for recent facts, matching the Oracle (0.997).
• History Preservation: The Full System retains some historical context (MRR 0.215), whereas the LRU baseline drops to 0.000 (completely forgetting history).
• Essential Fact Loss:
  • Full System: 0 essential facts lost.
  • LRU: Lost 2,416 essential facts on average, despite having the lowest latency (21.1 ms).
• Ablation Insights:
  • Gating is Critical: Removing the hybrid gate ("No Gate") caused Active MRR to plummet to 0.432, proving that falling back to L2 is essential when L1 is saturated.
  • Cross-Encoder Impact: Removing the cross-encoder ("No CE") had negligible impact on MRR but reduced latency significantly (20.86 ms vs 39.69 ms), suggesting it offers marginal gains in this specific synthetic setup.

Real-World Validation (BGL Logs)

• The Full System achieved an MRR of 0.336, closely approaching the Oracle (0.370).
• The LRU baseline performed poorly (0.069), confirming that importance-agnostic eviction fails on realistic logs where query entities are sparsely distributed.

5. Key Contributions

1. Importance-Preserving Eviction: Introduced a scoring mechanism that prioritizes semantic importance over access recency, preventing the loss of critical facts in bounded memory.
2. Hybrid Routing Strategy: Developed a dynamic fallback mechanism (similarity + entity gating) that ensures queries not satisfied by the small working memory automatically consult the larger archive, maintaining high recall.
3. Tiered Architecture: Demonstrated that a two-tier HNSW system with specific eviction and routing policies can approach the performance of an unbounded memory system for active queries while gracefully forgetting irrelevant history.
4. Empirical Validation: Provided rigorous ablation studies and real-world log validation showing that LRU is insufficient for long-running agents requiring fact preservation.

6. Significance and Conclusion

HTM-EAR addresses a critical bottleneck in long-running autonomous agents: the trade-off between memory efficiency and information fidelity. The paper demonstrates that naive eviction strategies (like LRU) are fundamentally flawed for agents that must retain critical historical context.

By combining importance-aware scoring with hybrid routing, HTM-EAR achieves a Pareto-optimal balance: it maintains near-perfect precision on active queries, minimizes the loss of essential information, and gracefully degrades performance on older, less relevant history. The system is particularly significant for safety-critical applications where forgetting a high-importance fact (e.g., a system error or user constraint) could be catastrophic.

Future Work: The authors suggest exploring adaptive parameter tuning (moving beyond heuristic weights) and scaling the system to larger production environments with more diverse benchmarks. The code is open-source, facilitating further research in bounded memory management for AI agents.