💬 NLP

MemRouter: Memory-as-Embedding Routing for Long-Term Conversational Agents

MemRouter introduces a lightweight, embedding-based write-side router that decouples memory admission from answer generation, achieving superior accuracy and significantly lower latency compared to autoregressive LLM-based memory managers in long-term conversational agents.

Original authors: Tianyu Hu, Weikai Lin, Weizhi Zhang, Jing Ma, Song Wang

Published 2026-05-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tianyu Hu, Weikai Lin, Weizhi Zhang, Jing Ma, Song Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are having a very long conversation with a friend, one that lasts for hundreds of turns and covers thousands of topics. You want to remember the important details (like your friend's birthday, their favorite hobby, or a plan you made for next Tuesday) so you can answer questions about them later. But you can't remember everything—your brain (or in this case, the computer's memory) has a limit.

The big question is: How do you decide what to write down in your notebook and what to let fade away?

The Old Way: The Exhaustive Scribe

In recent AI systems, the computer acts like a very diligent but exhausted scribe. Every single time you say something, the AI has to stop, think deeply, and write a full essay just to decide, "Should I save this sentence?"

The paper calls this "autoregressive generation." It's like asking a professor to write a 500-word essay every time you whisper a single word to them, just to decide if that word is worth saving. It works, but it is incredibly slow and expensive. If you have a 600-turn conversation, the computer might have to write thousands of essays before it even gets to answer your actual questions.

The New Way: MemRouter (The Smart Gatekeeper)

The authors of this paper, MemRouter, propose a much smarter, faster approach. Instead of asking the AI to write an essay for every turn, they built a lightweight gatekeeper.

Think of MemRouter as a bouncer at a club or a librarian with a quick glance.

The Glance: When you speak, MemRouter doesn't write an essay. It takes a quick "snapshot" (an embedding) of what you said and the recent context.
The Decision: It runs this snapshot through a tiny, specialized filter (trained on only about 12 million parameters, which is tiny compared to the massive AI models). It instantly decides: "Yes, save this" or "No, forget it."
The Result: If the answer is "Yes," the text goes into the memory bank. If "No," it's discarded.

This happens in a flash (about 58 milliseconds) compared to the old method (about 970 milliseconds). It's like the difference between a bouncer checking your ID in a split second versus a detective interviewing you for an hour before letting you in.

How It Works (The Metaphor)

The system is split into three distinct roles, like a well-organized office:

The Gatekeeper (MemRouter): This is the new invention. It looks at every sentence you say and decides if it's important enough to file away. It uses a "frozen" brain (a pre-trained AI model that doesn't change) to understand the meaning, but it only uses a tiny, custom-built "decision head" to make the yes/no call.
The Filing Cabinet (Memory Store): This is where the saved conversations live. It keeps the exact words you said, along with who said them and when.
The Expert (Answer Agent): This is the big, powerful AI that only wakes up when you ask a specific question. It looks through the Filing Cabinet, finds the relevant notes the Gatekeeper saved, and then writes the answer.

Why This Matters

The paper tested this on a dataset called LoCoMo (Long-term Conversational Memory). Here is what they found:

Better Accuracy: Even though MemRouter is much faster, it actually remembers better than the slow, essay-writing AI. It scored higher on answering questions about facts, plans, and emotions.
Massive Speed: The "Gatekeeper" is roughly 17 times faster than the old method. This means the system can handle conversations in real-time without lagging.
Flexibility: Because the Gatekeeper is separate from the Expert, you can swap out the Expert for a smarter or different AI later without having to retrain the Gatekeeper. They work independently.

The Bottom Line

The paper argues that we don't need a super-intelligent AI to decide what to remember. We just need a smart, fast filter. By separating the "deciding what to save" job from the "answering questions" job, MemRouter creates a conversational agent that is both smarter (because it stores the right things) and faster (because it doesn't waste time overthinking every single word).

In short: MemRouter stops the AI from writing an essay to decide if a sentence is worth saving, and instead gives it a quick, efficient glance that works wonders.

1. Problem Statement

Long-term conversational agents require external memory to retain personal facts, preferences, and plans across multi-session interactions. However, current state-of-the-art systems face two critical limitations:

Per-Turn Autoregressive Cost: Existing approaches (e.g., Memory-R1, AgeMem) use the Large Language Model (LLM) itself to decide whether to store a conversation turn. This requires full autoregressive generation for every turn, leading to massive computational overhead (e.g., ~1,200 generation calls for a 600-turn conversation) and high latency.
Backbone Coupling: Memory management policies are often trained jointly with or dependent on the specific downstream answer model. This creates tight coupling, meaning upgrading the answer model (e.g., to a larger or newer LLM) requires retraining the memory management policy.

The core question addressed is: Can the decision to store a turn be made efficiently at the embedding level without invoking autoregressive generation, and can this decision be decoupled from the answer generation backbone?

2. Methodology: MemRouter

MemRouter introduces a write-side memory router that operates entirely in embedding space, decoupling memory admission from the answer generation process.

Architecture

The system consists of three main components:

Memory Router (Write Path):
- Input: Processes the current turn ( $x_t$ ) and recent history ( $H_t$ ).
- Contextual Chunking: The dialogue is partitioned into fixed-size chunks (up to 5 turns per chunk).
- Embedding & Projection: A sentence transformer (BGE-large-en-v1.5) encodes chunks. These embeddings are projected via a lightweight 2-layer MLP (with LayerNorm and GELU) into the hidden dimension of a frozen LLM backbone (e.g., Qwen2.5-7B).
- Backbone Contextualization: The projected embeddings are fed into the frozen LLM backbone via a continuous input interface (bypassing tokenization). The backbone contextualizes the sequence in a single forward pass without generating text.
- Classification: A lightweight head predicts the operation (ADD or NOOP) and, if ADD, the content type (e.g., key facts, plans, emotions).
- Parameters: Only the projection layer and classification heads are trainable (~12M parameters), representing just 0.17% of the full 7B model.
Memory Store:
- Stores admitted turns as tuples containing timestamp, speaker identity, and verbatim text.
- Uses a hybrid retrieval strategy combining dense semantic matching (cosine similarity) and sparse keyword search (BM25) to handle both semantic queries and exact entity/date matching.
Answer Agent (Read Path):
- A frozen LLM (separate from the router) retrieves the top-60 relevant memories and generates the final answer. This is the only component performing text generation.

Training Strategy

Supervised Learning: The router is trained using turn-level labels generated by a "teacher" model (Qwen3.5-35B).
Objective: Weighted cross-entropy for the operation head and standard cross-entropy for the content-type head.
No Reinforcement Learning: Unlike previous works that use PPO/GRPO, MemRouter relies on stable supervised training.

3. Key Contributions

Embedding-Based Routing: Replaces expensive per-turn autoregressive decoding with a forward-only classifier built on a frozen backbone, drastically reducing latency.
Decoupled Design: The admission policy is trained independently of the answer agent, allowing the same router to be reused across different QA backends without retraining.
Efficient Training Pipeline: Achieves high performance with only ~12M trainable parameters, avoiding the need for full-model fine-tuning or complex RL loops.
Controlled Evaluation: Provides a rigorous "matched-harness" comparison where retrieval, prompts, and the answer model are held constant to isolate the performance of the admission policy.

4. Experimental Results

Experiments were conducted on the LoCoMo benchmark, comparing MemRouter against baselines like Memory-R1, A-Mem, and Mem0.

Performance Metrics (LoCoMo Test Set)

Accuracy: MemRouter (Qwen2.5-7B) achieved an overall F1 of 52.0, outperforming the LLM-based Memory-R1 baseline (43.1) and other state-of-the-art methods.
- Single-hop: 57.5 vs. 33.6 (Memory-R1).
- Multi-hop: 52.4 vs. 23.6 (Memory-R1).
- Statistical Significance: The 95% confidence intervals for MemRouter (51–53) and the LLM manager (42–49) did not overlap, confirming a reliable improvement.
Backbone Transfer: When the answer agent was swapped to a larger Qwen3.5-35B model without retraining the router, overall F1 rose to 55.5, demonstrating the router's backbone-agnostic nature.
LLaMA Transfer: Retraining only the router heads on a frozen LLaMA-3.1-8B backbone yielded an F1 of 49.1, confirming generalizability across model families.

Efficiency Metrics

Latency: MemRouter reduced memory-management p50 latency from 970 ms to 58 ms (a 16.7x improvement).
Throughput: End-to-end throughput improved by 10.8x (2.58 QA/s vs. 0.24 QA/s).
Cost: Eliminated the need for ~1,200 generation calls per conversation, significantly lowering the serving budget.

Ablation Studies

Budget Matching: Even when constrained to store the same percentage of turns (62%) as random or heuristic baselines, MemRouter significantly outperformed them, proving the quality of its selection logic.
Factor Analysis: Learned admission contributed the largest gain (+10.3 F1 over random storage), followed by category-specific prompting (+5.2).

5. Significance

MemRouter demonstrates that write-side memory admission does not require autoregressive generation. By treating memory management as a classification task in embedding space, the paper establishes a new paradigm for long-term conversational agents that is:

Scalable: Capable of handling thousands of turns without linear growth in inference cost.
Modular: Allows independent upgrades of the memory router and the answer model.
Efficient: Makes long-horizon memory feasible for real-time, high-throughput applications where current LLM-based managers are too slow and expensive.

The code is available at https://github.com/SongW-SW/MemRouter.