💬 NLP

Total Recall at What Cost? Benchmarking the Serving Cost of Agentic Memory Systems

This paper benchmarks the serving costs and accuracy of three agentic memory systems against standard strategies, revealing that costs are driven by internal system behaviors rather than conversation length alone, vary significantly across different backbones and systems, and involve a trade-off where no single solution optimizes both cost and accuracy.

Original authors: Natchanon Pollertlam, Witchayut Kornsuwannawit

Published 2026-08-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Natchanon Pollertlam, Witchayut Kornsuwannawit

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are talking to a very smart, but slightly forgetful, robot friend. You've been chatting for weeks, sharing stories, plans, and secrets. Now, you want to ask, "Remember that pizza place we talked about three weeks ago?" If the robot has no memory, you have to re-tell the entire story of your friendship from day one every single time you ask a question. This is like sending a massive, heavy backpack full of every word you've ever said to the robot just to ask a simple question. It works, but it gets heavy, slow, and expensive very quickly.

To fix this, engineers built "memory systems" for these robots. Instead of carrying the whole backpack, the robot writes down tiny notes, facts, or summaries in a special notebook. When you ask a question, it just flips to the right page and reads the note. This sounds perfect: lighter, faster, and cheaper. But here is the catch: writing the notes, organizing them, and finding the right page takes work too. The big question is: Does the robot save money by using the notebook, or does the cost of managing the notebook actually make the whole thing more expensive than just carrying the heavy backpack? This paper dives into that exact mystery, measuring the real-world price tag of these memory tricks to see if they are actually worth the cost.

The Great Memory Cost Showdown

In this study, the researchers set up a massive race to see how much it really costs to keep a chatbot's memory alive. They didn't just guess; they ran a controlled experiment with three different "memory systems" (think of them as three different ways of organizing that notebook): Mem0, Hindsight, and Mastra Observational Memory.

To make sure the test was fair, they compared these memory systems against two extreme strategies:

The "Rolling Window" (The Cheap Baseline): This is like only remembering the last 10 things you said. It's super cheap but forgets everything else.
The "Full Transcript" (The Expensive Baseline): This is the "heavy backpack" method. Every single time you ask a question, the robot re-reads the entire history of your conversation from the very first word.

They ran these systems through conversations of up to 400 turns (that's 400 back-and-forth messages) and tested them on 665 specific questions to see if the robot actually remembered the right things. They also tested two different "brains" (backbone models) for the robot to see if the type of brain changed the cost.

The Big Surprise: You Can't Predict the Price Tag

The first big discovery is that you cannot just look at how long the conversation is or how big the messages are to guess the cost. The researchers tried to build a simple math formula to predict the cost based on conversation length and message size.

For the "Full Transcript" and "Rolling Window" strategies, the math worked perfectly. If you know how many words you sent, you know exactly how much it costs.
For the Memory Systems, the math failed miserably. The formula missed the actual cost by 18% to 69%.

Why? Because the cost of a memory system isn't just about how much you talk; it's about what the system is doing internally while you talk. Sometimes the system decides to write a long summary, other times it just grabs a quick fact. Sometimes it reorganizes its whole notebook. These internal decisions are driven by the content of the conversation, not just the length. It's like trying to predict the cost of a road trip just by looking at the map, without knowing if the driver is going to stop for a fancy lunch or just grab a granola bar. The "lunch" (internal processing) varies wildly, making the total bill unpredictable.

The "Break-Even" Point: When Does the Notebook Pay Off?

The researchers asked a crucial question: At what point does using a memory system become cheaper than just re-reading the whole history?

The answer is: It depends entirely on the system and the robot's brain.

Mastra Observational Memory was the speedster. In some cases, it was cheaper than the "Full Transcript" method right from the very first turn (Turn 0).
Mem0 took a little longer, usually breaking even around Turn 82, but only if the messages were reasonably long.
Hindsight was the slowest. In many cases, it didn't become cheaper than the "Full Transcript" method even after 400 turns. In fact, for some setups, it was still more expensive at the end of the test.

This means that if you have a short conversation, using a complex memory system might actually cost more money than just re-sending the whole chat history. You have to talk for a certain amount of time (the "break-even" point) before the memory system starts saving you money.

Accuracy vs. Cost: No Free Lunch

The study also checked if the memory systems were actually good at answering questions. The results showed a wide range of performance:

Accuracy varied from 21% to 54%.
Mem0 had the biggest swings in accuracy, doing well on some robot brains and poorly on others.
Hindsight was generally the most accurate (often above 50%), but it was also the most expensive to run.
Mastra was a middle-ground performer but often the most cost-effective when you factored in how many correct answers it gave.

The researchers found that the choice of the robot's "brain" (the backbone model) mattered just as much as the choice of the memory system. A memory system that was cheap on one robot brain could be expensive on another.

The Final Verdict

The paper concludes that there is no single "best" memory system.

If you want the cheapest option per correct answer on a specific robot brain, Mastra on the gpt-oss-20b brain was the winner (costing about $0.028 per correct answer at 100 turns).
If you use a different robot brain (Gemma 4 26B A4B), Mem0 becomes the cheapest option.
Hindsight, while accurate, is often too expensive to be worth it unless the conversation is very long.

The main takeaway is that you can't just pick a memory system because it sounds cool. You have to look at your specific situation: How long will the conversation be? How big are the messages? And which robot brain are you using? Only then can you know if the memory system will save you money or just add to your bill. The "Total Recall" of a memory system comes at a cost, and that cost is far more complex than simply counting the words you type.

Technical Summary: Total Recall at What Cost? Benchmarking the Serving Cost of Agentic Memory Systems

Problem Statement

As conversational agents transition from single-session demonstrations to long-running deployments spanning weeks or months, they require mechanisms to retain information across turns. Two primary strategies exist: (1) Full-transcript resubmission, where the entire conversation history is re-sent to the LLM context window every turn, and (2) Agentic memory systems, which distill past interactions into compact records (facts, summaries, knowledge graphs) and retrieve only relevant subsets.

While memory systems are adopted to escape the unbounded cost growth of full-transcript resubmission, the actual serving cost of these memory systems remains poorly understood. Existing evaluations focus almost exclusively on accuracy and recall, often reporting isolated token-saving figures under specific conditions. There is a lack of systematic, cross-system benchmarking that measures the end-to-end serving cost of memory systems against transparent baselines, paired with accuracy metrics. Consequently, practitioners cannot determine if a memory system is economically viable compared to simply resubmitting the transcript.

Methodology

The authors conducted a controlled benchmark evaluating three agentic memory systems against two reference strategies across two backbone models and varying reasoning effort settings.

Systems Evaluated

Mem0: A flat extract-and-retrieve system that stores atomic facts in a vector database.
Hindsight: A system utilizing a retain–recall–reflect pipeline for memory ingestion.
Mastra Observational Memory: An observer–reflector–actor loop with threshold-triggered consolidation.
Reference Strategies:
- Rolling Window (Floor): A fixed-size window of the last 10 turns (no persistent storage).
- Full History (Ceiling): Resubmitting the entire transcript every turn.

Experimental Design

Backbones: Evaluated on gpt-oss-20b and Gemma 4 26B A4B at "low" and "medium" reasoning effort settings.
Workload: Synthetic conversations generated to specific lengths ( $N$ ) and per-turn token sizes ( $L$ ), up to 400 turns.
Cost Measurement: The study modeled the total billable input tokens across three pipeline stages: Ingest (fact extraction/consolidation), Retrieval, and Answer.
Cost Model: A separable log-log regression model was fitted to predict cost ( $C$ ) based on message size ( $L$ ) and conversation depth ( $t$ ):
$\log(C+1) = a + p\log(L+1) + q\log(t+1)$
where $p$ represents the cost sensitivity to message size and $q$ to conversation depth.
Accuracy Benchmark: Paired with cost measurements, accuracy was evaluated on a stratified subset of the LoCoMo benchmark (665 questions across multi-session dialogues).
Validation: Models were validated using leave-one-cell-out cross-validation (LOOCV) to test generalizability.

Key Results

1. Cost Modeling and Predictability

Reference Strategies: The cost model fits the reference strategies exceptionally well ( $R^2 \geq 0.91$ ). The full-history baseline shows $p \approx q \approx 1$ (cost scales with total content), while the rolling window shows $p \approx 0.9$ but $q \approx 0.1$ (cost scales with message size but is flat regarding depth).
Memory Systems: The model fails to predict the costs of the memory systems.
- Mem0: $p \approx 0.17, q \approx 0.08$ (low sensitivity to both).
- Hindsight: $p \approx 0.15, q \approx 0.42$ (high sensitivity to depth).
- Mastra OM: $p \in [0.61, 0.79], q \in [0.23, 0.36]$ .
- Held-out Error: While baselines generalize with <6.5% error, memory systems exhibit 18–69% error in held-out validation. This indicates that the serving cost of memory systems is driven by internal memory state (e.g., trigger thresholds, consolidation rates) rather than just conversation length or message size.

2. Break-Even Analysis

The study calculated the "break-even turn"—the point where a memory system becomes cheaper than resubmitting the full transcript.

High Variability: The break-even point is highly sensitive to the system, backbone, and workload.
- Mastra OM: Can break even immediately (turn 0) or within tens of turns.
- Mem0: Breaks even within tens of turns for larger messages, but can take up to 342 turns for smaller ones.
- Hindsight: Often fails to break even within 400 turns, particularly with smaller message sizes.
Magnitude: Once a memory system breaks even, it can be up to 12.7x cheaper than full-history resubmission. Conversely, systems that fail to break even (like Hindsight with small messages) can cost up to 3.3x more than the full transcript.

3. Cost-Accuracy Trade-offs

Accuracy Range: Accuracy on LoCoMo varied significantly (21%–54%) depending on the system and backbone.
No Dominant System: No single system wins on both cost and accuracy.
- Mastra OM on gpt-oss-20b (low effort) achieved the lowest cost-per-correct-answer ($0.028 at 100 turns).
- Mem0 on Gemma 4 26B A4B was the most cost-effective option for that backbone.
- Hindsight was generally the most expensive to serve, though it achieved higher accuracy in some configurations.
Backbone Impact: The choice of backbone model influences cost as much as the choice of memory system. For instance, Mem0's cost dropped significantly on Gemma 4 26B A4B due to lower reasoning token usage compared to gpt-oss-20b.

Significance and Claims

The paper claims to provide the first controlled, cross-system benchmark of agentic memory serving costs paired with accuracy. Its primary contributions and findings are:

Cost is Non-Linear and State-Dependent: A memory system's serving cost cannot be predicted solely by conversation length and message size. The internal behavior of the memory pipeline (triggers, consolidation) drives cost variance, rendering simple regression models insufficient for prediction.
Context-Dependent Economic Viability: Whether a memory system is cheaper than full-transcript resubmission is not an intrinsic property of the system but a function of the workload (conversation length, message size) and the backbone model. Break-even points range from immediate to "never" within 400 turns.
Joint Decision Making: Practitioners cannot optimize for cost or accuracy in isolation. The backbone model and memory system choice are coupled; a backbone that reduces cost for one memory system may increase it for another.
Limitations of Current Benchmarks: The study highlights that existing benchmarks focusing solely on accuracy miss the critical production constraint of serving cost, and that current cost models for memory systems are descriptive rather than mechanistic.

The authors conclude that selecting a memory system requires a specific analysis of the expected conversation length and the chosen backbone, as no universal "best" system exists across all settings.