DACCI: A Dual-Loop Diagnostic Protocol for Evidence-Preserving Agent Memory
This paper introduces DACCI, a dual-loop diagnostic protocol that enhances the reliability of LLM agent memory systems by enabling traceable, statistically grounded, and regression-aware evaluation to precisely localize failures and validate interventions across ingestion, retrieval, filtering, and generation stages.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of artificial intelligence, a new kind of assistant is emerging: one that does not just answer a single question and forget, but remembers a long conversation, learns a user's habits, and recalls facts from months ago. To make this possible, researchers have built systems with "persistent memory," a digital storage that holds onto information across many different interactions. However, building these systems is surprisingly difficult. When an assistant makes a mistake, it is often impossible to tell where the error happened. Did the system fail to save the fact in the first place? Did it retrieve the wrong piece of information? Did it filter out a crucial detail by accident? Or did it simply fail to write the answer correctly? Without knowing the specific stage where the failure occurred, developers are forced to guess, often making changes that improve one part of the system while accidentally breaking another.
A team of researchers at Xiaomi has introduced a new method called D2ACCI to solve this problem of hidden failures. Instead of just looking at the final score of a test, their protocol acts like a detailed flight recorder for the AI's memory. It breaks down every step of the process, from the moment a user speaks to the moment the AI answers, and records exactly what happened at each stage. By comparing two versions of the system side-by-side—one with a new feature and one without—they can pinpoint exactly which change caused an improvement or a mistake. This approach allows them to make decisions based on clear evidence rather than guesswork, ensuring that updates to the memory system are safe, effective, and truly helpful.
The core of this work is a two-part loop that mimics a careful scientific experiment. The first part is the "inner loop," where the AI actually runs, storing memories, searching for information, and generating answers. The second part is the "outer loop," which acts as a strict judge. This judge does not just look at whether the final answer was right or wrong. Instead, it examines the detailed logs, or traces, left behind by every step of the process. It asks specific questions: Was the right memory found? Was the right information kept? Was the wrong information correctly ignored? If the logs show that a change improved the answer but left no clear trail of how it happened, the system rejects the change. This ensures that every improvement is not only successful but also understandable and reproducible.
To test this method, the researchers built a memory system called MemStack and ran it through three different public tests designed to challenge long-term memory. These tests covered a wide range of scenarios, from remembering details in long conversations to recalling specific personal preferences and updating facts over time. The results were clear. The system achieved high accuracy scores, reaching 93.59 percent on one major test, 90.93 percent on another, and 57.20 percent on a third. More importantly, the new protocol revealed which specific features were actually responsible for these gains. They found that adding a feature to extract extra details from long conversations improved performance by nearly three percentage points. They also discovered that retrieving memories from past sessions helped with time-based questions, and a specific "forget guard" that prevents the AI from using information a user asked to delete improved accuracy by nearly two percentage points.
Crucially, the protocol also ruled out ideas that might have seemed helpful at first glance. One common method for searching text, known as BM25, was tested and found to provide no statistically significant benefit. In a traditional evaluation, this might have been overlooked or dismissed as a minor issue, but the new protocol flagged it as a feature that should not be promoted, saving the developers from wasting time on a dead end. This ability to distinguish between real improvements and random noise is a key strength of the method. The researchers also measured how well they could trace the cause of errors. When they used their detailed logs, they could identify the root cause of a mistake almost every time. In contrast, when they looked only at the final answer without the logs, they could not identify the cause of the error at all.
The study demonstrates that building reliable AI memory requires more than just chasing higher scores. It demands a system that can explain its own failures. By using this dual-loop approach, the researchers showed that they could iterate on their system with confidence, promoting only the changes that were backed by solid evidence and rejecting those that were not. This method transforms the development of AI memory from a process of trial and error into a disciplined, evidence-based practice. The result is a system that not only performs better but is also easier to understand, debug, and improve over time, paving the way for assistants that can truly remember and learn from us without losing their way.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.