💬 NLP

LiveMem: Maintaining Memory State Continuity in Long-Running LLM Inference

The paper introduces LiveMem, an intrinsic memory method for pretrained full-attention LLMs that maintains a persistent, load-bearing memory state independent of the active context window, thereby enabling state continuity and long-term information retention in long-running inference scenarios where historical evidence is no longer present in the immediate context.

Original authors: Zhichen Liu, Ruihan Sun, Hengjie Yang, Zipeng Wu, Zhaohan Chen, Xiaofan Zhang, Yang Xu

Published 2026-08-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhichen Liu, Ruihan Sun, Hengjie Yang, Zipeng Wu, Zhaohan Chen, Xiaofan Zhang, Yang Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are talking to a very smart friend who remembers everything you've ever said. In the world of artificial intelligence, this friend is a "Large Language Model" (LLM). Usually, these models are like students taking a test: they read a specific set of instructions and a story, answer a question, and then the test is over. But what if you wanted this friend to be a lifelong companion, chatting with you for years, helping you plan a trip, remember your favorite songs, and solve problems over hundreds of conversations?

The problem is that these AI friends have a "working memory" that is surprisingly small. Think of it like a whiteboard in a classroom. The teacher (the AI) can only see what's written on the board right now. If the class goes on too long, the teacher has to erase the old notes to make room for new ones. Once erased, those old notes are gone from the board. Traditional AI systems try to solve this by keeping a giant notebook of everything ever said and flipping through it whenever a question comes up. But this is slow and clunky; it's like having to stop the conversation to look up a fact in a library instead of just remembering it. The big question scientists are asking is: Can we give the AI a way to keep a "feeling" or a "summary" of the past on its whiteboard, even after the specific words have been erased, so it never loses the thread of the conversation?

This is exactly what the paper "LiveMem" tackles. The researchers, led by Zhichen Liu and Ruihan Sun, propose a new way for AI to handle long conversations without losing its mind. They introduce a system called LiveMem, which acts like a magical, permanent "backpack" for the AI's thoughts.

Here's how it works: Imagine the AI has two ways of thinking. The first is its "Main Brain," which looks at the current conversation on the whiteboard. This whiteboard is limited in size; when it gets full, the oldest notes are erased to make space for new ones. This is normal. But LiveMem adds a second part: a "Side Brain" or a Recurrent Memory State. This side brain is like a backpack that the AI wears at all times. As the AI reads new words, it updates this backpack. When the whiteboard gets full and old notes are erased, the important information from those erased notes isn't lost; it gets packed into the backpack.

The researchers trained this system using a special technique. They didn't just let the AI read long stories; they forced it to practice "forgetting" the whiteboard while keeping the backpack full. They used a method called "context turnover," where they would take a long conversation, show the AI the beginning, then erase the beginning from the whiteboard, and ask the AI a question about it. If the AI only relied on the whiteboard, it would fail because the notes were gone. But because LiveMem had packed the key details into its backpack, the AI could still answer correctly.

The paper shows that LiveMem is very good at this. In their tests, when the evidence for a question was completely erased from the AI's immediate view (the whiteboard), LiveMem could still answer the question with high accuracy. For example, in a test called "LongMemEval," when the supporting evidence was pushed far away from the current conversation window, LiveMem significantly outperformed other methods. While models without this "backpack" (like the standard Qwen3-4B) dropped to extremely low accuracy, LiveMem achieved results more than ten percentage points higher. This suggests that the AI isn't just guessing; it's actually carrying the history forward in its memory state.

However, the authors are careful to note that this "backpack" isn't a perfect recording studio. It doesn't store every single word exactly as it was spoken. If you ask the AI to recite a specific sentence that was erased three days ago, it might not get it 100% right. It's more like a summary or a feeling of what happened. The paper explicitly rules out the idea that this system is a perfect, lossless archive of every token. Instead, it's a "lossy" state that keeps the useful parts of the memory alive.

The team also found that how you train the AI matters a lot. If you just mix all kinds of data together, the AI might not learn to use the backpack effectively. They discovered that a specific training schedule—starting with long stories to "wake up" the memory side, and then switching to a mix of different tasks—worked best. This allowed the AI to learn how to write to and read from its backpack efficiently.

In short, LiveMem suggests a new way to build AI assistants that can truly remember their history. Instead of just looking up old notes in a library or relying on a tiny whiteboard, LiveMem gives the AI a persistent state that evolves with every conversation. While it's not a perfect archive of every word, it successfully keeps the "story" of the interaction alive, allowing the AI to stay coherent and helpful even after the specific details of the past have been erased from its immediate view. This brings us one step closer to AI agents that can truly be lifelong companions, maintaining a continuous sense of self and history over time.

Technical Summary: LiveMem

Problem Definition: State Continuity under Context Turnover

Long-running AI assistants and agents operate over interaction lifecycles that inevitably exceed the fixed context window of Large Language Models (LLMs). While existing solutions like Retrieval-Augmented Generation (RAG) and external memory systems provide "historical access" (retrieving specific past records on demand), they fail to provide state continuity. These systems act as persistent storage (like notepads) but do not maintain a seamless, continuous computational state that evolves as the working context changes.

The core problem identified is how to maintain inference continuity when the working context undergoes turnover (i.e., when old tokens are evicted to make room for new ones). Existing methods struggle to carry computation forward through a fixed-capacity state whose lifetime is independent of the active context. The paper posits that while external retrieval handles "historical access," a distinct mechanism is required for "state continuity"—a continuous, online-updated memory state that persists across the model's entire lifecycle.

Methodology: LiveMem Architecture and Training

LiveMem is introduced as an intrinsic memory method that augments a pretrained full-attention LLM with a parallel recurrent memory branch. The system is designed to operate under strict context turnover constraints.

1. Architecture

LiveMem modifies a standard decoder transformer by adding a side-path memory branch to each attention layer:

Main Path (Attention): Maintains a bounded working context (KV cache) for precise short-term access. When the context limit is reached, old Key-Value (KV) entries are evicted (context turnover).
Side Path (Recurrent Memory): Uses Gated DeltaNet-2 (GDN2) as the backbone for a parallel recurrent state ( $S_t$ ). This branch continuously reads and updates a fixed-size memory state from the same hidden representations as the main path.
Integration: The output of the GDN2 branch ( $r_t$ ) is added directly to the attention output ( $a_t$ ) at every decoder layer ( $o_t = a_t + r_t$ ).
Mechanism: The GDN2 state update involves learned decay, erase, and write gates. Crucially, when a chunk of tokens is evicted from the main attention path, its information is preserved in the recurrent memory state. The system retains the system prompt as an "attention sink" to ensure structural stability.

2. Context Turnover and Inference Consistency

The system enforces a strict "oldest-first" eviction policy during both training and inference.

Training: Dynamic attention masks are used to simulate context turnover, forcing the model to rely on the memory state for information that has been removed from the active window.
Inference: Evicted KV pages are released to memory, making them inaccessible to the attention mechanism. The recurrent state is the only remaining carrier of that historical information.

3. Memory-Oriented Post-Training

Simply adding a recurrent module does not guarantee it will carry effective memory. LiveMem employs a specific post-training regimen to ensure the state becomes "living" (i.e., it genuinely influences future predictions after source tokens are gone):

Supervised Fine-Tuning (SFT): The model is trained on long-document QA and multi-hop reasoning tasks where the history is truncated from the attention window. The loss is applied only to answer tokens, forcing the model to utilize the memory state to generate correct answers.
Reinforcement Learning (RL): The model is further optimized using the GRPO algorithm with a reward signal based on answer accuracy. This encourages the model to better preserve and retrieve information from the memory state, particularly in scenarios with multiple questions sharing a historical background.
Training Strategy: A curriculum is used where the model is first "warmed up" with long-text data to activate the side branch, followed by training on a rebalanced mixture to refine memory maintenance.

Key Contributions

Formalization of State Continuity: The paper defines "state continuity under context turnover" as a distinct inference capability, separating it from external retrieval mechanisms. It models this as a lifecycle-scale requirement where a fixed-capacity memory state persists independently of the active context.
LiveMem Architecture: An end-to-end design integrating a GDN2-based recurrent side branch into a full-attention LLM, enabling the model to maintain a continuous state while the main attention window rotates.
Training Paradigm: The proposal of memory-oriented post-training tasks (SFT and RL) specifically designed to force the model to learn how to write to, preserve, and read from a memory state after the originating tokens are no longer visible to the attention mechanism.
Empirical Validation: Comprehensive evaluation demonstrating that the memory state can carry task-relevant information beyond the active window, even when evidence is completely evicted.

Experimental Results

Performance Overview:
LiveMem was evaluated on Qwen3-4B across Wiki QA, Conversation QA, Test-Time Learning (TTL), and Long Document QA tasks, with context windows restricted to 8k or 32k tokens.

Overall Performance: LiveMem (specifically the RL variant) achieved leading overall performance, outperforming baselines like RAG, Context2LoRA, $\delta$ -Mem, and a Recurrent baseline.
Multi-Question (MQ) Tasks: While other methods (including RAG and $\delta$ -Mem) saw dramatic performance drops on multi-hop, multi-question tasks, LiveMem maintained its lead, indicating superior ability to handle multiple logic chains simultaneously via a persistent state.
Long QA: LiveMem achieved balanced and strong scores across long-document tasks, whereas other methods were often specialized to specific subsets.

State Continuity Analysis (LongMemEval):

Evidence Eviction: Experiments showed that when supporting evidence was completely removed from the active context window, LiveMem maintained accuracy significantly higher than baselines (e.g., ~16.5% vs ~3.1% for Qwen3-4B in fully evicted scenarios).
Distance Analysis: As the distance between the evidence and the current window increased, LiveMem's accuracy showed a slight decline but remained stable and measurable, proving the memory state retains information across multiple turnover intervals.
Ablation: Training the side path with full parameters yielded better stability than LoRA, suggesting the distribution gap between attention and GDN2 requires full adaptation.

Limitations:
The paper notes that LiveMem's memory state is lossy. It cannot reliably reconstruct arbitrary, fine-grained tokens (e.g., specific "needles" in a haystack) once they are evicted, as demonstrated by Needle-in-a-Haystack (NIAH) tests where LiveMem performed similarly to the base model under bounded contexts. The system is designed for task-relevant influence, not exact archival recall.

Significance and Claims

The paper claims that LiveMem establishes state continuity as a distinct and complementary abstraction for continual LLM inference.

Complementarity: It argues that "historical access" (retrieval) and "state continuity" (recurrent memory) are orthogonal. Retrieval determines what past information is read, while the recurrent state determines how information coherence is maintained over the lifecycle after reading.
Practicality: By maintaining a query-independent memory state that survives context turnover, LiveMem addresses a critical gap between limited-capacity attention and external retrieval, moving toward practical infinite-context inference where the model's internal state evolves continuously without relying solely on re-fetching history.
Modesty: The authors explicitly state that LiveMem is not universally optimal; retrieval and parametric memory still have strengths. However, the results sufficiently demonstrate that intrinsic memory states can carry historical information beyond the working context, a capability previously unaddressed by existing engineering-based memory patches.