Technical Summary: Diagnosing and Training the Memory-Update Gap in LLM Agents

1. Problem Definition: The Supersession Gap

The paper addresses a critical failure mode in Large Language Model (LLM) agents operating over long, multi-session interactions: supersession. This is the ability to correctly utilize the current value of a fact while discarding values that have been updated or retracted in previous sessions.

While agents can often read updates when provided with full context, they struggle to maintain these updates when constrained to a bounded, self-maintained memory (e.g., a notes field) where raw conversation history is never re-fed. The authors define the supersession gap as the accuracy loss an agent incurs when forced to rely solely on this compressed memory rather than the full context.

The central hypothesis is that this failure is not due to a lack of comprehension (the ability to understand the update) but rather a failure of memory maintenance (the policy of deciding what to keep, update, or discard).

2. Methodology

The Supersede Environment

The authors introduce Supersede, an open reinforcement learning environment built on the verifiers and Prime Intellect stacks.

• Task Structure: Agents engage in multi-session interactions. After each session, the agent must rewrite a bounded "notes" memory (capped at $B$ characters). Crucially, raw sessions are never re-fed; the agent must answer a final query about a fact's current value using only the notes.
• Reward Signal: The environment provides a programmatic, verifiable reward ($r_{cur}$) for answering with the current value of a fact. A penalty ($r_{stale}$) is applied for asserting superseded values. This reframes "forgetting-aware" metrics (previously used only for evaluation) into a dense training signal.
• Data Sources:
  • Real Data: The knowledge-update subset of LongMemEval, featuring implicit, paraphrased updates in real conversational data.
  • Synthetic Data: A procedural generator creating explicit "X now Y" timelines, used for training curricula and ablation studies.

Experimental Design

The study follows a four-step diagnostic framework to isolate the cause of the failure:

1. Isolation: Comparing Full Context vs. Bounded Memory on real data.
2. Model Scaling: Testing if frontier models (e.g., gpt-5.4) close the gap.
3. Memory Scaling: Testing if increasing memory size proportionally to conversation length closes the gap.
4. Training: Fine-tuning a small open model (Qwen2.5-3B) using GRPO (Group Relative Policy Optimization) on the Supersede environment to see if the gap can be trained down.

3. Key Contributions

1. Isolation of Memory Maintenance: The paper demonstrates that the supersession gap is a distinct failure mode caused by memory maintenance, not reading comprehension. Replacing full context with bounded memory significantly degrades accuracy even on frontier models.
2. Frontier Confirmation: The gap persists on state-of-the-art models (gpt-5.4). While full-context accuracy saturates near 92%, bounded-memory accuracy remains significantly lower (77%), with a statistically significant gap ($p < 0.005$).
3. Scale vs. Size: The failure scales with the length of the conversation, not the compression ratio. Increasing conversation length by 24× drops accuracy from 68% to 28%. Crucially, granting the agent 24× more memory yields no detectable recovery (accuracy remains at 28%), proving that simply allocating more memory is insufficient.
4. Trainable Environment: Supersede is the first environment where the reward signal explicitly targets temporal fact-currency (keeping the current value) rather than just final answer correctness or evidence relevance.
5. Training as a Solution: The paper provides the first evidence that the supersession gap is trainable. Fine-tuning a small model (Qwen2.5-3B) on Supersede nearly doubles its held-out accuracy on real, unseen conversations (from 9.0% to 16.7%).

4. Results

Diagnostic Findings

• Model Scaling: On the LongMemEval oracle split ($n=78$), gpt-5.4 achieves 92% accuracy in full context but drops to 77% in bounded memory. The gap does not vanish as model capability increases.
• Memory Scaling: In a scale study ($n=25$), extending the conversation from ~2 sessions to ~48 sessions (24× longer) caused accuracy to plummet from 68% to 28%. Increasing the memory budget proportionally (to ~7,150 characters) resulted in no recovery (28% $\to$ 28%). The failure is driven by conversation length, not memory size.
• Failure Modes: Errors are dominated by relevant facts being compressed away or failing to overwrite stale values. The model understands the update when reading but fails to retain it in the notes.

Training Findings

• Policy Learning: Using GRPO on synthetic procedural data, the model's training reward climbed from 0.66 to 0.97, eventually self-terminating when the advantage vanished (indicating the curriculum was solved).
• Transfer to Real Data: The learned policy transferred to the real LongMemEval held-out set.
  • Baseline: 9.0% accuracy (7/78 correct).
  • Trained: 16.7% accuracy (13/78 correct).
  • Trend: The improvement was monotonic along the training checkpoints, rising only after the training reward saturated, confirming the gain is due to learned behavior rather than a lucky seed.
• Significance: While the absolute accuracy (16.7%) remains below frontier capabilities, the result is significant because it demonstrates that neither larger models nor larger memories close the gap, but a trained memory policy does.

5. Significance and Claims

The paper claims to shift the understanding of long-horizon agent failures from a comprehension problem to a memory-policy problem.

• Reframing the Bottleneck: The authors argue that waiting for larger models or simply allocating more memory will not solve the supersession gap. The bottleneck is the policy for maintaining currency in a bounded state.
• Trainability: This is the first work to demonstrate that the supersession gap can be trained down rather than merely measured. By providing a verifiable, time-indexed reward signal, Supersede turns a diagnostic metric into a learning objective.
• Modest Scope: The authors are explicit that the training result is a "proof of mechanism" rather than a finished, production-ready policy. The gains are modest (+7.7 percentage points) and the model is small (3B parameters). However, the monotonic improvement curve provides evidence that the behavior is learnable.
• Future Direction: The paper suggests that the next step is not scaling models, but developing longer, more implicit training curricula that do not saturate, allowing the learned policy to generalize further.

In summary, the paper establishes that supersession is a trainable behavior that requires specific optimization of memory maintenance policies, a capability that current scaling laws (model size, context window size) do not automatically provide.