$\delta$-mem: Efficient Online Memory for Large Language Models

Technical Summary of $\delta$-mem: Efficient Online Memory for Large Language Models

1. Problem Statement

As Large Language Models (LLMs) are deployed in long-term assistants and agent systems, they face a critical challenge: the need to accumulate, update, and reuse historical information over extended interactions. Current strategies often rely on simply expanding the input context window. However, this approach is computationally expensive due to the quadratic cost of standard attention relative to context length. Furthermore, simply increasing the window size does not guarantee effective utilization; models often suffer from "context degradation" or "context rot" when handling very long sequences, suggesting that million-token windows do not fundamentally solve the memory problem.

Existing memory mechanisms fall into three paradigms, each with limitations:

• Textual Memory Mechanisms (TMMs): Store memory as text injected into the input context. While flexible, they suffer from context-window limits, retrieval noise, and compaction loss.
• Outside-Channel Memory Mechanisms (OMMs): Keep memory in external modules accessed via retrieval. This introduces overhead, integration complexity, and potential misalignment with the backbone.
• Parametric Memory Mechanisms (PMMs): Encode memory into prefixes or adapters. While efficient for frozen backbones, their static nature limits adaptation to dynamically evolving information.

There is a need for a mechanism that maintains a compact, dynamically evolving memory state and steers the backbone through a pathway tightly aligned with its internal attention computation, without requiring full fine-tuning or explicit context extension.

2. Methodology: $\delta$-mem

The authors propose $\delta$-mem, a lightweight memory mechanism that augments a frozen full-attention backbone with a compact Online State of Associative Memory (OSAM). The core design follows a "Read-Steer-Write" cycle at each position, leaving the backbone parameters frozen.

A. Online State of Associative Memory (OSAM)
Instead of storing all historical tokens, $\delta$-mem compresses past information into a fixed-size state matrix $S \in \mathbb{R}^{r \times r}$ (specifically an $8 \times 8$ matrix in experiments). This state stores key-value associations ($k \to v$) from the historical context.

• Update Rule: The state is updated sequentially using a delta-rule learning approach inspired by online regression. Given a memory key $k_t$ and value $v_t$, the state predicts $\hat{v}_t = S_{t-1}k_t$. The update minimizes the residual error:
  $$S_t = \lambda_t S_{t-1} + \beta_t (v_t - S_{t-1}k_t)k_t^\top$$
  Here, $\lambda_t$ (forget gate) controls retention of previous memory, and $\beta_t$ (write gate) controls the strength of the residual update. This allows the state to correct predictive discrepancies dynamically while retaining stable historical information.

B. Memory Projections
The hidden state $x_t$ from the frozen backbone is projected into a low-dimensional memory space ($r$ dimensions) to generate query ($q^m_t$), key ($k^m_t$), and value ($v^m_t$) vectors for the memory state. Normalization is applied to $q^m_t$ and $k^m_t$ to reduce instability during long-sequence recurrence.

C. Steering via Low-Rank Corrections
During generation, $\delta$-mem reads from the previous state $S_{t-1}$ using the current query $q^m_t$ to produce a read vector $r_t = S_{t-1}q^m_t$. This vector provides context-relevant signals that steer the backbone's attention without retrieving text tokens.

• The read signal is transformed into low-rank corrections for the query ($\Delta q_t$) and output ($\Delta o_t$) sides of the attention mechanism:
  $$\tilde{q}_t = q^0_t + \alpha \Delta q_t, \quad \tilde{y}_t = a_t + \alpha \Delta o_t$$
  where $q^0_t$ is the original query and $a_t$ is the standard attention output. These corrections allow the associative memory to directly participate in the forward computation.

D. Writing Granularity Strategies
The paper investigates three strategies for updating the state:

1. Token-State Write (TSW): Updates the state at every token. Preserves fine-grained details but is sensitive to noise.
2. Sequence-State Write (SSW): Averages hidden states within a message/segment before updating. Reduces redundancy and smooths evolution.
3. Multi-State Write (MSW): Maintains multiple parallel sub-states (e.g., for facts, preferences, task progress) to reduce interference, aggregating their readouts.

E. Training
$\delta$-mem is trained using standard Supervised Fine-Tuning (SFT) loss. The context tokens are written into the online state, but the state itself (not the raw context tokens) steers the frozen backbone during the prediction of the response.

3. Key Contributions

1. Novel Mechanism: Proposes $\delta$-mem, which couples a compact, dynamically updated online state directly with the attention computation of a frozen backbone, avoiding full fine-tuning or backbone replacement.
2. Compact Efficiency: Demonstrates that an extremely small memory state ($8 \times 8$ matrix) is sufficient to retain useful historical signals and recover context-relevant information even when explicit history is removed.
3. Performance Gains: Shows significant improvements on memory-heavy benchmarks while preserving general capabilities, without relying on extending explicit context or heavy external retrieval modules.

4. Experimental Results

The method was evaluated on general capability benchmarks (IFEval, GPQA-Diamond, HotpotQA) and memory-heavy benchmarks (LoCoMo, MemoryAgentBench) using Qwen3-4B-Instruct, Qwen3-8B, and SmolLM3-3B backbones.

• Overall Performance: $\delta$-mem improved the average score by 1.10$\times$ over the frozen backbone and 1.15$\times$ over the strongest non-$\delta$-mem memory baseline (Context2LoRA).
• Memory-Heavy Tasks:
  • MemoryAgentBench: Achieved a 1.31$\times$ improvement (from 29.54% to 38.85% average).
  • LoCoMo: Achieved a 1.20$\times$ improvement (from 40.79% to 49.12% average).
  • TTL Subtask: The score nearly doubled from 26.14 to 50.50.
• Backbone Variations:
  • On Qwen3-4B, the TSW variant achieved the best average (51.66%).
  • On Qwen3-8B, SSW performed best (50.86%), suggesting segment-level writing mitigates noise for stronger models.
  • On SmolLM3-3B, MSW drove a substantial leap (26.08% to 36.96%), indicating smaller models benefit from separating memory types to minimize interference.
• Context Recovery: In "No Context" settings (where explicit history is removed), $\delta$-mem recovered significant performance, increasing HotpotQA EM from 0.08% to 6.48% and LoCoMo average from 3.49% to 8.05%, proving the state stores reusable signals.
• Ablation Studies:
  • Heads: Injecting corrections into both query and output branches ($qo$) provided the best trade-off between performance and parameters.
  • Depth: Applying corrections across all layers yielded the best results, indicating memory signals benefit the full representation hierarchy.

5. Significance and Claims

The paper claims that effective memory for LLMs can be realized through a compact online state directly coupled with attention computation. The significance of $\delta$-mem lies in its ability to:

• Decouple memory maintenance from prompt reinsertion: Unlike textual memory, it does not route compressed history back through token space, avoiding context window limits and retrieval noise.
• Enable dynamic evolution: Unlike parametric memory, the state evolves continuously with the sequence via delta-rule learning, adapting to new interactions without discrete editing steps.
• Maintain efficiency: It achieves these gains with a fixed, tiny state size ($8 \times 8$) and without modifying the backbone architecture, offering a scalable interface for test-time memory in frozen Transformers.

The authors conclude that this approach offers a robust alternative to context expansion, allowing models to leverage historical information effectively without the computational burden of long-context processing or the complexity of external retrieval systems.