🤖 machine learning

DeltaLog: Deferred Materialization of Recurrent States for Linear Attention Decoding

DeltaLog is a recurrent-state decoding scheme that accelerates linear attention models by deferring full state materialization in favor of appending compact updates to a bounded log and periodically merging them, thereby significantly reducing memory traffic and improving end-to-end serving speed.

Original authors: Junqing Lin, Jingwei Sun, Guangzhong Sun

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junqing Lin, Jingwei Sun, Guangzhong Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern artificial intelligence systems that generate text, from chatbots to creative writing assistants, rely on a fundamental process called autoregressive decoding. In this process, the computer predicts the next word in a sentence one at a time, using the words it has already generated to inform its next guess. For years, the most powerful models have used a mechanism known as attention to decide which previous words matter most for the current prediction. While this approach is incredibly effective, it creates a growing bottleneck: as the conversation gets longer, the system must constantly store and retrieve an ever-expanding list of every word seen so far, consuming vast amounts of computer memory and slowing down the response time. To solve this, researchers have developed a newer class of models that replace the expanding list with a fixed-size summary, or state, that updates with every new word. This change eliminates the need to remember every single past token, but it introduces a different problem: the system must still constantly rewrite this entire summary every time a new word is added, creating a heavy traffic jam in the computer's memory.

A team of researchers at the University of Science and Technology of China has identified this constant rewriting as a major inefficiency and has devised a solution they call DeltaLog. Instead of forcing the computer to rewrite the entire summary of the conversation after every single word, DeltaLog allows the system to keep a stable, complete version of the summary and simply attach a small, compact note describing the most recent change. The system only rewrites the full summary occasionally, after it has accumulated a certain number of these small notes. This approach is like keeping a master ledger and a stack of sticky notes; rather than rewriting the entire ledger every time a new transaction occurs, you just add the transaction to the stack and update the ledger only when the stack gets too high. By doing this, the researchers found they could drastically reduce the amount of data the computer needs to move around, which is often the slowest part of the process.

The researchers tested this method on several different types of modern language models, including Gated DeltaNet, Kimi Delta Attention, and RWKV6. In their experiments, they measured how long it took for the computer to generate a single word and how much memory traffic was involved. They discovered that by deferring the full rewrite of the summary, they could speed up the core calculation that updates the model's memory by up to 1.86 times on high-end graphics cards. More importantly, they observed that the amount of data written to the computer's high-speed memory dropped by as much as 7.83 times. This reduction in traffic is significant because, in these types of models, the speed is often limited not by how fast the computer can calculate, but by how fast it can move data in and out of its memory.

When the researchers integrated this method into a full system designed to serve many users at once, the benefits translated into faster response times for the end user. In tests with large models containing tens of billions of parameters, the system generated words between 5% and 20% faster than before. The improvement was most noticeable when the system was handling many requests simultaneously, a common scenario for real-world applications. The researchers confirmed that this speedup did not come at the cost of accuracy; the text generated by the modified system remained mathematically equivalent to the original, meaning the quality of the output was preserved while the delivery became much more efficient.

The key insight behind this work is that the way a computer physically stores and updates information does not always need to match the logical steps the model takes. While the model logically updates its state with every word, the physical hardware does not need to immediately rewrite the entire state to reflect that change. By separating the stable history from the recent changes and only merging them periodically, DeltaLog reduces the "state-update tax," a term the authors use to describe the excessive memory traffic caused by eager, immediate updates. This strategy does not change the underlying model or its weights; it simply changes the schedule of how the computer handles the data. The results suggest that for large-scale language models, optimizing the physical movement of data is just as critical as improving the mathematical algorithms themselves, offering a clear path to faster and more efficient artificial intelligence.

Technical Summary: DeltaLog

Problem: The State-Update Tax in Linear Attention Decoding

Linear attention models (e.g., GDN, KDA, RWKV6) offer a compelling alternative to softmax attention by replacing the quadratic prefix computation and context-growing Key-Value (KV) cache with fixed-size recurrent state updates. In these models, the state $S_t$ is a dense matrix of size $d_k \times d_v$ , independent of sequence length.

However, existing decoding implementations suffer from a significant inefficiency termed the state-update tax. While the logical update contributed by a single token is compact (represented by vectors of size $O(d_k + d_v)$ ), standard "eager" implementations immediately materialize this update into the full dense state and write the entire $[B, H, d_k, d_v]$ tensor back to High-Bandwidth Memory (HBM) after every token generation.

Memory Traffic Mismatch: For a batch size $B$ and $H$ heads, the actual information added is $O(BH(d_k + d_v))$ , but the physical traffic involves reading and writing $O(BH d_k d_v)$ elements.
Memory-Bound Bottleneck: This eager materialization creates an arithmetic intensity of roughly 0.5–1.0 FLOPs/byte, which is far below the machine balance of modern GPUs (e.g., ~208 FLOPs/byte for H200). Consequently, recurrent state maintenance becomes a primary source of memory traffic, dominating decode latency, especially as batch sizes and state sizes increase. Measurements show this can consume up to 44.6% of decode latency in large-batch scenarios.

Methodology: Deferred Materialization via DeltaLog

The paper proposes DeltaLog, a decoding scheme that decouples the logical recurrence from the physical materialization of the state. DeltaLog does not alter model weights or the mathematical recurrence; it only changes the physical schedule of state updates.

Core Mechanism: Base + Log

DeltaLog represents the live recurrent state as two components:

Dense Base State ( $\bar{S}$ ): A dense matrix storing the accumulated history up to the last "merge" point.
Bounded Update Log: A buffer storing recent, compact update factors ( $a_t, b_t$ ) and their associated log-decay metadata ( $\Lambda$ ).

The decoding cycle operates in two phases:

Append Phase (Common Case): For $M-1$ steps, the system does not write back the full dense state. Instead, it appends the compact update factors and current log-decay snapshot to the log buffer. The dense base $\bar{S}$ remains unchanged.
Merge Phase (Periodic): Every $M$ steps, the system materializes the accumulated logical state (base + replayed log entries) back into the dense base $\bar{S}$ , resets the log, and clears the deferred updates.

Traffic Model

By deferring the dense write-back, DeltaLog replaces frequent full-state writes with:

Lightweight appends of compact factors ( $O(d_k + d_v)$ ).
Amortized dense writes occurring only once per $M$ steps.
Replay of log entries during the merge step.

The optimal merge interval $M$ balances the cost of increased replay traffic against the savings from reduced dense writes.

Kernel Implementation

To implement this efficiently on GPUs, the authors employ a tiled kernel decomposition:

Value-Tile Parallelism: The state is partitioned along the value dimension ( $d_v$ ). Each work unit handles a slice of the dense base and the corresponding shard of the value-side log.
Shared Metadata: Key-side factors and log-decay metadata are shared across value tiles to avoid redundant computation.
Staged Execution: The kernel is split into a shared "ReplayPrep" stage (computing coefficients once per slot-head) and parallel "AppendDecode" stages for value tiles.

The method is instantiated for three operators with different decay granularities and update semantics:

GDN (Gated DeltaNet): Head-wise scalar decay; uses Delta-rule correction.
KDA (Kimi Delta Attention): Per-key decay; uses Delta-rule correction.
RWKV6: Per-key decay; uses raw-value updates (no Delta-rule correction).

Key Contributions

Identification of the State-Update Tax: The paper characterizes the per-token overhead caused by eager dense materialization in large-batch linear attention serving, identifying it as a memory-bandwidth bottleneck distinct from the logical recurrence cost.
DeltaLog Execution Strategy: A deferred-materialization approach that represents the state as a dense base augmented with a bounded log of compact updates, allowing most steps to perform lightweight appends rather than full-state rewrites.
System Implementation and Evaluation: A complete implementation for GDN, KDA, and RWKV6 integrated into a vLLM-based serving stack, demonstrating that the optimization preserves exact arithmetic equivalence (within floating-point tolerance) while improving performance.

Results

Experiments were conducted on NVIDIA H200 and RTX 4090 GPUs across various batch sizes and context lengths.

Kernel-Level Performance:
- DeltaLog accelerates the recurrent-state update kernel by 1.19× to 1.86× compared to the highly optimized Fast Linear Attention (FLA) baseline.
- It reduces profiled recurrent-state write traffic by up to 7.83×.
- GDN benefits most due to its compact scalar decay metadata, while RWKV6 and KDA show significant gains despite heavier per-key metadata.
End-to-End Serving Performance:
- Integrated into a prototype serving stack, DeltaLog achieves 1.05× to 1.20× end-to-end speedups over dense recurrent baselines on models like Qwen3.6-35B and Kimi-Linear-48B.
- The gains are smaller than kernel-level improvements due to Amdahl's law, as the recurrent update is only one component of the full decode step (which includes MLP, normalization, and scheduling).
Numerical Correctness:
- The method maintains numerical equivalence to eager decoding, with maximum output relative errors well within standard repository tolerances (e.g., < 0.005 for all tested operators).

Significance

The paper argues that recurrent-state materialization is a critical, often overlooked bottleneck in linear attention decoding, particularly for large-batch serving. By optimizing the physical schedule of state updates rather than the algorithmic recurrence itself, DeltaLog demonstrates that significant efficiency gains can be achieved without modifying model architectures.

The work highlights that while linear attention removes the context-length dependency of KV caches, it introduces a new fixed-state maintenance cost that is heavily memory-bound. DeltaLog provides a practical solution to this "state-update tax," showing that deferring dense writes is a viable strategy to improve inference throughput for recurrent linear attention models. The authors note that the benefits are most pronounced in regimes where recurrent-state write-back constitutes a dominant portion of memory traffic (large batches, large state sizes).