🤖 AI

LaCache: Exact Caching and Precision-Adaptive Inference for Diffusion Large Language Models

LaCache is a training-free acceleration framework for Diffusion Large Language Models that achieves significant speedups by employing lossless state memoization to cache invariant intermediate results and utilizing per-group FP8 quantization to reduce memory bandwidth bottlenecks.

Original authors: Xingru Chen, Zelang Liang, Yongjia Ma, Jiqing Zhan, Shuling Yang, Lian Wen, Kun Zhan

Published 2026-07-21

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Xingru Chen, Zelang Liang, Yongjia Ma, Jiqing Zhan, Shuling Yang, Lian Wen, Kun Zhan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers don't just read words one by one, like a person turning pages in a book, but can look at a whole sentence at once and guess the missing pieces all together. This is the magic of "Diffusion Large Language Models," a new kind of AI that builds text by starting with a jumbled mess of symbols and slowly cleaning it up, step by step, until a clear story emerges. It's like a sculptor chipping away at a block of marble, but instead of stone, they are chipping away noise to reveal a sentence. The problem is, this process can be incredibly slow and energy-hungry. Every time the AI takes a step to clean up the text, it often re-calculates the parts of the sentence that haven't changed yet, wasting a massive amount of time and electricity. It's like a chef who, every time they add a new ingredient to a stew, decides to re-chop every single vegetable that was already perfectly diced and sitting in the bowl.

Enter LaCache, a clever new framework designed to stop this waste without changing the final taste of the dish. The researchers behind this project realized that in these AI models, most of the text stays exactly the same while only a small chunk is being updated. Instead of re-doing the math for the whole sentence every single time, LaCache acts like a super-efficient librarian. It keeps a "lossless" (perfectly accurate) record of the parts that haven't changed, so the AI can skip the boring, repetitive work and only focus on the new bits. They also introduced a way to do the heavy lifting with slightly less precision—like using a slightly smaller ruler for measurements that don't need to be perfect to the millimeter—which speeds things up even more. The result? The AI runs much faster, sometimes up to 40 times faster when combined with other tricks, but still gives the exact same correct answers.

The Problem: The "Re-Reading" Loop

To understand why LaCache is such a big deal, we first need to look at how these Diffusion models work. Imagine you are trying to solve a puzzle where you have to fill in a long sentence, but you can only fix a few words at a time. The model works in "blocks." It picks a small section of the sentence, tries to fix the words inside it, and then moves on. But here's the catch: while it's busy fixing that one small section, the rest of the sentence (the beginning and the end) stays exactly the same.

Despite this, the old way of doing things forced the computer to re-calculate the entire sentence from scratch every single time it took a step. It was as if, while you were fixing a typo in the middle of a letter, the computer insisted on re-writing the entire letter, including the parts you hadn't touched yet. This created a huge amount of "redundant computation"—wasted effort. The researchers found that this waste happened in three specific places:

The Translation Step: Converting words into numbers (Embedding).
The Positioning Step: Figuring out where words sit in the sentence (RoPE).
The Attention Step: Deciding which words matter to each other (FlashAttention).

The Solution: The Three Magic Caches

LaCache solves this by introducing a system called Lossless State Memoization (LSM). Think of "memoization" as a cheat sheet. If you've already done a math problem, you write the answer down so you don't have to solve it again. LaCache creates three specific cheat sheets for the AI:

EmbedCache (The Word Dictionary): This cache remembers the number-translation for any word that hasn't changed. If the word "apple" is at the start of the sentence and stays there, the computer doesn't need to translate "apple" to numbers again. It just grabs the saved number.
RoPECache (The Position Map): This cache remembers the "position math" for the unchanged words. It's like remembering that the first word is always "first," so you don't need to re-calculate its position every time you fix a word in the middle.
FACache (The Attention Cheat Sheet): This is the most complex one. It saves the "attention statistics" for the parts of the sentence that aren't being touched. Imagine a spotlight that shines on the words the AI is looking at. If the AI is only looking at the middle of the sentence, the spotlight doesn't need to re-calculate how it shines on the words at the very beginning and end. FACache saves those calculations so the AI can skip them entirely.

By using these three caches, the AI can skip the heavy lifting for the parts of the text that are already perfect. It only does the hard work on the specific block of words it is currently trying to fix.

The Speed Boost: A Precision Trick

Skipping the work is great, but the researchers wanted to go faster. They noticed that the AI's "brain" (specifically the parts called FFN layers) sometimes uses numbers that are very precise but don't actually need to be that precise to get the right answer. It's like measuring a room for a rug with a laser ruler that measures down to a billionth of an inch, when a standard tape measure would be just fine.

LaCache uses a technique called per-group FP8 quantization. This is a fancy way of saying they switch to a "smaller ruler" (FP8) for specific groups of calculations. They do this carefully, only on the parts of the model that can handle it without getting confused. This allows the computer's hardware to work much faster because it can process these smaller numbers more efficiently. It's like switching from carrying heavy stone blocks to carrying lightweight foam blocks; you can move them much faster, and as long as the foam is the right shape, the building still stands perfectly.

The Results: Fast, Accurate, and Ready to Go

The team tested LaCache on several different AI models and tasks, from solving math problems to writing code. The results were impressive:

Speed: On its own, LaCache made the AI about 1.3 times faster than the standard version. But when they combined it with other existing speed-up tricks, the AI became up to 40.2 times faster.
Accuracy: The most important part is that the AI didn't get "dumber." The researchers found that the answers were almost identical to the original, slower version. In some cases, like writing code, the speed-up even helped the AI generate slightly better answers because it could explore more options in the same amount of time.
Versatility: It worked well on different types of tasks, including reasoning (solving puzzles) and code generation (writing computer programs).

The paper also notes a few limits. The "perfect" caching only works on the very first layer of the AI's brain; deeper layers are more complex and might need slightly different tricks. Also, this speed boost relies on having modern computer chips that are good at handling these smaller numbers (FP8). If you have an old computer, it might not be able to use this trick yet.

Why It Matters

In the world of AI, speed and cost are everything. If an AI takes too long to think, it's expensive to run and frustrating to use. LaCache shows that we don't always need bigger, more powerful computers to get faster results; sometimes, we just need to stop doing the same work twice. By remembering what we already know and using smarter tools for the rest, we can make these powerful AI models run lightning-fast without losing their intelligence. It's a reminder that in the race for artificial intelligence, being efficient is just as important as being powerful.

Technical Summary: LaCache

Problem Statement

Diffusion-based Large Language Models (DLLMs) offer a promising alternative to autoregressive models by enabling parallel generation through Semi-Autoregressive (SAR) decoding. In this paradigm, text generation is formulated as an iterative denoising process where tokens within a block are updated simultaneously while bidirectional attention leverages full context. However, current open-source DLLMs suffer from severe operator-level redundancy.

During the multiple denoising steps required to generate a single SAR block, the token IDs for the prompt (prefix) and the masked suffix remain invariant. Despite this, standard inference executes a full forward pass at every step, redundantly recomputing intermediate states for these unchanged tokens. This redundancy manifests in:

Token-wise operators: Embedding lookups, normalization, QKV projections, and Rotary Position Embeddings (RoPE).
Attention mechanisms: Specifically, the tiled calculation within the first layer of FlashAttention, where the same Key-Value (KV) blocks are processed repeatedly for unchanged queries.

Existing acceleration techniques, such as approximate KV caching or context pruning (e.g., DPad), primarily reduce the number of denoising steps or effective context length but do not directly eliminate the redundant computation of invariant tokens within a SAR block.

Methodology

The authors propose LaCache, a training-free acceleration framework designed to eliminate this redundancy through Lossless State Memoization (LSM) and mixed-precision inference.

1. Lossless State Memoization (LSM)

LaCache caches three specific types of intermediate results for tokens that remain unchanged ( $U$ ) within a SAR block, allowing the model to skip redundant computations in subsequent steps while ensuring mathematically equivalent outputs:

EmbedCache: Stores the embedding outputs for tokens in the invariant region $U$ .
RoPECache: Stores the first-layer token-wise pre-attention states for tokens in $U$ . This includes the normalized hidden states after QKV projections and RoPE application.
FACache (FlashAttention Cache): Stores the online softmax statistics for the first-layer FlashAttention. Specifically, it caches the running maximum ( $m$ $m$ ), the normalizer ( $\ell$ $ℓ$ ), and the unnormalized output accumulator ( $\tilde{O}$ $\tilde{O}$ ) for query blocks that do not overlap with the current generation block.
- Mechanism: In subsequent steps, if a query block and a KV block both belong to the invariant region, the calculation is skipped entirely, and the cached state is reused. If only the KV block is invariant, the cached state initializes the registers, skipping the redundant accumulation.

2. Mixed-Precision Strategy

To address memory-bandwidth bottlenecks and further accelerate compute-intensive layers, LaCache employs a per-group FP8 quantization strategy for Feed-Forward Network (FFN) layers (Gate-Up and FF-Out linear layers).

Granularity: The strategy uses a fine-grained per-group approach (1x128 for activations, 128x128 for weights) to adapt to diverse activation distributions observed across different denoising steps and layers.
Layer Selection: Quantization is applied to all layers except the first. The first layer is kept in FP16 because applying quantization there causes significant precision degradation due to the sensitivity of initial embeddings.
Optimization: The quantization and dequantization operations are fused with surrounding operators (AddNorm and Swiglu) to minimize memory access overhead and leverage FP8 Tensor Cores.

3. Inference Workflow

The framework operates by partitioning the sequence into blocks. For the first denoising step of a block, the model computes full forward passes and updates the caches. For subsequent steps within the same block, the model:

Retrieves cached embeddings and RoPE states for invariant tokens.
Initializes FlashAttention with cached softmax statistics for invariant regions.
Computes only the necessary operations for the changing tokens in the current block.
Applies FP8 quantization to FFN layers for the remaining computation.

Key Contributions

Identification of Redundancy: The paper identifies and quantifies operator-level redundancy in SAR inference, specifically highlighting the repetition of token-wise operators and FlashAttention tiling on invariant tokens.
Lossless Caching Mechanism: The introduction of three specialized cache components (EmbedCache, RoPECache, FACache) that allow for the skipping of redundant computations without altering the model's output distribution.
Adaptive Mixed-Precision: A per-group FP8 quantization strategy tailored for the specific activation dynamics of DLLMs, which accelerates linear layers while maintaining accuracy.
Composability: The framework is designed to be training-free and composable with existing acceleration methods (e.g., confidence-based decoding, suffix dropout).

Experimental Results

Experiments were conducted on NVIDIA H200 GPUs using LLaDA (base, instruct, and 1.5) models across reasoning (GSM8K, MATH) and code generation (HumanEval, MBPP) benchmarks.

Speedup:
- Standalone: LaCache alone achieves approximately 1.3× end-to-end speedup over vanilla DLLM inference.
- Combined: When combined with existing acceleration methods (Fast-dLLM and DPad), LaCache achieves up to 40.2× end-to-end speedup on long-context tasks.
- Baseline Comparison: On standard sequences, it delivers 3–12× speedups.
Accuracy: The method maintains comparable task accuracy to the vanilla backbone. In some cases (e.g., HumanEval), accuracy slightly improved, attributed to the model generating more complete output formats due to the low-precision strategy's tendency to delay end-of-sequence tokens slightly.
Ablation: Removing the FP8 quantization (using only caching) still yields a ~3% speedup, confirming the efficacy of the caching mechanism alone.

Significance and Claims

The paper claims that LaCache provides a general, training-free, and complementary solution for efficient DLLM deployment. Its primary significance lies in addressing a specific, previously overlooked source of inefficiency: the redundant computation of invariant tokens within the SAR decoding loop.

By strictly memoizing intermediate states, LaCache achieves lossless acceleration, ensuring that the speedup does not come at the cost of generation quality. The authors emphasize that this approach is distinct from methods that approximate the model or reduce context length; instead, it optimizes the execution of the exact same mathematical operations by avoiding their repetition. The framework is presented as a practical enabler for deploying DLLMs in latency-sensitive scenarios without requiring model retraining or architectural changes.

Limitations

The authors acknowledge two primary limitations:

Layer Restriction: The lossless caching is theoretically restricted to the first layer. Implementing similar caching in deeper layers is difficult due to the bidirectional propagation of hidden states, which would require approximate methods potentially degrading quality.
Hardware Dependency: The mixed-precision strategy relies on modern accelerators with native FP8 support (e.g., NVIDIA H200). Deployment on older hardware lacking optimized low-precision matrix engines is restricted.