🤖 machine learning

FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximation

FastCache is a framework that accelerates Diffusion Transformer inference by combining spatial-aware token selection, transformer-level latent caching, and k-NN-based token merging to reduce computational redundancy while maintaining generation quality through learnable linear approximation.

Original authors: Dong Liu, Yanxuan Yu, Jiayi Zhang, Yifan Li, Ben Lengerich, Ying Nian Wu

Published 2026-03-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dong Liu, Yanxuan Yu, Jiayi Zhang, Yifan Li, Ben Lengerich, Ying Nian Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an artist painting a massive, detailed mural of a bustling city street. You are working on a computer that helps you generate this image pixel by pixel, but the computer is incredibly slow because it tries to redraw every single pixel of the entire city for every single frame of animation, even if nothing has changed.

If you are painting a scene where a car drives by, the computer wastes time re-calculating the color of the sky, the texture of the sidewalk, and the windows of buildings that haven't moved a millimeter. It's like a chef re-chopping onions and re-boiling water for every single bite of soup, even though the pot hasn't changed.

FastCache is a new "smart assistant" for these AI artists (specifically a type of AI called a Diffusion Transformer or DiT). Its job is to tell the computer: "Hey, stop wasting time! We already know what the sky looks like, and the sidewalk hasn't moved. Let's just reuse what we already calculated."

Here is how FastCache works, broken down into simple concepts:

1. The "Spot the Difference" Game (Spatial-Temporal Token Reduction)

In AI language, the image is made up of tiny pieces called "tokens" (like puzzle pieces).

The Problem: The AI checks every single puzzle piece in every single frame, even if 90% of them are just a static blue sky.
The FastCache Solution: FastCache acts like a vigilant security guard. It looks at the current frame and the previous frame.
- If a puzzle piece (token) is part of a moving car, the guard says, "Keep working on this! It's moving!"
- If a puzzle piece is part of a static building or the sky, the guard says, "Stop! We already know what this looks like. Just copy the last version."
The Analogy: Imagine you are watching a movie. If the camera is panning across a static wall, you don't need to re-read the bricks; you just remember the wall is there. FastCache does exactly this, skipping the work for the "boring" parts so the computer can focus on the "exciting" moving parts.

2. The "Lazy Reuse" Strategy (Transformer-Level Caching)

Even for the parts that are moving, the AI often does the same heavy math over and over again with very little change in the result.

The Problem: The AI runs a complex math formula (a "Transformer block") to figure out the next step. It's like solving a difficult math equation.
The FastCache Solution: FastCache checks: "Is the answer to this equation going to be totally different from the last time we solved it?"
- If the change is tiny (statistically insignificant), FastCache says, "Don't solve the whole equation again. Just use a simple shortcut (a linear approximation) to guess the answer based on the last one."
- If the change is huge, it forces the computer to do the full, heavy math.
The Analogy: Think of it like driving a car on a straight, flat highway. You don't need to constantly steer left and right; you can just hold the wheel steady (reuse the cache). You only need to steer hard when you hit a curve (the motion). FastCache knows when to hold the wheel steady and when to steer.

3. The "Smart Merge" (Token Merging)

Sometimes, there are so many tiny, identical puzzle pieces that they are just clutter.

The Solution: FastCache can group similar pieces together into one "super-piece" to save space, and then split them back out later if needed.
The Analogy: Instead of counting 1,000 individual grains of sand, you just say, "That's a pile of sand." You save time counting, and you still know exactly how much sand there is.

Why is this a big deal?

Before FastCache, generating high-quality AI video or images was like trying to run a marathon while carrying a heavy backpack full of bricks. It was slow and required massive amounts of computer power (and money).

FastCache takes off the backpack.

Speed: It makes the AI run 30% to 40% faster.
Quality: Unlike older methods that made the video look "frozen" or blurry when they tried to speed things up, FastCache is smart enough to know exactly what to skip. It keeps the moving parts sharp and the static parts efficient.
Memory: It uses less computer memory, meaning you can run these powerful models on cheaper hardware.

The Bottom Line

FastCache is like giving the AI a "brain" that understands redundancy. It realizes that in a video, most things don't change from one second to the next. By stopping the computer from doing unnecessary work, it allows us to generate high-quality AI videos and images much faster, cheaper, and with less energy, without sacrificing the beauty of the final result.

In short: It's the difference between a worker who re-paints the whole wall every day, and a smart worker who only paints the new cracks and leaves the rest alone.

1. Problem Statement

Diffusion Transformers (DiTs) have become the state-of-the-art architecture for high-fidelity image and video generation. However, they suffer from significant computational inefficiencies due to:

Iterative Structure: Generating content requires hundreds of denoising steps.
Deep Stacks: DiTs utilize deep transformer layers with extensive attention mechanisms.
Redundancy: In many scenarios (e.g., static backgrounds in video or low-motion regions), the hidden states across successive timesteps exhibit minimal variation. Current methods often recompute these redundant representations, leading to high latency and memory usage, which hinders real-time applications like video editing and VR.

Existing acceleration methods (e.g., token pruning, early exit, or frame-level caching) often fail to balance efficiency with generation quality or operate at coarse granularities (frame/pixel level) rather than exploiting redundancy within the latent hidden states of the transformer.

2. Methodology: FastCache

FastCache is a hidden-state-level caching and compression framework designed to accelerate DiT inference by exploiting spatial and temporal redundancy. It operates on the principle: "Cache the Background, Recompute the Motion."

The framework consists of three core components:

A. Spatial-Temporal Token Reduction Module

This module identifies and filters redundant tokens based on saliency:

Temporal Saliency: Calculates the $L_2$ norm of the difference between the current hidden state ( $X_t$ ) and the previous state ( $X_{t-1}$ ).
Classification: Tokens are split into Motion Tokens (high variation) and Static Tokens (low variation).
Processing:
- Motion Tokens: Processed through full transformer layers.
- Static Tokens: Bypassed through full layers and approximated using a learnable linear function ( $H_s = W_c X_s + b_c$ ), significantly reducing computation.

B. Transformer-Level Caching Module

This module decides whether to reuse cached outputs for specific transformer blocks across timesteps:

Decision Rule: It employs a statistical hypothesis test based on the relative change metric ( $\delta_{t,l}$ ) between the current hidden state and the cached state from the previous timestep.
Mechanism: If the change is statistically insignificant (below a threshold derived from a Chi-squared distribution), the transformer block is skipped. Instead of full computation, the output is approximated via a learnable linear projection ( $H_{t,l} = W_l H_{t,l-1} + b_l$ ).
Theoretical Guarantee: The method provides a bounded approximation error under the assumption of weak stationarity, ensuring that the cache does not degrade generation quality beyond a controllable limit.

C. Token Merging Module (Optional Enhancement)

To further reduce redundancy, FastCache introduces a kNN-based token merging strategy:

It computes a unified importance score combining spatial density (local feature clustering) and temporal saliency (motion dynamics).
Redundant tokens within dense clusters are merged into a single representative token via weighted averaging, reducing the sequence length before processing.

3. Theoretical Contributions & Interpretability

The paper provides a rigorous theoretical justification for FastCache, framing it as a Learnable Sparse Interaction Decomposition:

Motion-Background Separation: The hidden state is decomposed into a background component ( $B_t$ ) and a motion residual ( $M_t$ ).
Interaction Theory: Using Harsanyi interaction and Shapley values, the authors prove that for static regions, high-order interactions are negligible.
First-Order Approximation: FastCache effectively recovers first-order interactions (linear terms) while discarding higher-order terms that contribute minimally in static regions. This justifies the use of linear approximations for cached states, proving that the error is bounded by $O(\delta^2)$ where $\delta$ is the magnitude of motion.

4. Experimental Results

FastCache was evaluated on multiple DiT variants (DiT-S/2, B/2, L/2, XL/2) for both image and video generation tasks.

Performance Metrics:
- Latency: Achieved a 36.9% reduction in runtime on DiT-XL/2 compared to baselines.
- Memory: Reduced memory usage by 31.7%.
- Quality: Outperformed other caching methods (TeaCache, AdaCache, FBCache) in FID (Fréchet Inception Distance) and t-FID (temporal FID), indicating superior generation quality.
Comparisons:
- On DiT-XL/2, FastCache achieved an FID of 4.46 and t-FID of 13.15, surpassing AdaCache (4.64/13.55) and TeaCache (5.09/14.72) while being faster.
- It demonstrated strong generalization across different model sizes and motion conditions (static vs. dynamic videos).
Ablation Studies: Confirmed that the combination of Spatial Token Reduction, Statistical Caching, and Motion-aware Blending yields the best trade-off between speed and quality.

5. Key Contributions

Novel Framework: Introduced FastCache, the first hidden-state-level caching framework specifically for Diffusion Transformers, moving beyond frame-level or token-level pruning.
Dual Strategy: Developed a dual approach combining spatial-aware token selection and transformer-level statistical caching to maximize redundancy exploitation.
Theoretical Foundation: Provided a rigorous theoretical proof linking the caching mechanism to learnable linear approximation and sparse interaction decomposition, offering interpretability and error bounds.
State-of-the-Art Performance: Demonstrated that FastCache achieves the best balance of inference speed, memory efficiency, and generation quality (FID/t-FID) among existing acceleration methods.

6. Significance

FastCache addresses a critical bottleneck in the deployment of generative AI: computational cost. By intelligently identifying and skipping redundant computations in latent spaces without sacrificing visual fidelity, it makes high-fidelity video and image generation more accessible for real-time applications. The theoretical grounding ensures that these speedups are not just empirical hacks but are mathematically sound, paving the way for more efficient and scalable diffusion models in the future.