PreciseCache: Precise Feature Caching for Efficient and High-fidelity Video Generation

PreciseCache is a plug-and-play framework that significantly accelerates video generation by employing Low-Frequency Difference-based step-wise caching (LFCache) and block-wise caching (BlockCache) to precisely identify and skip truly redundant computations, thereby achieving substantial speedups without sacrificing output quality.

The Gist

Imagine you are an artist trying to paint a beautiful, moving video of a duck swimming in a pond. You don't paint the whole thing in one go; instead, you start with a blurry, static-filled canvas and slowly refine it, step by step, removing the noise until the picture is clear. This is how modern AI video generators work.

However, there's a problem: It's incredibly slow. The AI has to do thousands of tiny calculations for every single frame, making the process take forever.

Some previous attempts to speed this up were like a lazy painter who, instead of painting the next step, just said, "Eh, it looks the same as the last step," and copied the previous image. But this often backfired. Sometimes the AI did need to make a big change, but the lazy painter skipped it, resulting in a video that looked glitchy or weird.

Enter PreciseCache. Think of it as a super-smart, hyper-efficient art assistant that knows exactly when to work hard and when to take a shortcut without ruining the masterpiece.

Here is how it works, broken down into two simple tricks:

1. The "Low-Res Sketch" Trick (LFCache)

The Problem: To know if the next step of the video needs a full, detailed painting or if it's just a tiny tweak, the AI usually has to do the full, heavy painting first. That defeats the purpose of trying to save time!

The Solution: PreciseCache uses a quick sketch.
Before doing the full, high-definition painting, the assistant grabs a tiny, blurry, low-resolution version of the canvas (a "downsampled latent"). It quickly sketches on this tiny version to see: "Is the duck moving? Is the water changing?"

• If the sketch changes a lot: The assistant knows, "Okay, this is a critical moment! We need to do the full, high-quality painting now."
• If the sketch barely changes: The assistant says, "Ah, the duck is just floating. The scene is stable. I'll just copy the last painting and skip the heavy work."

This is like checking the weather forecast on your phone (the quick sketch) before deciding whether to pack a heavy raincoat (the full painting). If the phone says it's sunny, you don't waste time packing the coat.

2. The "Skip the Boring Parts" Trick (BlockCache)

The Problem: Even when the AI does decide to do a full painting, it still wastes time on parts of the canvas that don't need much attention. Imagine an artist spending hours painting the sky blue, even though the sky hasn't changed color at all.

The Solution: PreciseCache breaks the painting process into small "blocks" or layers. It analyzes each layer to see which ones are doing the heavy lifting and which ones are just coasting.

• Pivotal Blocks: These are the layers making big changes (like drawing the duck's beak). The AI works on these normally.
• Non-Pivotal Blocks: These are the layers making tiny, almost invisible changes (like slightly adjusting the shade of the water). The AI realizes, "I don't need to recalculate this; I'll just reuse the result from the last step."

It's like a construction crew building a house. They know they need to pour fresh concrete for the foundation (Pivotal), but for the interior paint on a wall that hasn't moved, they can just reuse the color from the blueprint without mixing a new batch of paint.

The Result

By combining these two tricks, PreciseCache acts like a conductor who knows exactly when the orchestra needs to play loudly and when they can rest.

• Speed: It makes video generation 2.6 times faster.
• Quality: Unlike previous "lazy" methods that made videos look bad, PreciseCache keeps the video looking crystal clear and high-quality.

In short, PreciseCache stops the AI from wasting energy on things that don't matter, allowing it to generate beautiful videos in a fraction of the time, without sacrificing the details that make them look real.

Technical Summary

1. Problem Statement

Video generation models, particularly those based on Diffusion Transformers (DiTs), suffer from extremely slow inference speeds and high computational costs due to the iterative denoising process required to generate high-fidelity, temporally coherent videos.

While feature caching (reusing features from previous steps to skip full network inference) is a popular training-free acceleration technique, existing methods face significant limitations:

• Uniform Caching: Many methods use fixed intervals for caching, ignoring the varying importance of different timesteps.
• Quality Degradation: Adaptive methods often rely on suboptimal metrics or require complex hyperparameter tuning, leading to noticeable artifacts or content shifts in the generated videos.
• Inability to Distinguish Redundancy: Current approaches fail to precisely distinguish between truly redundant computations and critical features, often skipping steps that are essential for structural integrity or retaining steps that only add perceptually insignificant details.

2. Methodology: PreciseCache

The authors propose PreciseCache, a plug-and-play, training-free framework that accelerates video generation by precisely identifying and skipping redundant computations at two levels: step-wise (timestep level) and block-wise (network layer level).

The framework consists of two core components:

A. LFCache (Step-wise Caching via Low-Frequency Difference)

LFCache determines when to skip a full denoising step by analyzing the redundancy of the timestep.

• Insight: The authors observe that in the early (high-noise) stages of diffusion, the model generates critical low-frequency structural information. Skipping these steps severely degrades quality. In later (low-noise) stages, the model refines high-frequency details, which are perceptually less significant and can be safely cached.
• Metric (LFD): They introduce the Low-Frequency Difference (LFD). Instead of comparing full feature maps, they decompose the model's prediction into low-frequency ($F^{LF}$) and high-frequency ($F^{HF}$) components using Fast Fourier Transform (FFT). The LFD measures the difference in low-frequency components between the current step and the previous cached step.
• Efficient Estimation: Calculating LFD directly requires a full forward pass, which defeats the purpose of acceleration. To solve this, LFCache feeds a downsampled latent (e.g., reduced temporal and spatial resolution) into the model for a "trial" inference. This lightweight estimation is sufficient to calculate the LFD with negligible overhead.
• Decision Logic: An accumulated error ($E$) of LFD is maintained. If $E$ exceeds a threshold $\delta$, a full inference is triggered; otherwise, the cached prediction is reused.

B. BlockCache (Block-wise Caching)

Even when a step is not skipped entirely, redundancy exists within the network's transformer blocks.

• Insight: Not all transformer blocks contribute equally to the output. Some blocks (pivotal blocks) make significant modifications to the input features, while others (non-pivotal blocks) have minimal impact.
• Mechanism: For non-skipped steps, BlockCache calculates the difference between the input and output of each transformer block. It identifies the top $c\%$ of blocks with the largest differences as "pivotal."
• Execution: In subsequent steps, the outputs of non-pivotal blocks are skipped. Instead of running the block, the cached difference (residual) from the last full inference of that block is added to the input. Only the pivotal blocks undergo full computation.

3. Key Contributions

1. Precise Redundancy Detection: The paper identifies that previous caching failures stem from a lack of distinction between structural (low-frequency) and detail (high-frequency) changes. The proposed Low-Frequency Difference (LFD) metric effectively quantifies step-wise redundancy.
2. Two-Stage Acceleration Framework: The integration of LFCache (skipping entire steps based on global redundancy) and BlockCache (skipping internal transformer blocks based on local redundancy) allows for maximal acceleration without quality loss.
3. Training-Free & Plug-and-Play: The method requires no additional model training or distillation. It operates as a wrapper around existing DiT architectures (e.g., Wan2.1, HunyuanVideo, CogVideoX).
4. Efficient Estimation Strategy: The use of downsampled latents for LFD calculation ensures that the overhead of the caching decision mechanism is negligible compared to the savings gained.

4. Experimental Results

The method was evaluated on state-of-the-art video generation models including Open-Sora 1.2, HunyuanVideo, CogVideoX, and Wan2.1-14B.

• Speedup:
  • On Wan2.1-14B, PreciseCache achieves an average 2.6× speedup (specifically 2.63× with the "Flash" configuration) compared to the baseline.
  • On Open-Sora, it achieves up to 2.60× speedup.
  • It significantly outperforms previous SOTA methods like PAB, TeaCache, and FasterCache in terms of acceleration ratio.
• Quality Preservation:
  • Despite the high speedup, the method maintains high visual fidelity. Metrics such as VBench scores, SSIM, and PSNR remain comparable to the baseline (no caching).
  • Qualitative results show that PreciseCache avoids the content shifts and artifacts often seen in other caching methods.
• Scalability:
  • The method scales effectively across different GPU counts (1 to 8 A800 GPUs), achieving up to 14.52× speedup on Wan2.1-14B with 8 GPUs.
• Ablation Studies:
  • Confirmed that a downsample ratio of 2×4×4 (Temporal × Height × Width) offers the best trade-off between estimation accuracy and computational cost.
  • Demonstrated that the specific feature reusing strategy (prediction vs. residual) has less impact than the precision of the caching decision itself.

5. Significance

PreciseCache represents a significant advancement in making high-fidelity video generation practical for real-world applications. By moving away from heuristic or uniform caching strategies to a frequency-aware, precision-based approach, it solves the long-standing trade-off between inference speed and generation quality.

The ability to achieve ~2.6× acceleration without retraining or sacrificing visual quality makes large-scale video diffusion models (like Wan2.1-14B) significantly more accessible for deployment in scenarios requiring low latency, such as real-time content creation, interactive media, and large-scale video synthesis pipelines. The modular nature of the framework suggests it can be easily adapted to future generations of diffusion models.