💻 computer science

Accelerating Diffusion-based Video Editing via Heterogeneous Caching: Beyond Full Computing at Sampled Denoising Timestep

This paper introduces HetCache, a training-free framework that accelerates diffusion-based video editing by exploiting architectural redundancy within Diffusion Transformers to selectively cache context tokens, achieving a 2.67 $\times$ latency speedup and significant FLOPs reduction with negligible quality degradation.

Original authors: Tianyi Liu, Ye Lu, Linfeng Zhang, Chen Cai, Jianjun Gao, Yi Wang, Kim-Hui Yap, Lap-Pui Chau

Published 2026-03-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tianyi Liu, Ye Lu, Linfeng Zhang, Chen Cai, Jianjun Gao, Yi Wang, Kim-Hui Yap, Lap-Pui Chau

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Over-Engineered" Video Editor

Imagine you have a very talented but incredibly slow artist. This artist is an AI that can take a rough sketch and turn it into a beautiful, moving video. This is what Diffusion Models do.

However, there's a catch:

The Process is Repetitive: To create the video, the artist has to make thousands of tiny adjustments (called "denoising steps"). It's like trying to sculpt a statue by chipping away a single grain of sand at a time, over and over again.
The Artist is Over-Working: In every single step, the artist looks at every single part of the video (every pixel, every frame) to decide what to change. Even if 90% of the video is just a static blue sky that doesn't need changing, the artist still spends time analyzing it in detail.

This makes the process extremely slow and expensive (in terms of computer power), making it hard to use for real-time editing.

The Old Solution: "Skip a Few Steps"

Previous methods tried to speed this up by saying, "Hey, the sky hasn't changed much in the last 10 steps, let's just copy the last picture of the sky and skip the work."

This helps a little, but it's like a chef who decides to stop tasting the soup every few minutes. Sometimes, the soup does need a tiny pinch of salt, but the chef skips it, and the flavor suffers. Also, these methods still treat the whole video the same way, even if only a small part of it is being edited.

The New Solution: HetCache (The Smart Editor)

The authors of this paper, HetCache, realized that video editing isn't uniform. When you edit a video, you usually focus on a specific area (like replacing a person's shirt or removing a trash can), while the rest of the background stays the same.

They came up with a strategy that acts like a Smart Project Manager for the AI artist. Instead of treating every part of the video equally, HetCache uses two main tricks:

Trick 1: The "Traffic Light" System (Time Management)

Imagine the video creation process is a long road trip.

Green Light (Full Compute): Sometimes the road is bumpy and changing fast (e.g., the AI is figuring out the shape of a new object). The artist must work hard and look at everything.
Yellow Light (Partial Compute): The road is mostly smooth, but there are a few bumps. The artist can take a shortcut, looking at only the important parts.
Red Light (Reuse): The road is perfectly flat and straight. The artist doesn't need to drive at all; they can just coast on the momentum from the last step.

HetCache automatically detects which "light" is on and decides whether to do the full work, a partial work, or just reuse the old result.

Trick 2: The "Focus Group" Strategy (Token Management)

This is the real magic. In video editing, you have:

The "Generative" Zone: The area you are editing (e.g., the new shirt). This needs 100% attention.
The "Context" Zone: The background (e.g., the sky, the trees). This needs to stay consistent but doesn't need constant re-analysis.

Old methods treated the Background and the Shirt the same way. HetCache is smarter. It realizes that the Background is huge and repetitive.

The Analogy: Imagine you are writing a report about a party. You need to describe the new guest (the Shirt) in detail. But for the old guests (the Background), you don't need to interview every single one of them. You just need to ask one representative from each group of friends to summarize what everyone is doing.
How it works: HetCache groups the background pixels into "clusters" (like groups of friends). It picks the most "representative" pixel from each group to talk to the new object. It ignores the rest of the background pixels because they are just repeating the same information.

Why is this a Big Deal?

By combining these two tricks, HetCache acts like a highly efficient editor who knows exactly when to work hard and when to take a shortcut.

Speed: It makes the video generation 2.67 times faster.
Quality: It doesn't ruin the video. Because it keeps the "representative" background pixels, the new object blends in perfectly, and the background doesn't get blurry or weird.
No Retraining: The best part? You don't need to teach the AI anything new. It's a "plug-and-play" tool that works with existing models immediately.

Summary

Think of HetCache as a way to stop a supercomputer from doing math problems it already knows the answer to. It looks at the video, sees what's changing and what's staying the same, and only does the heavy lifting where it actually matters. This turns a slow, expensive process into something fast and practical, bringing us one step closer to editing videos in real-time.

1. Problem Statement

Diffusion Transformers (DiTs) have become the state-of-the-art for high-quality video generation and editing (specifically Masked Video-to-Video or MV2V). However, they face significant computational bottlenecks due to:

Iterative Denoising: The need for repeated forward passes over many timesteps.
Quadratic Complexity: The self-attention mechanism in DiTs scales quadratically with the number of spatio-temporal tokens ( $O(N^2)$ ).
Redundancy: Existing acceleration methods (like feature caching) primarily focus on temporal redundancy (reusing features across timesteps) but ignore architectural redundancy within a single timestep.
The MV2V Challenge: In video editing, tokens are not uniform. There are masked (generative) regions requiring synthesis and unmasked (context) regions providing guidance. Uniform caching or random token sampling degrades editing quality because it fails to distinguish between tokens that are critical for semantic consistency and those that are redundant.

2. Methodology: HetCache

The authors propose HetCache, a training-free acceleration framework that exploits heterogeneity in two dimensions: denoising timesteps and spatio-temporal tokens.

A. Heterogeneous Timestep Scheduling

Instead of treating all timesteps equally, HetCache estimates the "drift" or variation in the model's output between steps to decide the computation mode:

Metric: It calculates the relative $L_1$ distance between modulated inputs ( $F_t = T_t \odot x_t$ ) of adjacent timesteps.
Accumulation: It accumulates this difference ( $D_{a \to b}$ ) over consecutive steps.
Three Regimes:
- Full-Compute Step: If the accumulated difference is large ( $D > 1.5\Delta$ ), the model performs a full forward pass and updates the cache.
- Partial-Compute Step: If the difference is moderate ( $\Delta < D \le 1.5\Delta$ ), the model performs a lightweight update using cached representations and a subset of tokens.
- Reuse Step: If the difference is small ( $D \le \Delta$ ), the cached output is reused without any new computation.

B. Token-Level Heterogeneous Caching (ROI-Aware)

Within Full-Compute and Partial-Compute steps, HetCache reorganizes tokens based on the editing mask (Region of Interest - ROI):

Token Partitioning:
- Generative Tokens: Masked regions (must be synthesized).
- Margin Tokens: Unmasked tokens adjacent to the mask boundary (ensure smooth transitions).
- Context Tokens: Unmasked tokens far from the mask (provide global semantics).
Selective Caching Strategy:
- Generative & Margin Tokens: Always fully computed to ensure fidelity and boundary smoothness.
- Context Tokens: These are the source of redundancy. HetCache selects only the most semantically representative context tokens to participate in attention calculations.
Selection Mechanism:
- Clustering: Context tokens are clustered using K-Means in semantic space.
- Importance Scoring: Within each cluster, tokens are ranked by their cached attention scores to generative tokens ( $\alpha_i$ ). This measures how much a specific context token influences the generation of the masked area.
- Sampling: The top- $r_{ctx}$ proportion of tokens from each cluster are selected to form the representative set $X^*_{ctx}$ .
- Result: The attention complexity is reduced from $O((X_c + X_m + X_g)^2)$ to $O((r_{ctx}X_c + X_m + X_g)^2)$ , significantly lowering FLOPs while preserving semantic guidance.

3. Key Contributions

Token-Level Analysis: The paper identifies and analyzes the inherent token heterogeneity in DiT-based MV2V tasks, revealing that context tokens have varying levels of importance based on their interaction strength with generative regions.
Heterogeneous Caching Framework (HetCache): A novel, training-free method that jointly optimizes:
- Temporal Dimension: Adaptive scheduling of full/partial/reuse steps based on output variation.
- Token Dimension: ROI-aware selection of context tokens based on semantic clustering and attention correlation.
State-of-the-Art Efficiency: The method achieves significant speedups and FLOPs reduction without requiring model retraining or fine-tuning, making it applicable to existing foundation models.

4. Experimental Results

The method was evaluated on Wan2.1-VACE and LTX-Video backbones using the VACE-Benchmark (Video Inpainting) and VPBench (Text-guided Editing).

Performance Metrics:
- Speedup: Achieved up to 2.67× latency speedup compared to the baseline (Wan2.1-VACE with 100 steps).
- FLOPs Reduction: Reduced computational cost to 23.60 PFLOPs (from 145.21 PFLOPs) for video inpainting.
- Quality: Maintained competitive visual quality with negligible degradation.
  - VFID: 54.51 (vs. 57.18 baseline).
  - VBench Score: ~75.88% (vs. 76.54% baseline).
  - PSNR/SSIM: Comparable or slightly improved in some configurations.
Comparisons: Outperformed existing methods like TeaCache (which only handles temporal redundancy) and PAB/AdaCache, particularly in maintaining boundary smoothness and avoiding "ghosting" artifacts in dynamic scenes.
Ablation Studies: Confirmed that both the K-Means clustering and the attention-based correlation scoring are essential; removing them (uniform sampling) leads to significant quality drops.

5. Significance

Practical Deployment: By being training-free, HetCache can be immediately applied to existing large-scale video diffusion models without the cost of retraining or collecting new datasets.
Paradigm Shift: It moves beyond simple temporal caching to a multi-dimensional redundancy exploitation strategy, addressing the specific structural inefficiencies of Transformer-based video editing.
Real-Time Potential: The significant reduction in inference latency and FLOPs brings diffusion-based video editing closer to real-time and interactive applications, which were previously constrained by high computational costs.
Generalizability: The approach is demonstrated to work across different model backbones (Wan2.1, LTX), resolutions, and video lengths, suggesting broad applicability to future DiT architectures.