CHAI: CacHe Attention Inference for text2video

CHAI accelerates text-to-video diffusion inference by introducing Cache Attention, a mechanism that reuses cached latents across semantically related prompts to achieve 1.65x–3.35x speedups over OpenSora 1.2 while maintaining high video quality with as few as 8 denoising steps.

The Gist

Imagine you are an artist tasked with painting a movie scene from scratch, frame by frame. You start with a blank canvas covered in static noise (like TV snow). To get a clear picture of a "sunny beach," you have to slowly wipe away the noise, refining the image step-by-step.

The Problem:
Current AI video generators (like OpenSora) are like incredibly talented but slow painters. To make a smooth, high-quality video, they need to wipe away the noise 30 to 50 times for every single frame. This takes a long time, making it hard to use these tools for real-time applications or quick projects.

The Old Solutions:

1. Retraining: Teaching the painter a new, faster way to paint. This is expensive and takes months.
2. Skipping Steps: Telling the painter, "Just skip the middle steps, I'll guess what happens." This is fast, but the painting often looks blurry, glitchy, or the objects melt into each other.

The New Solution: CHAI (CacHe Attention Inference)
The authors of this paper, CHAI, came up with a clever trick. Instead of forcing the painter to start from total noise every time, they say: "Let's look at what we painted before."

Here is how CHAI works, explained with simple analogies:

1. The "Whole Prompt" vs. "Entity" Mistake

Imagine you ask an AI to draw "A tiger running in a jungle."

• The Old Way (NIRVANA): The system looks at your entire sentence. It asks, "Have I ever drawn a tiger running in a jungle before?" If the answer is "No, I only drew a tiger sitting in a cave," it says, "No match! Start from scratch."
• The CHAI Way: CHAI is smarter. It breaks the sentence down into entities (objects and scenes). It asks, "Have I ever drawn a tiger?" Yes. "Have I ever drawn a jungle?" Yes.
  • Analogy: It's like a chef. If you order "Spicy Chicken Tacos," and the chef has already made "Spicy Chicken Burritos," a normal chef might say, "That's a different dish, I'll start from zero." A CHAI chef says, "I already have the spicy chicken sauce and the tortillas ready! I just need to change the shape."

2. The "Magic Attention" Lens (Cache Attention)

This is the secret sauce. You can't just copy-paste the old "tiger" into the new video, because the new video might need the tiger to be running, not sitting.

CHAI introduces a mechanism called Cache Attention.

• Analogy: Imagine the old video is a library of reference photos. When the AI starts painting the new video, it doesn't just grab the whole photo. Instead, it puts on special glasses (the attention mechanism).
• These glasses let the AI look at the old "tiger" photo and say, "Okay, I'll use the shape of the tiger and the color of the fur from the old photo, but I will ignore the 'sitting' pose."
• It selectively "picks out" the useful parts (the entities) from the old work and mixes them with the new instructions. This allows the AI to skip the boring, repetitive early steps of figuring out "what a tiger looks like" and jump straight to "how to make this tiger run."

3. The Result: Speed without the Blur

Because CHAI reuses the "skeleton" of previous videos (the objects and scenes), it doesn't need to start from scratch.

• The Baseline: Needs 30 steps to paint a perfect video.
• CHAI: Only needs 8 steps.

The Analogy:

• Standard AI: Like building a house from the ground up every time you want a new room. You have to pour the concrete, lay the bricks, and frame the walls from scratch.
• CHAI: Like having a warehouse of pre-built walls and windows. If you want a new room, you just grab the pre-made walls (the cached entities), assemble them, and paint the final details. You skip the heavy lifting.

Why is this a big deal?

• Speed: It makes video generation 1.6 to 3.3 times faster.
• Quality: Unlike other "fast" methods that make videos look like glitchy messes, CHAI keeps the quality almost identical to the slow, perfect version.
• Efficiency: It works even if you only have a small amount of computer memory (cache), because it's smart enough to know that "beach" and "ocean" are similar enough to reuse parts of the work.

In a nutshell:
CHAI is like a super-efficient assistant who remembers the details of your past projects. Instead of making you start over every time you ask for a video, it says, "I've already done the hard part of figuring out what a 'beach' looks like. Let's just tweak that for your new request." This saves hours of waiting time while keeping the video looking crisp and clear.

Technical Summary

1. Problem Statement

Text-to-video diffusion models (e.g., OpenSora, Sora) have achieved high visual fidelity but suffer from significant inference latency. This bottleneck stems from the sequential nature of Denoising Diffusion Probabilistic Models (DDPMs), which require 30–50 iterative denoising steps to generate coherent 3D (spatiotemporal) latents.

Existing acceleration strategies face two main limitations:

• Training-based approaches: Require expensive retraining and are tightly coupled to specific architectures, making them impractical for rapid deployment.
• Training-free approaches:
  • Intra-inference caching (e.g., AdaCache): Skips redundant steps within a single generation based on feature differences. However, it cannot skip early steps where major structural changes occur, limiting latency reduction without severe quality degradation.
  • Cross-inference caching (e.g., NIRVANA for images): Reuses latents from previous generations. While effective for text-to-image, naive extension to video fails because video prompts are long, diverse, and descriptive, resulting in extremely low whole-prompt similarity and poor cache hit rates.

2. Methodology: CHAI

The authors propose CHAI (CacHe Attention Inference), a training-free system that accelerates video generation by reusing diffusion latents across inference runs based on entity-level similarity rather than whole-prompt similarity.

Core Concepts

• Entity-Level Similarity: The paper observes that while full video prompts differ, they often share common entities (objects like "tiger," "car" or scenes like "beach," "forest"). By caching at this granularity, the system achieves significantly higher cache hit rates (up to 80% in realistic workloads) compared to whole-prompt matching (<40%).
• Cache Attention: A novel attention mechanism designed to integrate cached latents without introducing noise or artifacts.
  • Mechanism: Instead of directly substituting the entire latent (which fails when prompts differ slightly), Cache Attention injects cached information into the Key (K) and Value (V) vectors of the attention mechanism.
  • Operation: The query (Q) remains derived from the current prompt-modulated noise. This allows the model to selectively attend to relevant features from the cached latent (e.g., the "beach" structure) while preserving prompt-specific details (e.g., "spring waves").
  • Scheduling: Cache Attention is applied only to the 2nd, 3rd, and 4th denoising steps.
    • Step 1 is skipped: The query is random Gaussian noise, making cache injection ineffective.
    • Steps >4 are skipped: Empirical results show diminishing returns after the 4th step, and storing more steps increases memory overhead unnecessarily.

System Architecture

1. Entity Extractor: Uses NLP (Spacy) to identify objects and scenes in the input prompt.
2. Vector DB: Stores embeddings of entities to facilitate fast similarity search.
3. Latent Store: Caches the actual diffusion latents (tensors) associated with previous prompts.
4. LRU Policy Engine: Manages cache eviction when storage limits are reached.
5. Dual-Mode Inference:
   • Full Mode (Cache Miss): Runs standard 30-step diffusion with vanilla self-attention.
   • Fast Mode (Cache Hit): Runs a reduced 8-step diffusion, enabling Cache Attention layers to utilize stored latents.

3. Key Contributions

1. CHAI System: A training-free framework for accelerating text-to-video diffusion via cross-inference caching at the entity level.
2. Cache Attention Mechanism: A novel attention layer that enables selective reuse of entity-level information from cached latents, allowing for high-quality generation with drastically fewer steps.
3. Empirical Validation: Demonstrated that high-quality video can be generated with as few as 8 denoising steps (compared to the standard 30) while maintaining quality comparable to the baseline.
4. Scalability: Proved the system is practical under modest storage budgets (1–5 GB), achieving high cache hit rates (>80%) without retraining.

4. Experimental Results

The system was evaluated against OpenSora 1.2 (30 steps), NIRVANA-VID (naive whole-prompt reuse), and AdaCache (intra-inference skipping) on the VBench and VidProM datasets.

• Speed vs. Quality (VBench):
  • CHAI (8 steps): Achieved a 3.35x speedup over OpenSora 1.2 (3.75s vs. 12.54s) with a VBench score of 0.7985 (only 0.3% lower than the baseline 0.8018).
  • NIRVANA-VID: Suffered severe quality degradation due to direct latent substitution on mismatched prompts.
  • AdaCache: Achieved a 2.87x speedup but with lower quality (0.7883) compared to CHAI.
• Real-World Workload (VidProM):
  • Under a constrained 10% cache budget, whole-prompt reuse (NIRVANA) had a hit rate of only 14.7%, offering negligible speedup.
  • CHAI achieved a 52% hit rate using entity similarity, resulting in a 1.65x speedup (7.63s vs. 12.59s) with minimal quality loss.
• Storage Efficiency:
  • With just 1 GB of cache (approx. 780 prompts), CHAI achieved an 83.4% hit rate, saturating to 89.4% at 5 GB.

5. Significance

CHAI addresses a critical bottleneck in the deployment of generative video AI. By shifting the caching paradigm from whole-prompt to entity-level and introducing Cache Attention, the authors demonstrate that:

1. Latency can be reduced by >3x without retraining the underlying model.
2. Video quality is preserved even with a drastic reduction in denoising steps (from 30 to 8), challenging the assumption that fewer steps inherently lead to poor coherence.
3. Cross-inference reuse is viable for video, provided the system exploits semantic granularity (objects/scenes) rather than literal prompt matching.

This work provides a scalable, deployment-ready strategy for making interactive, real-time text-to-video applications feasible on current hardware.