🤖 AI

Kaleido: Algorithm-Hardware Co-Design for Video Diffusion Transformers by Exploiting Latent Space Correlations

Kaleido is an algorithm-hardware co-design that accelerates Video Diffusion Transformers by exploiting channel-wise spatiotemporal correlations in latent space to enable a lightweight reuse algorithm and a reconfigurable systolic array accelerator, achieving up to 5.9x speedup and 16.0x energy savings while maintaining high generative quality.

Original authors: Wenxuan Miao, Haosong Liu, Weiming Hu, Zihan Liu, Aiyue Chen, Jianlin Yu, Yiwu Yao, Yiming Gan, Jieru Zhao, Jingwen Leng, Minyi Guo, Yu Feng

Published 2026-07-16

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Wenxuan Miao, Haosong Liu, Weiming Hu, Zihan Liu, Aiyue Chen, Jianlin Yu, Yiwu Yao, Yiming Gan, Jieru Zhao, Jingwen Leng, Minyi Guo, Yu Feng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers can dream up entire movies from a single sentence, creating scenes so real you could almost touch them. This magic is powered by "video diffusion models," a type of artificial intelligence that learns to create videos by starting with a screen full of static noise and slowly, step-by-step, cleaning it up until a clear picture emerges. Think of it like a sculptor starting with a rough block of stone and chipping away the excess to reveal a statue. However, this process is incredibly slow and hungry for power. The computer has to perform millions of tiny calculations for every single frame, over and over again, like a chef tasting a soup hundreds of times before serving it. As these AI models get better at making longer, higher-quality videos, the time and energy required to run them have become a massive bottleneck, slowing down everything from movie editing to virtual world creation. Scientists and engineers are racing to find ways to speed this up without ruining the quality of the final movie.

Enter Kaleido, a clever new invention from researchers at Shanghai Jiao Tong University and Huawei that acts like a super-smart shortcut for these video-making computers. The team discovered that while the AI is busy "cleaning" the video, it often repeats the same calculations over and over again because the video frames are so similar to each other in specific ways. Instead of recalculating everything from scratch, Kaleido's algorithm acts like a brilliant librarian who remembers, "Hey, I already did this part for the last frame, so let's just reuse that answer!" But there's a catch: these shortcuts create a messy, irregular pattern of work that standard computer chips hate, much like a chef trying to chop vegetables with a dull knife that keeps slipping. To solve this, the researchers didn't just write a new recipe; they built a brand-new kitchen. They designed a custom hardware chip with a special "data dispatcher" that organizes the messy shortcuts into neat, efficient batches, allowing the computer to skip redundant work without dropping a single drop of quality.

The result is a system that is significantly faster and more energy-efficient than anything currently available. In their tests, Kaleido made video generation up to 5.9 times faster and saved 16.0 times more energy compared to the best existing accelerators and high-end graphics cards. Perhaps most impressively, the videos it produced were not just fast; they were sharper and more accurate than those made by other shortcut methods, scoring over 17 dB higher in image quality metrics. The researchers proved that by understanding exactly how the video data is structured—specifically how different parts of the image relate to time and space—they could build a system that skips the boring parts of the work while keeping the creative parts perfect. This isn't just a small tweak; it's a fundamental change in how we tell computers to make movies, proving that sometimes the best way to go faster is to know exactly what you don't need to do.

Technical Summary: Kaleido

Problem Statement

Video Diffusion Transformers (vDiTs) have emerged as the state-of-the-art architecture for high-fidelity video generation, yet they face severe computational bottlenecks. The inference process involves iterative denoising over many timesteps, where the self-attention mechanism dominates the execution time, accounting for approximately 68% of total runtime. As video resolutions and durations increase, the token sequence length grows, causing self-attention complexity to scale quadratically ( $O(N^2 \times d)$ ).

Existing acceleration strategies largely adapt sparse attention techniques from Large Language Models (LLMs), which rely on pruning insignificant attention scores based on semantic sparsity. However, these approaches are fundamentally mismatched for video data. Unlike text, video tokens carry dense, continuous visual semantics where every token contributes to spatial coherence and temporal consistency. Naively pruning attention correlations leads to motion discontinuities and flickering artifacts. Furthermore, as diffusion timesteps are reduced via distillation, self-attention becomes the sole dominant bottleneck, rendering timestep-reduction techniques insufficient.

Methodology

Kaleido addresses these challenges through an algorithm-hardware co-design that exploits channel-wise spatio-temporal correlations within the latent space of vDiTs, rather than relying on token-level sparsity.

1. Algorithm: Channel-Wise Reuse

The core insight is that vDiT models utilize Rotary Positional Embedding (RoPE), which partitions token channels into three distinct groups: temporal ( $t$ ), horizontal ( $x$ ), and vertical ( $y$ ). These groups encode positional information at different frequencies, inducing specific patterns in attention scores.

Correlation Discovery: The authors demonstrate that adjacent tokens within a specific channel group exhibit high similarity along dimensions orthogonal to their encoding direction. For instance, $t$ -channels are highly correlated across spatial dimensions ( $x$ and $y$ ), while $x$ -channels are correlated across the temporal dimension.
Reuse Mechanism: Instead of skipping computations (as in sparse attention), Kaleido reuses partial results.
- Similarity Check: Before computation, the system calculates the similarity ( $\Delta$ ) between adjacent tokens along a predefined axis.
- Thresholding: Two thresholds, $\theta_{th1}$ $θ_{t h 1}$ and $\theta_{th2}$ $θ_{t h 2}$ , determine the reuse strategy:
  - If $\Delta < \theta_{th1}$ : Full Reuse. The token completely reuses the partial attention score of the previous token.
  - If $\theta_{th1} \le \Delta < \theta_{th2}$ : Partial Reuse. The token reuses the most significant bits (e.g., top 3 bits) of the previous result and only computes the remaining lower bits.
  - If $\Delta \ge \theta_{th2}$ : Full Computation.
- Applicability: This strategy is applied to self-attention, cross-attention, and MLP layers. For MLPs, reuse is applied to activations rather than weights.

2. Hardware: Reconfigurable Systolic Accelerator

The irregular data access patterns introduced by the reuse algorithm are incompatible with standard systolic arrays. Kaleido introduces a custom hardware architecture to support this:

Reconfigurable Processing Elements (PEs): The PE array is augmented with additional registers and multiplexers to support four execution modes. These modes allow PEs to handle full computation, full reuse, and partial reuse (half-bit computation) dynamically within a single tile. The PEs can accumulate partial results from previous frames without reloading them from on-chip memory.
Data Dispatcher: To mitigate the irregular sparsity that causes low PE utilization, a dedicated data dispatcher is introduced. It consists of:
- Scheduler: Groups tokens with similar reuse patterns (represented as binary strings) into clusters.
- Matcher: Pairs compatible input channels within a cluster to merge data streams, ensuring that adjacent rows fed into the PE array can be processed simultaneously, thereby maximizing utilization.

Key Contributions

Systematic Characterization of vDiTs: The paper is the first to systematically explain the spatio-temporal correlations in vDiT latent spaces, identifying RoPE-based channel partitioning as the root cause of distinct attention patterns.
Channel-Wise Reuse Algorithm: A lightweight algorithm that skips redundant computations by reusing partial results at the channel level. Unlike token-level merging, this preserves pixel-level precision, achieving significantly higher generative quality (>17 dB PSNR improvement over prior co-designs).
Co-Designed Accelerator Architecture: A reconfigurable systolic array with tailored PEs and a data dispatcher that handles irregular sparsity, improving PE utilization from ~69% to ~85%.
Comprehensive Evaluation: Extensive evaluation across four mainstream vDiT models (HunyuanVideo, Wan, CogVideoX, TurboDiffusion) against state-of-the-art GPUs and accelerators.

Results

Evaluated on a 16nm technology node with a 128x128 PE array configuration:

Performance: Kaleido achieves up to 5.9× speedup over state-of-the-art accelerators (e.g., AdapTiV, Exion) and 6.6× speedup over an Nvidia A100 GPU.
Energy Efficiency: The system delivers up to 16.0× energy savings compared to the A100 baseline.
Generative Quality: Kaleido maintains superior video quality, achieving PSNR scores of 29.9 dB (HunyuanVideo) and 24.5 dB (Wan), which are >17 dB higher than approximation-based baselines like Cambricon-D. It also outperforms software-optimized sparse attention methods (e.g., MInference, SVG) in image quality metrics (PSNR, SSIM, LPIPS).
Scalability: The performance gap widens as video resolution increases from 480p to 720p, as higher resolutions strengthen the spatio-temporal correlations exploited by the algorithm.

Significance

The paper positions Kaleido as a necessary evolution in video generation acceleration. It argues that simply transferring LLM-based sparse attention techniques to video is ineffective due to the continuous nature of visual data. By shifting the focus from "pruning" to "reusing" based on the unique channel-wise correlations of vDiTs, Kaleido offers a principled approach that simultaneously addresses the computational bottleneck and preserves the high-fidelity requirements of generative AI. The work demonstrates that algorithm-hardware co-design is essential to unlock the efficiency of next-generation video diffusion models without compromising output quality.