Technical Summary: Kandinsky 5.0 – A Family of Foundation Models for Image and Video Generation

1. Problem Statement

Despite rapid advancements in diffusion models and flow matching approaches for image generation, video generation remains a critical challenge in generative AI. The primary obstacles include the exponential growth in computational complexity when handling time-dependent 3D video data, difficulties in maintaining temporal consistency and motion realism, and the need for complex, multi-stage optimizations for both training and inference. While models like Sora and Veo have demonstrated high quality, creating efficient, controllable, and high-resolution video generation systems that are accessible to the research community remains a significant hurdle. Kandinsky 5.0 aims to address these challenges by providing a family of foundation models capable of high-resolution image and 10-second video synthesis with state-of-the-art (SOTA) performance and operational efficiency.

2. Methodology

2.1 Data Processing Pipeline

The training framework relies on a comprehensive, multi-stage data curation lifecycle involving collection, processing, filtering, and clustering:

• Text-to-Image (T2I): A dataset of over 500 million images undergoes rigorous filtering (resolution, deduplication, watermark detection, quality assessment via TOPIQ and Q-Align, text/complexity filtering) and synthetic caption generation using multimodal models (InternVL2, Qwen2.5VL).
• Text-to-Video (T2V): Over 250 million video scenes are segmented, deduplicated, and filtered for structural dynamics, technical/aesthetic quality (DOVER, Q-Align), and content. Synthetic captions are generated using Tarsier2-7B.
• Image Editing (I2I): A specialized pipeline constructs ~150 million image pairs with precise instructions. This involves similarity matching (CLIP, DINO, face recognition), geometric verification (LoFTR, RANSAC), and quality filtering to ensure high-fidelity transformation examples.
• Cultural & SFT Datasets: A Russian Cultural Code (RCC) dataset is manually curated for cultural specificity. A high-quality Supervised Fine-Tuning (SFT) dataset is created through expert manual selection and domain classification (9 domains: animals, architecture, art, etc.) to enhance aesthetics and instruction following.

2.2 Architecture: CrossDiT and NABLA

The core architecture is a unified Diffusion Transformer with Cross-Attention (CrossDiT) trained using the Flow Matching paradigm.

• CrossDiT: Unlike previous MMDiT-like architectures that required concatenation operations slowing down training, CrossDiT utilizes a cross-attention mechanism for better compatibility with sparse attention. It integrates text encoders (Qwen2.5-VL) and visual encoders (FLUX.1-dev VAE for images, HunyuanVideo VAE for video).
• NABLA (Neighborhood Adaptive Block-Level Attention): To handle high-resolution (up to 1024px) and long-duration (up to 10s) video generation, the authors introduce NABLA. This sparse attention mechanism dynamically constructs content-aware masks via block-wise dimensionality reduction and adaptive sparsification (CDF thresholding). It reduces the quadratic complexity of standard spatio-temporal attention, achieving a 2.7× speedup in training and inference while maintaining video quality.
• Token Reordering: NABLA employs fractal flattening to group spatial tokens, optimizing memory access patterns.

2.3 Training Pipeline

The training process is multi-staged and tailored for different model sizes (Image Lite, Video Lite, Video Pro):

1. Pre-training: Models are pre-trained on large-scale T2I, T2V, and I2V datasets across low, medium, and high resolutions. Video models are trained on a mix of tasks (T2I, T2V, I2V) with specific probability distributions.
2. Supervised Fine-Tuning (SFT): A "model souping" technique is employed. Data is clustered into thematic domains and subdomains. Independent full fine-tuning is performed on each subdomain, and the resulting checkpoints are aggregated via weighted averaging (equal or root-proportional weights) to create a robust final model. This prevents overfitting and improves generalization.
3. Distillation: For video models, a combined approach is used:
   • Classifier-Free Guidance (CFG) Distillation: Reduces sampling steps.
   • Trajectory Segmented Consistency Distillation (TSCD): Further reduces steps to 16.
   • Adversarial Post-training: A second-stage refinement using a discriminator (inspired by LADD) with a Hinge loss and stochastic perturbation (re-noising) to restore visual fidelity lost during aggressive distillation.
4. RL-based Post-training (Image Only): For image generation, a reward model (based on Qwen 2.5VL) is trained to compare generated images against real SFT images. The generator is then fine-tuned using Direct Reward Fine-Tuning (DRaFT-K), maximizing the reward model's preference while minimizing KL divergence from the SFT model.

2.4 Optimizations

• VAE Acceleration: Optimized HunyuanVideo VAE encoder with code refactoring and torch.compile, yielding a 2.5× speedup.
• Inference Optimization: Use of Flash/Sage Attention for standard resolutions and NABLA for HD/long videos. Caching diffusion steps via MagCache provides a 46% speedup.
• Memory Management: Implementation of HSDP (Hybrid Sharded Data Parallel), Sequence Parallel, and activation checkpointing with host offloading to reduce GPU memory consumption.

3. Key Contributions

1. Comprehensive Data Pipeline: A detailed description of data curation for T2I, T2V, I2V, and instruction-based editing, including specialized handling for Russian cultural content and high-quality SFT datasets.
2. Multi-Stage Training Framework: A unified pipeline encompassing pre-training, domain-specific SFT (via model souping), distillation, and RL-based post-training (for images) to enhance realism and alignment.
3. CrossDiT and NABLA Architecture: Introduction of a cross-attention transformer backbone and the NABLA sparse attention mechanism, which overcomes quadratic complexity for high-resolution video, enabling 2.7× faster training/inference.
4. Optimization Techniques: Implementation of VAE acceleration, text encoder quantization, and memory-efficient training strategies (HSDP, offloading) to enable high-fidelity generation on consumer and professional hardware.
5. Distillation Strategy: A novel combination of CFG distillation, TSCD, and adversarial post-training to reduce Function Evaluations (NFE) from 100 to 16 while preserving visual quality.
6. Open-Source Release: Full release of code, weights, and training checkpoints for all models (Image Lite, Video Lite, Video Pro, and their Flash variants) via Hugging Face and GitHub.

4. Results

• Human Evaluation: Extensive Side-by-Side (SBS) evaluations were conducted on the MovieGen benchmark (1,000+ prompts) with ~20 trained annotators.
  • Video Lite vs. Sora/Wan: Kandinsky 5.0 Video Lite demonstrated superior Visual Quality and Motion Dynamics compared to Sora and Wan models, though Wan models showed stronger Prompt Following (semantic alignment).
  • Video Lite vs. Kandinsky 4.1: Significant improvements in motion dynamics, object realism, and artifact suppression were observed.
  • Video Pro vs. Veo 3/Wan 2.2: Kandinsky 5.0 Video Pro achieved higher scores in Visual Quality and Motion Dynamics compared to Veo 3 and Wan 2.2 A14B, though Veo 3 variants outperformed in Prompt Following.
  • Image Lite: Outperformed FLUX.1 and Qwen-Image in Visual Quality while remaining competitive in Prompt Following.
• Performance Metrics:
  • Speed: Distilled "Flash" models reduce generation time significantly (e.g., Video Lite 10s generation drops from ~224s to ~61s on an H100).
  • Quality: Quantitative metrics (FVD, VBench, CLIP-score) and human evaluations confirm that the multi-stage training and NABLA mechanism maintain high quality despite computational reductions.

5. Significance and Claims

The paper positions Kandinsky 5.0 as a significant milestone in open-source generative AI, aiming to democratize access to high-quality foundation models for image and video.

• Accessibility: By releasing models with varying parameter counts (2B, 6B, 19B) and distilled "Flash" variants, the framework allows deployment across different hardware constraints.
• Technical Advancement: The integration of Flow Matching, CrossDiT, and NABLA addresses the scalability and efficiency bottlenecks inherent in video generation, offering a viable alternative to proprietary closed-source systems.
• Community Impact: The authors hope that the release of open-source code, training checkpoints, and a detailed technical report will substantially advance the development and accessibility of high-quality generative models for the research community.
• Ethical Stance: The model is released under the MIT license without built-in content filtering to foster innovation, placing the responsibility of ethical use and liability on the end-user. The authors acknowledge inherent biases in the training data and encourage specific prompting to mitigate stereotypes.

The report concludes that while Kandinsky 5.0 achieves SOTA performance in visual quality and motion consistency, challenges remain in text-visual alignment (due to encoder context limits) and modeling complex long-range physical interactions, which are identified as directions for future work.