💻 computer science

KD-CVG: A Knowledge-Driven Approach for Creative Video Generation

This paper introduces KD-CVG, a knowledge-driven framework that leverages a comprehensive Advertising Creative Knowledge Base and two novel modules—Semantic-Aware Retrieval and Multimodal Knowledge Reference—to overcome semantic alignment and motion adaptability challenges in creative video generation for advertising.

Original authors: Linkai Liu, Wei Feng, Xi Zhao, Shen Zhang, Xingye Chen, Zheng Zhang, Jingjing Lv, Junjie Shen, Ching Law, Yuchen Zhou, Zipeng Guo, Chao Gou

Published 2026-04-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Linkai Liu, Wei Feng, Xi Zhao, Shen Zhang, Xingye Chen, Zheng Zhang, Jingjing Lv, Junjie Shen, Ching Law, Yuchen Zhou, Zipeng Guo, Chao Gou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a marketing manager trying to create a 3-second video ad for a new toothpaste. You have a list of selling points: "Balances oral pH" and "Contains natural mint essence."

If you ask a standard AI video generator to make this, it might get confused. It might show a toothbrush floating in space (because it doesn't understand the connection between "mint" and "freshness") or make the water droplets fall upwards because it doesn't understand physics.

This paper introduces KD-CVG, a new system designed to fix exactly those problems. Think of KD-CVG as a super-smart creative director who has read a massive library of successful ads and knows exactly how to translate a boring text description into a lively, realistic video.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Lost in Translation" Gap

Current AI video makers are like actors who can recite lines but don't understand the feeling of the scene.

The Semantic Gap: If you say "fresh mint," the AI might just show a green leaf. It misses the feeling of coolness or the specific way water interacts with mint.
The Motion Gap: If you ask for water droplets, the AI might make them move like jelly or float in impossible directions because it hasn't seen enough real-world physics in its training.

2. The Solution: The "Advertising Creative Knowledge Base" (ACKB)

Before the AI can do its job, the researchers built a massive library of inspiration.

The Analogy: Imagine a chef who wants to cook a perfect dish. Instead of guessing, they have a cookbook filled with 10,000 photos of successful dishes, along with the exact recipes and the stories behind them.
What it is: The team collected 10,000 real e-commerce videos (like toothbrushes, face masks, etc.) and paired them with the text descriptions used to sell them. This is their "Cookbook" (ACKB).

3. The Two Magic Modules

KD-CVG uses two special tools to turn your text into a video, acting like a two-step creative process.

Step A: The "Smart Librarian" (Semantic-Aware Retrieval)

When you give the system a selling point (e.g., "Balance Oral pH"), it doesn't just guess. It acts like a highly intelligent librarian.

How it works: Instead of just matching keywords, it uses a "Graph Attention Network." Think of this as a web of connections. It knows that "mint" is connected to "coolness," which is connected to "water droplets."
The Result: It searches the library (ACKB) and finds the best reference videos that match the vibe of your text, not just the words. It's like the librarian handing you a reference photo of a mint leaf splashing in water, saying, "Use this as inspiration."

Step B: The "Master Choreographer" (Multimodal Knowledge Reference)

Once the librarian finds the reference, the "Choreographer" takes over to make the video.

How it works: It looks at the reference video and learns two things:
1. The Script: What is happening? (e.g., "Water pouring into a glass").
2. The Dance: How does it move? (e.g., "The water falls in a smooth arc, not a straight line").
The Magic: It takes your specific product (your toothbrush) and forces it to "dance" exactly like the reference video. It uses a technique called Motion Distillation, which is like teaching a robot to mimic a dancer's moves perfectly so the water droplets fall naturally and the toothbrush doesn't warp or twist.

4. Why It's Better (The Results)

The researchers tested this against other top AI video generators.

The Old Way: Other models often produced videos where objects looked weird, moved unnaturally, or didn't match the text (e.g., a toothbrush that looked like a snake).
The KD-CVG Way: Their videos were smoother, the movements were physically realistic (water falls down, not up), and the connection between the text and the video was much stronger.
Speed: It's also surprisingly fast, taking about 100 seconds to generate a video, which is much faster than some competitors that take hours.

The Bottom Line

KD-CVG is like giving an AI a mentor. Instead of letting the AI guess how to make an ad, it gives the AI a library of real examples and a step-by-step guide on how to copy the meaning and the movement of those examples. The result is advertising videos that actually look like they were made by a human creative team, with realistic physics and clear messaging.

1. Problem Statement

The paper addresses the gap between general Text-to-Video (T2V) models and the specific requirements of Creative Video Generation (CVG) for e-commerce advertising. While T2V has advanced, it struggles in commercial scenarios due to two primary challenges:

Ambiguous Semantic Alignment: E-commerce selling points (e.g., "Balance Oral pH") are often abstract and lack direct visual correlation. General T2V models fail to map these subtle text concepts to appropriate visual content (e.g., using "water droplets" or "mint leaves" to symbolize freshness), leading to irrelevant or generic video outputs.
Inadequate Motion Adaptability: Advertising videos require specific, realistic movements to highlight product features (e.g., a toothbrush bristle bending, water flowing). General T2V models lack e-commerce-specific motion priors, resulting in non-physical movements, distortions, and a lack of dynamic realism.

2. Methodology: KD-CVG Framework

The authors propose KD-CVG (Knowledge-Driven Creative Video Generation), a framework built upon a newly constructed Advertising Creative Knowledge Base (ACKB). The system consists of two core modules:

A. Advertising Creative Knowledge Base (ACKB)

Construction: Curated from 58,000 raw videos from a major e-commerce platform, filtered for quality, and processed to remove watermarks.
Content: Contains 10,000 text-video pairs covering 377 daily product types.
Enrichment: Uses Qwen2-VL to extract selling points and CogVLM2 to generate detailed script descriptions for semantic references.
Characteristics: Videos are short (~3s) with high resolution (240p–2120p), but the selling point texts are often vague (avg. 8.8 words).

B. Module 1: Semantic-Aware Retrieval (SAR)

To solve the semantic alignment issue, SAR retrieves relevant references from the ACKB based on the input selling point text.

Graph Attention Network (SC-GAT): Models the semantic space as a graph where nodes are selling points and edges represent semantic relationships. It uses CLIP embeddings and a Graph Attention Network to compute context-aware relevance scores.
Reinforcement Learning (RL): The retrieval policy is optimized using a policy gradient approach to maximize the CIDEr score (a metric for text similarity) between the generated script and ground truth. This ensures the model retrieves the most semantically relevant video scripts to guide generation.

C. Module 2: Multimodal Knowledge Reference (MKR)

MKR leverages the retrieved references to guide the T2V model, addressing motion adaptability through a two-stage process:

Progressive Script Generation:
- The system decomposes the top-ranked reference script into Subject ( $S_{subj}$ ), Scene ( $S_{scene}$ ), and Motion ( $S_{mot}$ ) components.
- It preserves the motion template ( $S_{mot}$ ) while adapting the subject and scene to the target product via prompt engineering.
- An LLM synthesizes a final script combining the preserved motion prior with the adapted semantic context.
Motion Prior Distillation:
- Differential Motion Encoding: Computes motion vectors between frames in the latent space to capture dynamic patterns.
- Motion Distillation Loss: Minimizes the discrepancy between the target motion gradients and the predicted motion, ensuring temporal coherence.
- MR-LoRA: Integrates Motion Reference Low-Rank Adaptation into the temporal attention blocks of the Sora-based (ST-DiT) architecture. This allows the model to learn specific motion patterns from references without full retraining.

3. Key Contributions

KD-CVG Framework: The first framework to automatically convert e-commerce selling points into advertising videos using a multimodal knowledge base, GAT-based semantic alignment, and motion priors.
ACKB Dataset: Release of a comprehensive, high-quality dataset of 10k e-commerce text-video pairs, addressing the lack of domain-specific data.
SAR Module: A novel retrieval mechanism using Graph Attention Networks and RL to resolve ambiguous semantic alignment between abstract selling points and visual content.
MKR Module: A dual-stage generation approach that extracts and distills motion priors from references, significantly improving the physical realism and adaptability of generated videos.

4. Experimental Results

The authors evaluated KD-CVG against state-of-the-art models (Show-1, VideoCrafter2, Open Sora) on 27 diverse selling-point texts.

Quantitative Performance:
- KD-CVG achieved the highest Min-Max Score (81.80%), outperforming baselines significantly.
- It excelled in Dynamic Degree (72.00%) and Motion Smoothness (98.65%), proving its ability to generate realistic, dynamic movements.
- Textual Alignment (30.68%) and Temporal Consistency (97.95%) were also superior.
Qualitative Improvements: Visual comparisons show KD-CVG produces videos with correct object interactions and fluid motion, whereas baselines often suffer from deformations (e.g., distorted toothbrushes) or static/irrelevant scenes.
User Study: In a blind test with 27 participants, KD-CVG scored highest in Text Alignment (3.74), Motion Regularity (3.81), and Highlight Degree (3.89).
Efficiency: Inference time is 102 seconds, which is significantly faster than Show-1 (3692s) and competitive with Open Sora (52s), while offering much higher quality.
Ablation Studies:
- Removing SAR or MKR components caused significant drops in CIDEr scores and user ratings.
- High-quality script generation (driven by SAR) directly correlated with better video performance.

5. Significance

This work bridges the gap between general generative AI and practical e-commerce applications. By introducing a knowledge-driven approach, KD-CVG overcomes the "black box" limitations of standard T2V models, ensuring that generated videos are not only visually coherent but also semantically faithful to marketing goals. The release of the ACKB dataset and the open-source code provides a critical foundation for future research in domain-specific video generation, moving the field from generic content creation to targeted, high-value advertising automation.