EduVQA: Benchmarking AI-Generated Video Quality Assessment for Education

This paper introduces EduAIGV-1k, the first benchmark dataset for evaluating AI-generated educational videos on math concepts, and proposes EduVQA, a novel assessment framework featuring a Structured 2D Mixture-of-Experts module that leverages fine-grained perceptual and prompt-alignment annotations to outperform existing video quality baselines.

The Gist

Imagine you've just built a magical video machine that can turn any sentence you write into a moving picture. It's amazing! You type "a cat eating pizza," and poof, there's a cat eating pizza.

But now, imagine you want to use this machine to teach a 5-year-old how to count or understand shapes. Suddenly, the stakes get higher. If the machine draws "three blue blocks" but accidentally makes them red, or if the blocks melt into a puddle halfway through the video, the lesson fails. The child learns the wrong thing.

This is the problem the paper EduVQA is trying to solve. Here is the story of how they fixed it, explained simply:

1. The Problem: The "Magic" Machine is a Bad Teacher

Current AI video generators are great at making cool, artistic videos for movies or TikTok. But they are terrible at being precise teachers. They often:

• Miscount: They might draw "five apples" but only show four.
• Get Confused: They might make a triangle rotate the wrong way.
• Glitch: The video might flicker or the objects might jump around weirdly.

Existing tools for judging video quality are like art critics. They ask, "Is this pretty? Is the lighting good?" They don't ask, "Did the AI actually follow the math instructions?"

2. The Solution Part 1: Building a "Report Card" (The Dataset)

To fix this, the researchers built a massive test bank called EduAIGV-1k. Think of this as a giant library of 1,130 math videos.

• The Prompts: They wrote 113 specific math instructions (like "Show a square turning into a circle" or "Count three red balls").
• The Generators: They fed these instructions into 10 different AI video machines (the "students").
• The Grading: Instead of just giving a video a score of "5 out of 10," human experts graded them on a detailed report card with two main sections:
  1. The "Look" (Perceptual Quality): Is the video smooth? Do the edges look sharp, or is it blurry? (Like checking if a drawing is neat).
  2. The "Meaning" (Prompt Alignment): Did the AI actually do what you asked? If you said "four blue cars," did it show exactly four, and were they blue? (Like checking if the student answered the math problem correctly).

3. The Solution Part 2: The "Super-Grader" (The EduVQA Model)

Now that they had a library of graded videos, they needed a robot that could grade new videos automatically. They built EduVQA.

Think of EduVQA as a super-smart teaching assistant with a special brain structure called S2D-MoE (Structured 2D Mixture-of-Experts). Here is a simple analogy for how it works:

• The Old Way (Single Grader): Imagine one teacher trying to grade a student's essay. They have to check grammar, spelling, plot, and math all at once. They might get tired and miss small details.
• The EduVQA Way (The Expert Panel): Imagine a team of specialists working together:
  • Expert A only looks at the spatial stuff (is the drawing clear?).
  • Expert B only looks at the time stuff (is the movement smooth?).
  • Expert C only checks the words (did the AI count correctly?).
  • Expert D checks the whole story (does it make sense?).

The magic of EduVQA is that these experts talk to each other. They share their findings so the final grade isn't just a sum of parts, but a smart, connected judgment. If the "Word Expert" sees a mistake, it tells the "Overall Expert," "Hey, the whole video is wrong because the numbers are wrong!"

4. Why This Matters

Before this paper, if you wanted to make an AI video for a school, you'd have to guess if it was good. Now, with EduVQA:

• Teachers can trust that the AI videos they use will actually teach the right concepts.
• Developers have a clear target to aim for. They can say, "My AI needs to get a higher score on 'counting accuracy' before I release it."
• Kids get better learning tools where the math is actually correct, not just pretty.

The Bottom Line

The researchers built a giant math-video test and a smart grading robot to ensure that when AI makes videos for kids, it doesn't just look cool—it actually teaches the lesson correctly. They are turning the "magic" of AI into a reliable tool for education.

Technical Summary

1. Problem Statement

While Text-to-Video (T2V) models have achieved significant success in generating photorealistic content for entertainment, their application in education—specifically for teaching foundational math concepts to young learners—remains underexplored and critically under-evaluated. Existing benchmarks and quality assessment (VQA) metrics face three main limitations in this context:

• Lack of Domain Specificity: Current benchmarks focus on general realism, coherence, or entertainment, failing to evaluate the pedagogical accuracy required for educational content (e.g., correctly counting objects or visualizing geometric transformations).
• Coarse-Grained Evaluation: Existing methods often provide a single global quality score, lacking the fine-grained feedback necessary to diagnose specific failures in spatial fidelity, temporal stability, or semantic alignment.
• Semantic Grounding Gap: Standard metrics (like CLIP similarity) often overlook the precise ordering of concepts and the accurate grounding of specific keywords (e.g., "four" vs. "five") within the video, which is critical for math education.

2. Methodology

The paper introduces a comprehensive framework consisting of a new dataset (EduAIGV-1k) and a novel assessment model (EduVQA).

A. Dataset: EduAIGV-1k

• Scale & Composition: Contains 1,130 short videos generated by 10 state-of-the-art T2V models (including CogVideo, Kling, Gen-3, etc.) using 113 expert-curated prompts.
• Content Domains: Prompts are derived from educational standards (TIMSS 2023, Common Core) covering four core math areas: Numbers (43 prompts), Geometry (40), Measurement (20), and Probability (10).
• Fine-Grained Annotation Scheme: Unlike global scoring, each video is annotated along two complementary axes with 5-point Likert scales:
  1. Perceptual Quality: Disentangled into Spatial Quality (texture, sharpness), Temporal Quality (motion smoothness, flickering), and Overall Perceptual Quality.
  2. Prompt Alignment: Labeled at Word-Level (accuracy of specific entities/concepts) and Sentence-Level (overall semantic consistency).
• Data Collection: 19 trained annotators performed subjective evaluations following ITU-R BT.500 standards, with rigorous outlier removal and consistency checks.

B. Model: EduVQA

EduVQA is a dual-path framework designed to jointly predict perceptual and alignment qualities.

• Feature Extraction:
  • Uses Video Swin Transformer for video quality-sensitive features ($F_{VST}$).
  • Uses BLIP as a multimodal encoder to fuse visual and textual features ($F_{BLIP}$).
• Cross-Modal Interaction: Employs bidirectional cross-attention to generate perceptual-aware features ($F_p$) and alignment features ($F_a$).
• Core Innovation: Structured 2D Mixture-of-Experts (S2D-MoE):
  • Motivation: Standard 1D MoE layers treat tasks independently. EduVQA recognizes that overall quality is hierarchically dependent on sub-dimensions (e.g., overall quality depends on spatial and temporal quality).
  • Mechanism:
    • Shared Expert Pool: Enforces representation coherence by requiring overall and sub-dimension predictors to rely on a common set of experts.
    • Adaptive 2D Gating Matrix: Models interactions between sub-tasks dynamically. For the perceptual path, it creates a 2D matrix representing interactions between spatial and temporal experts. For the alignment path, it models token-level vs. sentence-level dependencies.
  • Benefit: This structure ensures the overall quality inference is guided by collective sub-dimension knowledge, enhancing interpretability and generalization.
• Optimization: Trained using a multi-task objective minimizing Pearson Linear Correlation Coefficient (PLCC) loss across all five dimensions (Spatial, Temporal, Overall, Word-Level, Sentence-Level).

3. Key Contributions

1. First Educational AIGV Benchmark: Introduction of EduAIGV-1k, the first dataset dedicated to assessing AI-generated videos for early math education, featuring 1,130 videos and rich, multi-dimensional annotations.
2. Fine-Grained Annotation Scheme: A novel labeling protocol that separates perceptual quality (spatial/temporal) from semantic alignment (word/sentence level), enabling precise diagnosis of generation failures.
3. EduVQA Framework: A unified VQA model featuring the S2D-MoE module, which explicitly models hierarchical dependencies between overall and sub-dimension qualities, outperforming existing baselines.
4. Generalization: Demonstrated that the model trained on EduAIGV-1k generalizes effectively to unseen AIGVQA datasets (LGVQ, EvalCrafter).

4. Experimental Results

• Performance on EduAIGV-1k:
  • Perceptual Quality: EduVQA achieved SRCC 0.869 and PLCC 0.879, outperforming the strongest baseline (IP-IQA) by +2.00% (SRCC) and +1.85% (PLCC). It significantly improved upon video-specific baselines like BVQA.
  • Prompt Alignment: Achieved SRCC 0.757 and PLCC 0.768, surpassing the specialized T2VQA baseline by +3.56% (SRCC) and +4.77% (PLCC).
• Cross-Dataset Generalization: When tested on unseen datasets (LGVQ and EvalCrafter) without fine-tuning, EduVQA maintained state-of-the-art performance, demonstrating robustness to domain shifts.
• Qualitative Analysis:
  • gMAD Competition: EduVQA correctly identified video pairs where baselines (like T2VQA and IP-IQA) failed to distinguish subtle temporal artifacts or semantic mismatches (e.g., wrong object counts).
  • Interpretability: The model successfully highlighted specific words in prompts that were poorly grounded in the video, providing actionable feedback for educators and model developers.
• Ablation Studies: Confirmed that removing the cross-modal fusion, sub-dimension branches, or replacing S2D-MoE with standard 1D MoE led to significant performance drops, validating the necessity of the proposed architecture.

5. Significance

• Bridging AI and Education: This work provides the necessary infrastructure to evaluate and improve AI tools for educational content creation, ensuring that generated videos are not just visually appealing but pedagogically accurate.
• Advancing VQA Methodology: By introducing the S2D-MoE architecture, the paper offers a new paradigm for multi-dimensional quality assessment that moves beyond single-score regression to structured, interpretable prediction.
• Future Research Foundation: The release of the dataset and code establishes a solid baseline for future research in "quality-aware" generation, encouraging the development of T2V models specifically optimized for educational fidelity and semantic grounding.