🤖 AI

VideoAgent: Personalized Synthesis of Scientific Videos

The paper introduces VideoAgent, a modular framework that redefines scientific video synthesis as an intent-driven planning problem to create personalized, audience-adaptive videos by dynamically interleaving static slides with animations, alongside the proposed SciVidEval benchmark for assessing multimodal quality and pedagogical utility.

Original authors: Xiao Liang, Bangxin Li, Zixuan Chen, Hanyue Zheng, Zhi Ma, Di Wang, Cong Tian, Quan Wang

Published 2026-04-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiao Liang, Bangxin Li, Zixuan Chen, Hanyue Zheng, Zhi Ma, Di Wang, Cong Tian, Quan Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very dense, complicated recipe for a 10-course gourmet meal written in a language you barely understand. It's full of technical jargon, precise measurements, and complex chemical reactions. If you just read it, you might get a headache and give up.

Now, imagine if you could instantly turn that recipe into a fun, engaging cooking show where a friendly host explains the steps, shows you the ingredients sizzling in the pan, and adapts the explanation based on whether you are a professional chef or a curious 10-year-old.

That is essentially what VideoAgent does, but for scientific research papers.

Here is the breakdown of how it works, using simple analogies:

1. The Problem: The "Wall of Text"

Scientific papers are like massive, unorganized libraries. They are packed with data, charts, and complex logic.

The Old Way: Previous tools tried to turn these papers into static PowerPoint slides or posters. It's like taking that recipe and just printing it on a piece of paper. It's useful, but it's boring, rigid, and doesn't explain why the steps matter.
The Goal: We need a way to turn that "wall of text" into a dynamic video story that grabs your attention and actually teaches you the material.

2. The Solution: VideoAgent (The "Smart Director")

VideoAgent is like a super-smart film director who reads the paper and then directs a movie about it. It doesn't just copy-paste; it plans the story.

It works in four main stages:

Stage 1: The Librarian (Document Parser)

First, the system acts like a super-efficient librarian. It takes the messy PDF paper and organizes it.

It separates the text (the story) from the images and charts (the props).
It cleans up the formatting so the computer knows exactly what is a graph, what is a math equation, and what is a paragraph.

Stage 2: The Scriptwriter (Requirement Analyzer)

This is where the "personalization" happens. You tell the system who the audience is.

Scenario A: "Explain this to a 12-year-old." -> The system decides to use simple words, analogies, and bright colors.
Scenario B: "Explain this to a PhD researcher." -> The system keeps the technical jargon and focuses on the deep logic.
It creates a script (a plan) that decides which parts of the paper need a static slide and which parts need a moving animation.

Stage 3: The Director & Editor (Personalized Planner)

Now, the system builds the video. This is the magic part.

The Hybrid Approach: It doesn't just use one type of visual. It mixes static slides (like a standard presentation) with dynamic animations (like a cartoon explaining a process).
The "Manim" Magic: If the paper talks about a complex algorithm or a robot moving, VideoAgent writes code to generate a smooth animation of that process, rather than just showing a static picture of it.
Self-Correction: If the computer tries to draw a chart and it looks messy, the system notices, fixes the code, and tries again automatically. It's like a director yelling "Cut! Do that again, but better!" until it's perfect.

Stage 4: The Sound Engineer (Multimodal Synthesizer)

Finally, it puts it all together.

It generates a voiceover (narration) that matches the speed of the visuals.
If the narrator speaks fast, the video speeds up. If they pause for effect, the video pauses.
It stitches the slides, the animations, the voice, and the subtitles into one seamless video file.

3. How Do We Know It Works? (The "Test Kitchen")

The authors created a special testing ground called SciVidEval.

The Quiz: They didn't just ask, "Is the video pretty?" They asked, "Did you learn anything?"
They showed the videos to real humans (graduate students) and asked them questions about the paper.
The Result: People who watched the VideoAgent videos understood the complex science almost as well as if they had watched the original author's video, and much better than videos made by other automated tools.

The Big Picture Takeaway

Think of VideoAgent as a translator that doesn't just translate words from one language to another, but translates a boring textbook into an exciting movie.

Before: You had to struggle through a dense paper to understand a new scientific breakthrough.
Now: You can watch a personalized video that adapts to your level of knowledge, uses animations to show how things work, and helps you actually learn the material in minutes.

It turns the "hard stuff" of science into a story anyone can enjoy.

1. Problem Statement

Scientific research papers are often too technically complex for general audiences, limiting their dissemination. While existing automated tools can generate static media (posters, slides), they suffer from three main limitations:

Linearity and Rigidity: They follow the linear structure of the source paper, failing to adapt to specific audience needs or narrative goals.
Static Nature: They lack the temporal dynamics required to explain complex processes, relying solely on static layouts.
Synchronization Issues: They struggle to jointly synchronize disparate multimodal assets (static slides, dynamic animations, and narration) with semantic density.

The core challenge is to shift from template-based static generation to audience-adaptive, non-linear video synthesis that orchestrates heterogeneous visual forms and ensures precise audio-visual synchronization.

2. Methodology: VideoAgent Framework

VideoAgent is a modular, multi-stage framework that redefines scientific video synthesis as an intent-driven planning problem. It decouples content understanding from multimodal synthesis through four key modules:

A. Document Parser

Input: Raw PDF of a scientific paper.
Process: Uses tools like Docling and Marker to parse complex layouts (multi-column text, tables, figures).
Output:
1. Refined Textual Representation ( $D$ ): Structured Markdown/JSON representing the paper's hierarchy.
2. Visual Asset Library ( $M$ ): Indexed high-resolution crops of equations, figures, and tables with metadata.

B. Requirement Analyzer

Function: Translates ambiguous user intent (e.g., "explain to a middle schooler" or "focus on methodology") into a structured JSON configuration ( $R$ ).
Dimensions:
- Technical Specs ( $R_{spec}$ ): Resolution, frame rate, typography.
- Functional Requirements ( $R_{func}$ ): Enables/disables features like dynamic animation, voiceover, or visual abstracts.
- Content Requirements ( $R_{cont}$ ): Defines the narrative paradigm (e.g., "Core 5 Questions," "Dialectical Debate," "Audience-Adaptive").

C. Personalized Planner

Core Logic: An LLM-driven agent that constructs a non-linear narrative outline ( $O$ ) and allocates assets ( $M_{alloc}$ ).
Dual-Path Synthesis:
1. Static Slide Synthesis: Generates Python code (python-pptx) for slides. If a "visual abstract" is requested, it generates thematic images. It handles layout logic (e.g., placing figures next to text based on aspect ratio).
2. Dynamic Animation Synthesis: For procedural content (algorithms, workflows), it generates Manim code. It refines the narrative node into a storyboard with relative layout hints (e.g., "place B below A") rather than absolute coordinates, ensuring scientific grounding.
Self-Correction Loop: Includes a verification step using a Vision-Language Model (VLM) to detect layout anomalies (overlapping text, out-of-bounds elements) and regenerate code if necessary.

D. Multimodal Synthesizer

Synchronization: Aligns the generated narration (Text-to-Speech), static slides, and animation clips.
Process:
- Converts scripts to audio ( $A$ ).
- Adjusts the playback speed of visual assets to match the audio duration.
- Uses MoviePy to composite the final video, ensuring temporal alignment between narration and visual emphasis.

3. Benchmark: SciVidEval

To evaluate the system, the authors introduced SciVidEval, a comprehensive benchmark comprising:

Dataset: 50 high-quality paper-video pairs from top AI conferences (CVPR, ICML, NeurIPS, etc.), covering Computer Vision, NLP, and ML.
Extraction Toolchain: An automated pipeline that segments videos into time-aligned semantic units (OCR for slides, ASR for audio) for comparison.
Evaluation Metrics:
1. Content Quality:
  - Narration: Perplexity (PPL), Rouge-L, and LLM-as-Judge for fidelity.
  - Visual: Layout fluency (PPL), Asset Match Accuracy, and Semantic Fidelity.
  - Synchronization: CLIP Score (embedding similarity) and Logic Consistency.
2. Knowledge Transfer:
  - Automated: VLM-as-Judge answering MCQs based on the video.
  - Human: Blind study with graduate students answering MCQs to measure actual learning outcomes.

4. Key Results

Experiments compared VideoAgent against commercial services (NotebookLM, Pictory), open-source baselines (AutoSlides, PresentAgent), and Oracle baselines (Author videos).

Overall Performance: VideoAgent (using Gemini-2.5 Pro) achieved the highest Rouge-L (0.14) and LLM-as-Judge scores for narration, indicating superior content consistency compared to baselines.
Visual Quality: It achieved a Visual Quality score of 8.03 (VLM-judged) and 79.24% Asset Match Accuracy, significantly outperforming commercial tools that often hallucinate or omit technical assets.
Knowledge Transfer:
- Automated: Achieved 99.50% accuracy on Video-Quiz (comparable to the Oracle Author baseline of 97.00%).
- Human: Graduate students achieved 87.5% accuracy, approaching the 90.0% achieved by watching original author videos.
Personalization: Different personas (e.g., "Middle School Teacher" vs. "Tech Consultant") successfully altered the narrative structure and complexity, with audience satisfaction scores varying appropriately (e.g., 8.2 for students vs. 8.7 for researchers).
Animation Quality: The Manim generation module achieved a 90% Pass@1 rate (vs. 82% for Manimator) and higher logical flow scores, thanks to the self-correction mechanism.
Efficiency: The average cost per video is approximately $0.35, with a total token consumption of ~32k input / 30k output, making it scalable.

5. Significance and Contributions

Paradigm Shift: Moves scientific communication from static, linear templates to dynamic, intent-driven, non-linear video synthesis.
Modular Architecture: Proposes a decoupled framework that separates content understanding, planning, and synthesis, allowing for flexible adaptation to different audience needs.
Robust Evaluation: Introduces SciVidEval, the first benchmark to rigorously evaluate scientific videos through both automated multimodal metrics and human knowledge transfer studies.
Practical Impact: Demonstrates that AI can effectively bridge the gap between complex technical research and broader audiences, enhancing the accessibility and impact of scientific discoveries without sacrificing technical fidelity.

In conclusion, VideoAgent represents a significant advancement in multimodal generation, proving that personalized, high-fidelity scientific videos can be automatically synthesized with a level of quality and educational utility comparable to human experts.