A Good Talk Does not Look Like a Summary, It Teaches You! Measuring Takeaways from Paper-to-Video Talks

Problem
Automatically generated videos from scientific papers are increasingly used for education and research dissemination. However, existing evaluation metrics primarily focus on visual quality (e.g., realism, fluency) or surface-level content presence (e.g., whether key points appear in the video). These metrics fail to assess whether a video actually helps viewers understand the paper's core ideas. Current benchmarks often prioritize short-form, visually appealing clips, which are poorly suited for instructional content that requires logical structure, prerequisite grounding, and pedagogical coherence. Consequently, there is a gap between benchmark performance (what is optimized) and instructional utility (what should be optimized): systems may produce fluent summaries that mention correct topics but fail to explain prerequisite concepts, clarify causal relationships, or support reasoning about why a method works.

Methodology
The authors introduce EffectivePresentationScorer, a framework designed to evaluate the instructional quality of scientific presentation videos by measuring "instructional utility." This concept is grounded in educational research (McConnell et al., 2017) and theories of multimedia learning and cognitive load (Mayer, 2002; Sweller, 1994, 2024).

1. Dataset Construction (EffectivePresentation-EvalBench):

   • The authors constructed a benchmark using 20 research papers from the VISTA dataset.
   • For each paper, they generated seven video variants: one human-authored conference presentation and six automatically generated variants.
   • The six automated variants were created through controlled perturbations in content (e.g., omitting prerequisites, reordering concepts) and delivery (e.g., poor pacing, audio-visual misalignment, dense slides).
   • Human annotators, screened for prerequisite knowledge, watched these videos and answered paper-grounded questions ranging from factual recall to complex reasoning (based on Bloom's Taxonomy).
   • Annotators also reported perceived difficulty and viewing effort. A utility score was computed by combining answer correctness with penalties for high difficulty and effort, creating a learner-centered metric.

2. EffectivePresentationScorer Framework:

   • The framework decomposes evaluation into content-level and delivery-level diagnostics using a multi-agent system.
   • Claim Decomposition: Given a question and the source paper, the system retrieves relevant text to decompose the question into a dependency-structured set of explanatory claims.
   • Claim Importance: An agent assigns importance scores to claims based on their role in the paper's explanatory structure.
   • Diagnostics:
     • Presence: Checks if required claims are explicitly stated in the video.
     • Faithfulness: Verifies if expressed claims align with the source paper (no hallucinations or overstatements).
     • Coherence: Evaluates if claims are introduced in an order that respects conceptual prerequisites (no prerequisite violations).
     • Delivery Quality: Measures explanatory depth, time allocation, and audio-visual alignment, weighted by claim importance.
     • Engagement: Assesses pacing and prosody.
   • Meta-Evaluator: Aggregates these signals into a final utility score and generates a diagnostic rationale explaining why a video succeeded or failed (e.g., missing core causal claims vs. poor pacing).

Key Contributions

• EffectivePresentationScorer: A novel, interpretable evaluation framework that moves beyond surface-level metrics to assess whether a video supports understanding through clear explanation, prerequisite introduction, and logical connection of technical details.
• EffectivePresentation-EvalBench: A new dataset of 140 instructional videos (20 papers × 7 variants) with human-annotated utility scores and diagnostic labels for instructional failures.
• Diagnostic Feedback: The system provides actionable feedback (e.g., "missing background," "poor coherence") that aligns with human-identified improvement needs, rather than just a scalar score.

Results

• Correlation with Human Judgment: EffectivePresentationScorer demonstrates significantly stronger correlation with human-measured instructional utility than existing baselines (e.g., VideoScore, EvalCrafter, PresentQuiz). This is particularly true for non-recall (reasoning) questions, where baselines degrade substantially.
• Diagnostic Alignment: The framework's diagnostic feedback (e.g., identifying missing prerequisites or ordering violations) aligns strongly with human feedback (high precision, recall, and F1 scores), confirming it captures the specific instructional failure modes humans notice.
• System Comparison: When applied to state-of-the-art paper-to-video generation systems (Paper2Video, PPTAgent, Doc2PPT), the framework reveals that while these systems achieve strong recall utility (covering content), they exhibit substantial degradation in reasoning utility. They frequently fail to motivate prerequisites, establish causal links, or actively use figures to support reasoning, resulting in lower instructional utility compared to human-authored videos.

Significance
The paper argues that current evaluation paradigms for scientific videos are insufficient because they optimize for visual fluency and content presence rather than pedagogical effectiveness. By introducing a framework that measures instructional utility through learner-oriented dimensions, the authors provide a tool to:

1. Diagnose specific instructional failures in generated videos (e.g., missing causal links, disordered prerequisites).
2. Guide the development of future generation systems to prioritize explanatory depth and logical structure over mere summarization.
3. Bridge the gap between automated benchmark performance and the actual educational value of scientific communication.

The work is scoped as an evaluation and diagnostic framework; the authors note that integrating these diagnostics into a closed-loop generation pipeline to improve video quality is a direction for future work.