PresentBench: A Fine-Grained Rubric-Based Benchmark for Slide Generation

This paper introduces PresentBench, a fine-grained, rubric-based benchmark comprising 238 instances and detailed binary checklists that enables more reliable, human-aligned evaluation of automated slide generation, revealing that NotebookLM currently outperforms other methods in this domain.

The Gist

Imagine you are a teacher preparing for a big class. You have a thick, complex textbook (the background material) and you need to turn it into a set of clear, colorful, and accurate PowerPoint slides (the slide deck) for your students.

In the past, if you asked an AI to do this, it was like asking a student to grade their own homework. The AI would just say, "Looks good!" or "Needs work," without telling you why. It was a coarse-grained judgment—a quick glance that missed the details.

This paper introduces PresentBench, a new, super-strict "grading system" for AI-generated slides. Think of it as moving from a teacher who just gives a smiley face or a frown, to a teacher who uses a 54-item checklist to grade every single slide.

Here is how it works, broken down with some fun analogies:

1. The Problem: The "Vibe Check" vs. The "Forensic Audit"

• Old Way (The Vibe Check): Previous benchmarks asked an AI judge, "Is this slide deck good?" The judge would look at the whole thing and say, "Yeah, the colors are nice, and the text is okay." It was like judging a meal just by looking at the plate. You might miss the fact that the soup is cold or the salt is missing.
• The New Way (The Forensic Audit): PresentBench is different. It doesn't just look at the "vibe." It acts like a detective. It has a specific list of questions for every single slide.
  • Did you include the chart from page 42?
  • Is the font size readable?
  • Did you accidentally change the number 500 to 50?
  • Is the background color consistent?

2. The Ingredients: Real-World Chaos

To test the AIs, the researchers didn't use made-up, easy examples. They gathered 238 real-world scenarios.

• The Sources: They grabbed materials from real university textbooks, financial reports from giant banks (like JPMorgan), and actual conference papers.
• The Challenge: Imagine asking an AI to summarize a 34-page financial report into 20 slides without making a single math error. That's the level of difficulty they set.

3. The Grading System: The "Atomic" Checklist

This is the paper's biggest innovation. Instead of one big grade, they break the evaluation down into 54.1 tiny questions (checklist items) per task.

Think of it like building a house:

• Old Grading: "Is the house nice?" (Answer: Yes/No).
• PresentBench Grading:
  • Is the roof made of the right shingles? (Yes/No)
  • Are there exactly 4 windows on the front? (Yes/No)
  • Did you use the blue paint requested, or did you use red? (Yes/No)
  • Is the foundation level? (Yes/No)

They split these questions into two categories:

1. The "Look and Feel" (Material-Independent): Does the slide look professional? Is the text too crowded? Is the font consistent? (You can judge this just by looking at the slides).
2. The "Truth and Accuracy" (Material-Dependent): Does the slide actually match the source material? If the source says "Profit is $10M," and the slide says "$100M," the AI gets a FAIL. This is the hardest part, where most AIs currently hallucinate (make things up).

4. The Results: Who Passed the Test?

The researchers tested famous AI tools (like NotebookLM, Gamma, Qwen, and others) using this strict checklist.

• The Winner: NotebookLM (by Google) came out on top. It was the only one that really understood how to stick to the source material and organize the facts correctly.
• The Reality Check: Even the winner only got about 62.5% on the test. This tells us that while AI is getting good at making slides, it's still struggling to be a reliable fact-checker and a perfect designer at the same time.
• The Bottleneck: The biggest failure point wasn't the text; it was the visual design. AIs often make slides that look messy, have text overlapping images, or use inconsistent colors. They are great at writing, but bad at "decorating."

5. Why Does This Matter?

Imagine you are a doctor using an AI to create a presentation for a medical conference. If the AI hallucinates a drug dosage or misses a crucial chart, it could be dangerous.

PresentBench proves that we can't just trust AI to "do a good job." We need a system that checks the facts against the source and the design against the rules.

In a nutshell:
This paper built a super-strict, 54-question pop quiz for AI slide makers. It showed us that while AI is getting better, it still makes too many mistakes in design and facts to be fully trusted yet. PresentBench gives us the ruler we need to measure exactly how far we have to go before AI can truly replace the human slide-maker.

Technical Summary

Here is a detailed technical summary of the paper "PresentBench: A Fine-Grained Rubric-Based Benchmark for Slide Generation."

1. Problem Statement

Creating high-quality slide decks is a cognitively demanding task requiring content selection, structural organization, visual design, and strict adherence to source materials. While generative AI models (e.g., NotebookLM, PPTAgent) have advanced in automated slide generation, evaluating their performance remains a critical bottleneck.

Existing evaluation frameworks suffer from four main limitations:

• Coarse-Grained Assessment: Current methods often rely on holistic "LLM-as-a-Judge" scoring (e.g., rating 1–5 on "content quality"), which lacks diagnostic precision.
• Instance-Agnostic Criteria: Many benchmarks apply the same generic questions to all slide decks, failing to account for specific background materials or unique task constraints.
• Lack of Grounding: Some benchmarks do not require models to ground their output in provided source documents, making it impossible to verify factual correctness or hallucination.
• Subjectivity: Aggregate scores obscure specific failure modes (e.g., layout errors vs. factual errors), hindering model iteration and error diagnosis.

2. Methodology: PresentBench

The authors propose PresentBench, a fine-grained, rubric-based benchmark designed to evaluate automated slide generation in real-world scenarios. The framework consists of three core components:

A. Dataset Construction

• Scale & Diversity: The dataset contains 238 expert-curated instances spanning five domains: Academia, Education, Economics, Talk, and Advertising.
• Source Materials: Each instance is paired with authentic, long-context background materials (e.g., top-tier conference papers, textbooks, financial reports, speeches), averaging 22.2k input tokens and 34 pages per task.
• Instructions: Each task includes highly constrained, instance-specific instructions defining slide count, mandatory sections, audience tone, and strict fidelity requirements (e.g., "no unsupported information").

B. Fine-Grained Evaluation Checklist

Instead of holistic scoring, PresentBench decomposes evaluation into ~54.1 binary (Yes/No) checklist items per instance. These items are divided into two tiers:

1. Material-Independent Checklist (Intrinsic Quality):
   • Presentation Fundamentals: Logical flow, conciseness, theme clarity, and safety.
   • Visual Design & Layout: Consistency, text-visual balance, legibility, and absence of clutter.
2. Material-Dependent Checklist (Grounding & Fidelity):
   • Content Completeness: Verification that all instruction-mandated sections and key points are present.
   • Content Correctness: Verification that the content matches the source material (e.g., correct numbers, definitions).
   • Content Fidelity: Page-by-page verification to ensure no hallucinations, contradictions, or fabricated data exist. This acts as a "precision" check, distinguishing it from "recall" (completeness).

C. Scoring Protocol

• Binary Verification: A Judge MLLM (Gemini-3-flash-preview) evaluates each checklist item independently, providing a binary verdict and localized evidence (slide index, specific text).
• Aggregation: The final score is the average of the five dimension scores (Fundamentals, Visual, Completeness, Correctness, Fidelity), normalized to a 0–100 scale.

3. Key Contributions

1. First Fine-Grained Rubric for Slides: Introduces a systematic, checklist-based evaluation framework that moves beyond holistic scoring to item-level verification.
2. Real-World Grounding: Unlike previous benchmarks that use open-ended prompts, PresentBench strictly enforces grounding in provided background materials, simulating real-world professional workflows.
3. Diagnostic Capability: By breaking down errors into specific dimensions (e.g., layout vs. factual accuracy), the benchmark provides actionable insights for model improvement.
4. Comprehensive Dataset: A curated collection of 238 diverse, high-difficulty tasks with expert-designed constraints and checklists.

4. Experimental Results

The authors evaluated 8 state-of-the-art slide generation systems (including NotebookLM, Manus, Gamma, Doubao, PPTAgent v2, and others) using PresentBench.

• Performance Gap: NotebookLM achieved the highest score (62.5), significantly outperforming other commercial systems (Manus 1.6: 57.8; Gamma: 49.2) and open-source agents (PPTAgent v2: 50.2).
• Systemic Challenges: Even the best system scored only ~62.5, indicating that grounded, end-to-end slide authoring is far from solved.
  • Visual Design was identified as the primary bottleneck, with scores generally in the 40s, suggesting models struggle to translate content into polished layouts.
  • Fidelity Issues: Models frequently hallucinated numbers or omitted specific data points (Content Fidelity scores were low, e.g., 45.1 for NotebookLM).
• Human Alignment: PresentBench achieved a Spearman correlation of 0.532 with human preferences, significantly outperforming existing frameworks like PPTEval (0.303) and MLLM-as-a-Judge ranking (0.258).
• Comparison with PPTEval: PresentBench yields stricter, more realistic scores. PPTEval tends to give overly optimistic scores due to its coarse-grained nature, whereas PresentBench exposes specific failures in grounding and layout.

5. Significance

• Standardization: PresentBench establishes a rigorous, verifiable standard for evaluating slide generation, shifting the field from subjective "vibes" to objective, item-level verification.
• Research Catalyst: The benchmark highlights specific weaknesses (particularly in visual design and factual grounding), guiding future research toward better layout engines and retrieval-augmented generation (RAG) pipelines.
• Real-World Applicability: By focusing on long-context, multi-source grounding, PresentBench addresses the actual needs of professional users who require accurate, source-backed presentations rather than generic summaries.
• Open-Source Gap: The results reveal a significant capability gap between closed-source commercial products and open-source agents, suggesting that proprietary end-to-end pipelines (planning + grounding + rendering) are currently superior to modular open-source approaches.

In conclusion, PresentBench reframes slide generation evaluation as a source-grounded, item-level verification task, providing a more reliable, diagnostic, and human-aligned metric for the next generation of AI office assistants.