🤖 machine learning

PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks

This paper introduces PPT-Eval, a benchmark of 120 PowerPoint tasks designed to evaluate computer-use agents, featuring a novel rubric-based framework that captures partial progress and correlates strongly with human judgment to reveal that current frontier models still struggle with complex presentation editing.

Original authors: Apurva Gandhi, Vishwas Suryanarayanan, Raja Hasnain Anwar, Firoz Shaik, Shubhang Desai, Thong Q. Nguyen, Muhammad Taqi Raza, Vishal Chowdhary, Graham Neubig

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Apurva Gandhi, Vishwas Suryanarayanan, Raja Hasnain Anwar, Firoz Shaik, Shubhang Desai, Thong Q. Nguyen, Muhammad Taqi Raza, Vishal Chowdhary, Graham Neubig

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to use Microsoft PowerPoint. You want to see if the robot can actually look at a slide, understand a request like "add a blue circle here," and then click the right buttons to make it happen.

The paper "PPT-EVAL" is essentially a giant, tricky test designed to see how good these "computer-use" robots are at doing real-world presentation work.

Here is the breakdown of the paper using simple analogies:

1. The Problem: The Robot vs. The Real World

Most previous tests for robots were like giving them a video game controller with a limited set of buttons. The robot could only do what the game allowed (like "add text" or "change color" via a code command).

But in the real world, humans don't use code; they use a mouse, a keyboard, and a screen. They click menus, drag shapes, and use design tools that don't have simple code buttons.

The Paper's Claim: Existing tests didn't check if robots could actually use the mouse in PowerPoint. They only checked if robots could use a "cheat code" (API). PPT-EVAL is the first test that forces the robot to use the actual PowerPoint interface (the GUI) just like a human would.

2. The Test: The "PowerPoint Gym"

The researchers built a gym with 120 different exercises (tasks) spread across 12 different PowerPoint files.

The Files: They are like real-world decks you might find in a library—some are about medicine, some about history, some about accounting. They have charts, weird layouts, and animations.
The Difficulty: The exercises are graded like a video game:
- Easy: "Change the background color to blue." (Simple)
- Medium: "Add a new team member to the list and change their font." (A few steps)
- Hard: "Rearrange this complex diagram, add a new section, and make sure the animation plays correctly." (Requires thinking and many steps).

3. The Scoring: The "Rubric" (The Most Important Part)

This is the paper's biggest innovation. In the past, tests were like a Pass/Fail exam. If the robot got the answer 90% right but missed one tiny detail, it got a zero. That's unfair because the robot did try and did most of the work.

The researchers created a Rubric (a detailed grading sheet), which is like a talent show judge instead of a strict teacher.

Partial Credit: If the robot adds the circle but puts it in the wrong spot, the judge gives it points for adding the circle but deducts points for the placement.
Penalties: If the robot accidentally deletes a slide or adds a weird animation it wasn't asked for, the judge takes points away.
The "Human" Touch: The scoring system uses AI to write a natural language explanation, like: "You got the circle, but it's blocking the text. Good job on the color, but the position needs work."

The Result: This "Talent Show Judge" system was 77% aligned with how actual humans would grade the work. This means the test is fair and accurate.

4. The Results: The Robots Are Still Learning

The researchers put the world's smartest AI robots (like Claude-4.5 and OpenAI's models) through this test.

The Score: Even the best robots only got about 45% of the tasks "perfect."
The Average: On average, they got about 57% of the points when you count their partial progress.
The Comparison: A human expert, given the same test, would score around 80%.
The API vs. GUI Gap: Interestingly, when the robots were allowed to use "cheat codes" (APIs) instead of the mouse, they did much better (62% success). This proves that while the robots are smart, they are still clumsy with a mouse and keyboard compared to how they handle code.

Summary

Think of PPT-EVAL as a driving test for robots.

Old Tests: Asked the robot, "Can you calculate the speed of the car?" (Code/API).
PPT-EVAL: Puts the robot in the driver's seat, hands it the steering wheel, and says, "Drive to the store, park in the spot, and don't hit the curb." (GUI/Mouse).

The paper concludes that while our current robots are getting better, they still struggle with the fine motor skills and visual reasoning required to use PowerPoint like a human. There is still a long way to go before they can fully take over our presentation jobs.

Technical Summary: PPT-EVAL

Problem Statement

Creating and editing presentation slides is a ubiquitous professional activity, yet it remains an underexplored domain for evaluating computer-use agents. Existing benchmarks either lack depth in presentation software (e.g., OSWORLD, WindowsAgentArena) or are restricted to programmatic APIs (e.g., PPTC) and text-to-slide generation (e.g., SlidesBench). These limitations prevent agents from accessing the full feature set of modern presentation tools, such as design tools, advanced graphics, transitions, and animations, which are typically only available through a Graphical User Interface (GUI). Furthermore, current evaluation metrics often rely on binary success/failure criteria, which fail to capture the nuance of partial progress or penalize unnecessary changes in complex, open-ended tasks.

Methodology

The PPT-EVAL Benchmark

The authors introduce PPT-EVAL, a benchmark consisting of 120 tasks across 12 PowerPoint files sourced from the Internet Archive. The tasks cover diverse domains (Medicine, CS, Accounting, etc.) and are stratified into three difficulty levels:

Easy: Simple tasks requiring $\le$ 5 steps.
Medium: Compound tasks requiring 5–10 steps.
Hard: Complex tasks requiring $\ge$ 10 steps or advanced reasoning.

The benchmark is designed for GUI-based interaction using PowerPoint Online. Agents operate within a sandboxed Ubuntu environment using a Chromium browser, allowing them to interact with the full end-user feature surface (mouse, keyboard, scrolling) rather than being limited to API calls like python-pptx.

Evaluation Framework: Tree-Structured Rubrics

To address the challenge of evaluating multimodal, open-ended tasks with multiple valid solutions, the authors designed a rubric-based evaluation system. Key components include:

Tree-Structured Rubrics: Each task is decomposed into a hierarchy of criteria. Nodes are classified as critical (core requirements) or non-critical (stylistic choices, penalizing extraneous changes).
Partial Credit Scoring: Unlike binary metrics, the system awards partial credit for intermediate progress. The aggregation strategy uses a modified formula:
$\bar{s}_{parent} = \max\left(0, \bar{s}_{crit} - \lambda(1 - \bar{s}_{non})\right)$
where $\lambda = 0.3$ . This ensures critical failures significantly impact the score while non-critical errors introduce penalties without zeroing out meaningful progress.
Multi-Modal Verification: Leaf nodes utilize:
- Programmatic checks (python-pptx, custom PPTDiff class) for structural changes, animations, and transitions.
- LLM/VLM calls for semantic correctness and visual verification (e.g., color matching, layout alignment) where exact pixel matches are insufficient.
Natural Language Feedback: The system recursively aggregates explanations from child nodes to generate human-readable feedback for the final score.

Experimental Setup

The authors evaluated a range of agents on PPT-EVAL:

Closed Frontier Models: OpenAI's COMPUTER-USE-PREVIEW, Anthropic's CLAUDE-4-SONNET, and CLAUDE-4.5-OPUS.
Open-Weights Models: OPENCUA (7B, 32B) and QWEN3-VL (8B, 32B).
Baselines: A human baseline (5 participants) and an API-based baseline using CLAUDE-4.5-OPUS with the pptx skill.

Key Results

Agent Performance

Significant Gap: Frontier agents struggle with PowerPoint tasks. The strongest model, CLAUDE-4.5-OPUS, achieved a 45% success rate (perfect score) and an average partial score of 0.57.
Human Baseline: Human participants achieved an 80% success rate and an average partial score of 0.90, highlighting substantial room for improvement in agent capabilities.
API vs. GUI: The API-based baseline (CLAUDE-4.5-OPUS via pptx skill) significantly outperformed the best GUI agent (62% success rate vs. 45%), demonstrating that current GUI agents lag behind CLI/API-trained models in maturity, despite having access to richer features.
Task Difficulty: Performance degrades sharply with difficulty. CLAUDE-4.5-OPUS dropped from 56% success on Easy tasks to 35% on Hard tasks.
Failure Modes: Analysis revealed agents struggle with advanced structures (tables, animations, non-standard layouts) and often fail to maintain consistency across slides or avoid unintended modifications.

Rubric Reliability

The rubric-based evaluation showed strong alignment with human judgment:

Correlation: Kendall's $\tau_b$ of 0.77 and Spearman's $\rho$ of 0.84 between rubric scores and human-expected categories.
Granularity: The system effectively distinguished between "No Progress," "Some Progress," "Significant Progress," and "Perfect Completion," though some disagreements arose from subjective aesthetic judgments or VLM hallucinations.

Significance and Claims

The paper claims that PPT-EVAL fills a critical gap in the evaluation of computer-use agents by providing a realistic, GUI-level benchmark for presentation editing. Its primary contributions are:

A Realistic Testbed: It moves beyond API-bound tasks to evaluate agents on the full feature set of PowerPoint, including transitions, animations, and design tools.
Nuanced Evaluation: By introducing tree-structured rubrics with partial credit and natural language feedback, it overcomes the limitations of binary success metrics, allowing for a more granular assessment of agent capabilities.
Benchmarking Reality: The results demonstrate that while frontier models can make meaningful progress, they still significantly lag behind human performance and API-based agents in complex, iterative GUI workflows.

The authors conclude that there is "significant headroom for progress" in realistic GUI-based computer-use capabilities and that their benchmark provides a robust framework for measuring this progress beyond simple pass/fail metrics.