Technical Summary: AgenticVBench

Problem Statement

Current multimodal benchmarks primarily evaluate AI agents through single-shot question answering (QA) settings, focusing on perception and reasoning rather than complex, multi-step workflows. Existing agentic benchmarks largely target software engineering or general desktop control, failing to address the specific composite capabilities required for real-world video production. These capabilities include long-horizon planning, composite understanding across text, image, audio, and video, and the ability to manipulate media files to satisfy both technical and editorial constraints. This gap hinders the systematic design of models and harnesses (agent frameworks) capable of performing professional post-production tasks.

Methodology

To address this, the authors introduce AgenticVBench, a benchmark comprising 100 agentic tasks across four distinct task families spanning the post-production workflow. The benchmark construction involved 20 industry experts with an average of six years of professional experience from traditional film studios, AI film studios, video agent companies, and independent creator backgrounds.

Task Families

The four task families are designed to cover the full multimodal stack and span human completion times from 0.5 hours to one week:

1. Assembly: Selecting clips that match a storyboard. Agents receive a storyboard with 3–6 slots defined by shot descriptions and cinematic variables (shot size, angle, lens, movement) and must select the correct "golden" clip from a set of candidates containing AI-generated distractors.
2. Repair: Finding and fixing technical defects. Agents receive a broken video with an injected defect (audio, visual, or timeline) and a prompt describing the issue without explicit timestamps. They must localize the defect, apply a fix, and generate a report.
3. Sequencing: Restoring the order of shuffled shots. Given a story overview and shuffled clips, agents must recover the correct narrative order.
4. Repurpose: Converting long source videos into brief-driven short deliverables (e.g., recaps, trailers, social cuts). This is an open-ended task where agents must condense 4-minute to 3-hour videos based on a creative brief, with no single correct answer.

Evaluation Framework

Evaluation combines programmatic verifiers and expert rubrics:

• Programmatic Verifiers: Used for Assembly, Repair, Sequencing, and the "Format" pillar of Repurpose. These provide deterministic checks (e.g., clip selection accuracy, defect localization precision, narrative ordering metrics like Normalized Distance and Longest Increasing Subsequence).
• Expert Rubrics: Used for the Visual, Narrative, and Sound pillars of Repurpose. These consist of binary (Yes/No) items authored by experts to ensure reproducibility, covering aspects like scene preservation, story arcs, and audio synchronization.

Experimental Setup

The authors evaluated 7 frontier vision-language models (VLMs) including Claude Opus 4.7, GPT-5.5, and Qwen3-VL-235B. These were tested across 20 combinations of models and harnesses:

• Vendor-native harnesses: Claude Code CLI, Codex CLI, Gemini CLI.
• Open-source harnesses: OpenClaw, OpenCode.
  Each task was run 3 times with fixed tool schemas and random seeds. A human baseline was established using trained editors working under the same constraints.

Key Results

The evaluation reveals a significant performance gap between current AI agents and human experts:

• Overall Performance: The best-performing agent stack achieved a score of only 31%, substantially below human expert performance (which ranged from 0.81 to 0.95 depending on the task family). The gap between the best agent and human experts ranged from 43 to 65 percentage points.
• Task-Specific Failures:
  • Repurpose: The largest gap (65 pp) occurred here, primarily due to long-context information loss. Agents often exhausted their rollout budget on transcription or thumbnailing without reaching the assembly step.
  • Repair: The dominant failure mode was temporal reasoning, where agents produced valid video files but localized the defect window incorrectly (median error of 15–100+ seconds).
  • Assembly: Agents relied heavily on prose descriptions rather than cinematic variables (shot size, angle), whereas human editors prioritize these variables.
• Impact of Harness Design: The choice of harness significantly affected performance, with differences reaching 20 percentage points for the same model.
  • Tool Use Patterns: "Smart parallelizers" (e.g., GPT-5.5 with Codex) that batched operations outperformed "direct executors" that locked onto single tools.
  • Multimodal Primitives: Harnesses with typed primitives for specific modalities (e.g., OpenClaw's image/TTS routing) enabled better performance in Sequencing but sometimes reduced performance in Assembly due to a lack of direct pixel access. Conversely, harnesses relying on generic shell tools forced agents to compose multimodal pipelines from scratch, often leading to failure.

Significance and Claims

The paper positions AgenticVBench as a foundational diagnostic tool for the field of agentic video production. Its primary contributions are:

1. Systematic Coverage: It is the first benchmark to systematically cover four major post-production task families across the full text-image-video-audio stack.
2. Production Realism: Tasks are authored by industry experts based on real workflows, ensuring rigorous quality control and relevance to professional standards.
3. Diagnostic Value: The results demonstrate that current frontier models are far from expert performance and that harness design is a first-order variable in agent capability. The paper argues that progress in this domain requires not only stronger multimodal models but also better tool interfaces, state management, and workflow-aware harness design.
4. Open Specification: The authors release full evaluation rubrics and specifications to facilitate reproducible research and the development of safer, more capable agentic video systems.

The authors conclude that while AI agents can produce watchable outputs, they currently lack the composite reasoning and tool-use reliability required for professional post-production, necessitating continued human supervision.