Technical Summary: Office Comprehension Bench (OCB)

Problem Statement

Office documents (Word, Excel, PowerPoint) are the primary medium for knowledge work, combining free-form text, structured tables, embedded images, charts, and application-specific constructs (e.g., slide layouts, formulas, named ranges). Despite rapid progress in Large Language Models (LLMs), rigorous evaluation of "office comprehension" remains limited. Existing benchmarks often focus on narrow slices: single-artifact images (DocVQA, SlideVQA), rendered PDFs (MMLongBench-Doc, GDP.PDF), or spreadsheet manipulation (SpreadsheetBench). Crucially, few evaluate systems on native file formats (.docx, .xlsx, .pptx) where the model must parse the file structure itself, not just extracted text or rendered images. Furthermore, current evaluations often lack the granularity to distinguish between simple retrieval and the multi-step, expert-level reasoning required for professional tasks.

Methodology

The authors introduce Office Comprehension Bench (OCB), the first public benchmark to jointly evaluate LLM systems on Word, Excel, and PowerPoint comprehension over native file formats. OCB is structured around two complementary tracks:

1. Dataset Composition

• File Fidelity Q&A: Tests structural and visual perception. It comprises 240 files and 902 queries targeting specific artifacts (tables, charts, headers, formulas, speaker notes, etc.). The evaluation density is low, averaging ~1.45 atomic assertions per query.
• Domain Q&A: Tests expert-level reasoning grounded in real-world industry documents. It comprises 64 files and 120 queries across 12 professional domains (Finance, Healthcare, Supply Chain, etc.). These queries require multi-step analysis and synthesis across documents, resulting in high evaluation density (~45.12 atomic assertions per query).

2. Construction Pipeline

Both tracks follow a four-stage pipeline: source curation, question authoring, rubric construction, and quality assurance.

• Source Curation: Domain Q&A uses authoritative industry artifacts (SEC filings, slide decks) curated by experts. File Fidelity uses a broad public pool filtered for artifact coverage.
• Rubric Construction: Reference answers are decomposed into atomic, binary-gradable claims. For Domain Q&A, rubrics are constructed via three authorship variants (independent, derived, and cross-organization) to reduce bias.
• Quality Assurance: Assertions undergo iterative review to eliminate duplicates, compound claims, ambiguities, and prompt-restating assertions.

3. Evaluation Framework

• Input: Systems are evaluated via their public web chat interfaces (not APIs) to simulate real-world user interaction with native files.
• Scoring: The primary metric is assertion-level accuracy. Each query is associated with a set of atomic assertions; a system's score is the fraction of satisfied assertions.
• LLM-as-a-Judge: To score responses, an ensemble of three LLM judges (GPT-5.4 Thinking, Gemini 3.1 Pro, Claude Opus 4.6) evaluates each assertion independently. The final verdict is a majority vote. This approach leverages the LLM-as-a-Judge paradigm while mitigating individual model bias through diversity.
• Variance Analysis: The authors employ a 3-scrape × 3-eval protocol for Domain Q&A to decompose variance into "Scrape Var" (model response sampling) and "Eval Var" (judge noise). They find that response-sampling noise dominates, justifying the focus on multiple scrapes rather than repeated judge evaluations.

Key Contributions

1. First Public Native-Format Benchmark: OCB is the first to jointly cover Word, Excel, and PowerPoint in their native formats under a unified methodology, addressing the gap left by PDF-rendered or single-artifact benchmarks.
2. Scalable Open-Ended Evaluation: The authors demonstrate a methodology for decomposing complex, open-ended professional analyses into atomic, binary-gradable claims. They provide a failure-mode taxonomy for rubric construction (e.g., compound, ambiguous, overly broad assertions) and a tooling suite for the community.
3. Empirical Findings on Frontier Systems: The benchmark provides a rigorous stress test for state-of-the-art models, revealing specific failure modes and capability ceilings.

Results

The authors evaluated three frontier systems: OpenAI GPT-5.5 Thinking, Anthropic Claude Opus 4.7, and Google Gemini 3.1 Pro.

• Domain Q&A Performance: Even the strongest system (GPT-5.5 Thinking) reaches only ~59.3% accuracy in its default reasoning mode. Increasing "thinking depth" within a model family (e.g., from Light to Heavy) yields negligible gains (statistically indistinguishable from noise). Moving to a higher product tier (e.g., GPT-5.5 Pro-Extended) yields modest gains (~63%), but performance remains far from perfect.
  • GPT-5.5 Thinking: 59.35%
  • Claude Opus 4.7: 56.83%
  • Gemini 3.1 Pro: 45.75%
• File Fidelity Performance: Frontier systems outperform human annotators (who lack specialized domain expertise) on structural perception tasks.
  • Word: Claude leads (91.5%) significantly over GPT-5.5 (68.7%) and Gemini (55.3%).
  • PowerPoint: Claude (86.6%) and GPT-5.5 (85.9%) are closely tied, both far ahead of Gemini (64.9%).
  • Excel: All systems struggle with "Table-Like Ranges" (unstructured contiguous cell ranges), scoring between 48–58%. GPT-5.5 leads slightly (72.6%) over Claude (71.1%) and Gemini (61.0%).
• Failure Modes: Qualitative analysis reveals four recurring error types:
  1. Cross-document sourcing failures: Dropping or mixing values across multiple files.
  2. Value hallucinations: Reporting numbers not present in the source.
  3. Methodology failures: Ignoring explicit formulas or constraints in the prompt.
  4. Computational errors: Basic arithmetic or ratio mistakes.

Significance and Claims

The paper claims that OCB reveals a split capability profile for current frontier systems:

• Super-Annotator: Systems excel at artifact-level perception and recall (File Fidelity), often exceeding human baselines on routine structural tasks.
• Sub-Expert: Systems remain "sub-expert" on multi-step analytical reasoning (Domain Q&A), plateauing well below 60% accuracy even with increased compute.

The authors position OCB as a necessary prerequisite for trustworthy automation. They argue that reliable comprehension is a fundamental bottleneck for agentic workflows (like those evaluated in OfficeBench or OdysseyBench). By providing a public dataset, evaluation tooling, and a leaderboard, the authors aim to accelerate progress toward LLM systems that are accurate, grounded, and robust across the artifacts that define modern office work. The paper remains modest, acknowledging limitations such as the lack of a multi-rater human baseline for Domain Q&A, single-turn evaluation, and the current limitation to English.