OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning

The paper introduces OfficeQA Pro, a challenging enterprise benchmark using a massive corpus of U.S. Treasury Bulletins to demonstrate that current frontier AI agents struggle significantly with grounded, multi-document reasoning, achieving low accuracy even with direct document access and benefiting notably from structured document representations.

The Gist

Imagine you are hiring a team of super-intelligent, hyper-fast interns to work for the U.S. Treasury Department. Your goal isn't to ask them to write a poem or solve a riddle; you want them to dig through a massive, century-old attic filled with 89,000 pages of dusty financial reports, tax tables, and charts.

Your job is to ask them specific, tricky questions like: "What was the exact difference in national defense spending between 1940 and 1953, adjusted for inflation?"

This paper, OfficeQA Pro, is essentially a report card on how well these AI "interns" (the world's smartest AI models) can handle this specific, messy, real-world job.

Here is the breakdown of what they found, using some everyday analogies:

1. The "Attic" (The Data)

The researchers didn't use a clean, digital library. They used the U.S. Treasury Bulletins from 1939 to 1982.

• The Analogy: Imagine a library where some books are pristine digital PDFs, but others are 80-year-old photocopies of handwritten notes, with coffee stains, torn pages, and tables that look like spaghetti.
• The Challenge: The AI has to find a specific number in a table from 1952, realize that the number was updated in 1955, and then do some math to adjust it for inflation.

2. The "Interns" (The AI Models)

The researchers tested the "big three" of AI: Claude, GPT (OpenAI), and Gemini (Google). They treated them in two ways:

• The "Memory Test": They asked the AI questions without letting them look at the documents.
  • Result: The AI failed miserably (less than 5% accuracy). It's like asking a student to recite a specific page from a book they haven't read in 10 years. They guessed, and they were wrong.
• The "Open Book Test": They gave the AI the documents to read.
  • Result: Still not great. Even with the books open, the best AI only got about 34% of the answers right.

3. The "Translator" Problem (Why they failed)

Why did the AI struggle even with the books open?

• The Analogy: Imagine handing a human a book written in a language they don't speak, or a book where the text is smudged and the tables are drawn in crayon. The AI has to "read" the PDF first.
• The Issue: Standard AI tools are bad at reading messy, old PDFs. They often misread a "6" as an "8," or they get confused by a table with nested headers (like a family tree of numbers).
• The Fix: The researchers used a special tool called ai_parse_document (made by Databricks) to act as a "super-translator." It cleaned up the messy PDFs, organized the tables, and turned them into clear text.
• The Result: When the AI got the "cleaned-up" version of the documents, its performance jumped by 16%. It's like giving the intern a highlighter and a clean copy of the book instead of a blurry photocopy.

4. The "Search Engine" Problem

Even with clean documents, the AI had trouble finding the right page.

• The Analogy: Imagine asking a librarian to find a fact in a library with 100,000 books. If the librarian just grabs the first book that has the word "tax" in it, they might miss the specific book you need.
• The Finding: The AI often grabbed the wrong data or used the wrong formula. It was like a student who knows the concept of math but keeps picking the wrong numbers from the textbook.

5. The "Human vs. Robot" Showdown

The researchers also hired real humans to do the same test.

• The Twist: The humans were actually slower and made more mistakes than the AI in some ways!
  • Humans got confused by the formatting, made typos when typing numbers, and got tired.
  • The AI was faster and more consistent, but only if the documents were cleaned up first.
  • The Verdict: The AI is a faster, tireless worker, but it needs a good "foreman" (the parsing tool) to prepare the work for it.

The Big Takeaway

The paper concludes that while AI is getting incredibly smart at abstract puzzles (like solving math Olympiads), it is still not ready for the messy reality of enterprise work.

• Current State: If you ask an AI to do a real-world financial analysis today, it will likely get it wrong about 60-70% of the time.
• The Bottleneck: The problem isn't the AI's "brain" (reasoning); it's its "eyes" (reading messy documents) and its "search skills" (finding the right needle in the haystack).
• The Future: We need better tools to clean up documents and better strategies for searching them before we can trust AI to run a company's finances.

In short: OfficeQA Pro is a reality check. It shows that while AI is a genius at theory, it's still a clumsy apprentice when it comes to the dirty, detailed work of real-world business.

Technical Summary

Here is a detailed technical summary of the paper "OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning."

1. Problem Statement

Current frontier AI benchmarks (e.g., HLE, ARC-AGI-2) often focus on abstract reasoning or closed-world tasks where the full context is provided in the prompt. These fail to capture the complexities of enterprise-grade grounded reasoning, which requires:

• Navigating large, heterogeneous document corpora (millions of pages).
• Faithfully retrieving relevant information from unstructured text and complex tables.
• Performing multi-step analytical reasoning and precise numerical computation.
• Handling data drift, revisions, and varying document formats (scanned PDFs vs. digital-native).

Existing benchmarks often rely on human-expert grading (making evaluation slow) or lack the scale to test retrieval capabilities over decades of historical data.

2. Methodology: The OfficeQA Pro Benchmark

Dataset Construction

• Corpus: The benchmark is built on U.S. Treasury Bulletins published from 1939 to the present.
  • Scale: ~89,000 pages, 26+ million numerical values.
  • Complexity: Contains prose, nested tables, charts, footnotes, and data revisions. It transitions from scanned physical documents (pre-1996) to digital-native PDFs.
• Question Set:
  • OfficeQA Pro: 133 "hard" questions requiring precise parsing, retrieval, and analysis (e.g., linear regression, inflation adjustments, multi-document reconciliation).
  • OfficeQA Full: Includes 113 additional "easy" questions for iteration and testing smaller models.
• Creation Process:
  • Questions were generated by annotators and verified by USAFacts (a non-profit data organization) to ensure they reflect real-world analyst queries.
  • Verification: A rigorous multi-stage process involving human re-annotation, AI agent cross-checking, and ground-truth correction to ensure a single, unambiguous answer.
  • Filtering: Questions solvable by parametric knowledge alone were removed to force reliance on the document corpus.

Evaluation Metrics

• Correctness: Deterministic exact match for non-numerical answers.
• Numerical Tolerance: For the 99% of numerical answers, accuracy is measured using Absolute Relative Error (ARE). The primary metric is 0.0% ARE (exact match), with secondary analysis at 0.1%, 1.0%, and 5.0% thresholds.
• Efficiency: Latency (time to answer), cost, and tool call counts are tracked.

3. Key Contributions

1. A New Enterprise Benchmark: OfficeQA Pro is the first benchmark specifically designed to evaluate end-to-end grounded reasoning over a century-long, heterogeneous document archive with verifiable, automated grading.
2. Comprehensive Agent Evaluation: The paper evaluates state-of-the-art models (Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro Preview) across three frameworks (Anthropic, OpenAI, Google) in both LLM-only and Agent (tool-using) configurations.
3. Analysis of Document Parsing: A critical contribution is the quantification of how document parsing quality impacts downstream reasoning. The authors compare raw PDFs against structured representations generated by Databricks' ai_parse_document.
4. Failure Mode Analysis: The paper identifies specific bottlenecks in current AI systems, including temporal revision handling, visual reasoning, and analytical precision.

4. Key Results

Performance Baselines

• Parametric Knowledge: Frontier models achieve <5% accuracy when relying solely on internal knowledge.
• Web Search: Adding web search improves performance slightly (up to ~12%), but models still struggle with multi-step reasoning and often cite incorrect sources.
• Oracle Retrieval (Full Corpus): Even when provided with the correct documents (Oracle Pages) and web search, frontier agents average only 34.1% accuracy.
• Best Agent Performance: The Claude Opus 4.6 agent, using Databricks' parsed documents and Oracle pages, achieved the highest score of 66.9% (with 0.0% error tolerance).

Impact of Document Parsing

• Structured Representation: Providing agents with documents pre-parsed by Databricks' ai_parse_document (converting complex tables/layouts into structured text) yielded a 16.1% average relative performance gain across all agents.
• Latency & Cost: Parsed documents reduced latency by 4–9x and significantly lowered costs by eliminating the need for agents to install and run heavy OCR/PDF libraries during execution.
• Comparison: Raw PDFs caused significant extraction errors (misaligned tables, OCR noise), leading to a 40–50% failure rate in parsing-dependent tasks.

Ablation Studies (Custom Agent)

• Model Selection: Claude Opus 4.6 dominated the correctness vs. latency Pareto frontier, while OpenAI models offered better cost-efficiency due to lower input token pricing.
• Table Representation: HTML generally performed slightly better than Hierarchical Markdown for table representation, though results were model-dependent.
• Retrieval Strategy: Combining File Search (grep/sed style) with Contextual Vector Search (embedding with metadata) provided the best quality-cost tradeoff, outperforming pure vector search which struggled with table context.
• Test-Time Scaling: Plurality voting (running multiple rollouts and voting) improved accuracy for weaker models but offered diminishing returns for top-tier models.

5. Failure Modes Identified

The paper highlights several persistent challenges preventing reliable enterprise AI:

1. Temporal Revision Verification: Agents fail to identify the latest revised values in historical documents, often settling on the first plausible number found.
2. Parsing Faithfulness: Even with state-of-the-art parsers, subtle errors (shifted rows, lost footnotes) propagate to calculation errors.
3. Visual Reasoning: Agents struggle significantly with charts and graphs, especially when visual data is dissociated from its semantic context in the parsing pipeline.
4. Analytical Reasoning: Errors in applying correct formulas (e.g., sample vs. population variance), incorrect inflation adjustments, and premature rounding.

6. Significance and Future Work

• Real-World Gap: The results demonstrate a massive gap between "Olympiad-style" reasoning (where models excel) and "Enterprise Grounded Reasoning" (where they struggle). Current agents are not yet reliable for high-stakes financial workflows.
• Human vs. AI: Interestingly, while AI agents outperformed human annotators in speed and accuracy when given parsed documents, humans outperformed agents on raw PDFs due to superior visual interpretation and lack of parsing errors.
• Future Directions: The authors suggest future work should focus on:
  • Revision-aware search strategies.
  • Improved visual reasoning and chart understanding.
  • Better handling of complex table topologies.
  • Expanding the benchmark to include databases and multi-modal inputs.

Conclusion: OfficeQA Pro establishes a rigorous standard for evaluating AI in enterprise settings. It proves that while frontier models are powerful, their reliability in grounded reasoning is currently bottlenecked by document parsing quality and retrieval fidelity, not just raw reasoning capability.