Technical Summary: GDP.pdf Benchmarking Grounded Multimodal Reasoning over Professional PDF Documents

1. Problem Statement

Current benchmarks for document AI and multimodal reasoning typically evaluate capabilities in isolation—such as Optical Character Recognition (OCR), layout analysis, chart reasoning, or table Question Answering (QA). While models score highly on these isolated tasks, these scores do not reliably predict performance on realistic professional workflows. In professional domains (e.g., law, healthcare, finance), answering a question often requires navigating complex, variably formatted PDFs where the decisive evidence may be located in a footnote, a superseded amendment, a cross-referenced table, or a specific symbol in a floor plan.

The core problem identified is that current models often fail to distinguish between a superficially correct answer and one grounded in the correct evidence. A model may cite a real clause or a number present on the page but fail because it is not the governing clause or the correct value for the specific context. Furthermore, existing benchmarks often rely on cleaned inputs (extracted text or isolated charts), removing the structural noise and complexity inherent in real-world PDFs.

2. Methodology: GDP.pdf Benchmark Design

2.1 Task Formulation and Curation

The GDP.pdf benchmark consists of 100 question-document pairs across ten professional domains: Finance, Healthcare, Legal, STEM/Research, Engineering, Construction, Manufacturing/Supply Chain, Insurance, Real Estate, and Human Resources.

The curation process followed strict principles:

• Expert Authorship: Questions were written by working professionals based on their actual workflows, using original, uncleaned PDF files.
• Adversarial Screening: A candidate task was only admitted if at least two frontier multimodal models failed it in a "major" way. A major failure is defined as a materially wrong final answer, a missed piece of decisive evidence, or a fabricated claim. Superficial stylistic differences were not considered failures.
• Prior Knowledge Filtering: Tasks answerable via general world knowledge without the document were rejected.
• Rubric-Based Grading: Each item includes an expert-authored rubric containing atomic yes/no criteria. This allows for a graded rubric score (measuring partial progress) and a strict pass rate (requiring all criteria to be met).

2.2 Capability Taxonomy

The benchmark utilizes a taxonomy of 11 capability axes organized into three tiers to tag tasks and analyze errors:

• Tier 1 (Extraction & Grounding): Correctness/completeness of text extraction, grounding of facts in the PDF (no hallucination), and spatial awareness.
• Tier 2 (Structural & Multimodal Comprehension): Semantic reading flow (handling sidebars/columns), typographic hierarchy, standard table parsing (merged cells), and chart/visual interpretation.
• Tier 3 (Advanced Reasoning): Handling complex/multi-page tables, cross-referencing (footnotes, distant definitions), managing artifact/noise (scans, superseded content), and correctly abstaining on unsupported queries.

2.3 Evaluation Protocol

Evaluation relies on an atomic rubric rather than overlap metrics (like ANLS). A response can be textually similar to the reference but fail if it includes a disqualified entity or misses a critical exclusion.

• Metrics: Models are ranked by Strict Pass Rate (percentage of items where all rubric criteria are satisfied). A Mean Rubric Score is also reported for finer-grained analysis.
• Judging: Criteria are graded by an LLM judge (Gemini 3.5 Flash) calibrated against human experts. The judge evaluates criteria in isolation without access to the source PDF or gold answer to prevent anchoring.

3. Key Contributions

1. The GDP.pdf Benchmark: A dataset of 100 expert-authored tasks over professional PDFs in ten domains, screened to ensure every item defeats at least two frontier models.
2. A Capability Taxonomy: A three-tiered framework of 11 axes used to tag tasks and organize error analysis, bridging the gap between extraction and advanced reasoning.
3. Evaluation Protocol: A rubric-based system that reports both a graded score and a strict pass rate, emphasizing grounded evidence over superficial text matching.
4. Pilot Study & Failure Analysis: An evaluation of seven frontier models revealing specific, recurring failure patterns in professional document processing.

4. Results: Pilot Evaluation

Seven frontier models (Gemini 3.1 Pro, Claude Opus 4.7, GPT-5.4, Grok 4.20, Kimi K2.5, Mistral Large 3, and Nova 2 Pro) were evaluated on the 100-item benchmark.

• Performance: The best-performing model achieved a strict pass rate of only 15%. The worst model achieved 1%. The top three models showed overlapping confidence intervals, suggesting the ranking is provisional.
• Failure Patterns: Errors were concentrated in specific areas:
  • Table Alignment: Misreading rows/columns, particularly in tables with merged cells or stacked headers.
  • Chart Misreads: Identifying the correct figure but extracting implausible values (e.g., misreading logarithmic scales).
  • Dropped Evidence: Ignoring footnotes, cross-references, and exclusions that govern the answer.
  • Prior Knowledge Override: When document content contradicted general training priors, models often defaulted to the prior (e.g., recommending standard machining practices despite specific document instructions to the contrary).
  • Spatial Reasoning: Significant difficulty in floor plans and construction drawings, such as matching symbols to legends or counting fixtures across pages.
  • Noise and Supersession: Models frequently quoted sections of deeds or reports that had been superseded by amendments or misinterpreted status labels (e.g., "PENDING").

5. Significance and Claims

The paper claims that GDP.pdf reveals a critical gap between current multimodal model capabilities and the requirements of professional document automation.

• Misleading Standard Scores: High scores on academic or broad multimodal benchmarks (like MMMU or DocVQA) do not translate to reliability in professional settings. The gap exists because GDP.pdf tests where the answer comes from, not just what the answer is.
• The "Floor" Question: The benchmark shifts the focus from the "ceiling" (what is the hardest task a model can solve?) to the "floor" (what can a model be counted on to get right every time?). The current 1–15% pass rate indicates that models are not yet ready for unsupervised use in these workflows.
• Interplay of Perception and Reasoning: Failures are rarely purely perceptual or purely reasoning-based; they often involve a model correctly reading text but failing to apply domain-specific logic (e.g., knowing to check exclusions in an insurance policy).
• Implications for Development: The authors suggest that simply increasing context window size is insufficient. Future improvements likely require better page representations that preserve structure, parsers that handle charts/tables robustly, and retrieval systems that do not strip visual evidence. Additionally, better uncertainty calibration is needed for models to correctly abstain on unsupported queries.

The authors conclude that GDP.pdf provides a necessary tool for measuring whether models can handle the routine, high-stakes document work that underpins a significant share of professional economic activity.