Technical Summary: ExtractBench

Problem Statement

Enterprise workflows are increasingly shifting from manual data entry to agent-driven, schema-guided document extraction. In this paradigm, an agent must take a document (often a complex PDF) and a user-defined JSON schema, then produce a structured output that faithfully adheres to the schema, including source evidence (grounding) for every extracted value.

Existing benchmarks for document extraction suffer from critical limitations when applied to this specific enterprise use case:

• Fixed Ontologies: Classic benchmarks (e.g., SROIE, DocILE) test recovery of a fixed set of fields, failing to evaluate a system's ability to follow arbitrary, user-defined schemas.
• Narrow Scope: Recent schema-guided benchmarks often cover only a single dimension of the problem (e.g., only long lists, only synthetic data, or only specific document types) and lack real-world noise like scans and handwriting.
• Missing Metrics: Existing evaluations typically ignore visual grounding (the ability to point to the source of a value) and cost per page, both of which are critical for production-scale deployment.
• Lack of Challenge Tagging: Aggregate scores fail to distinguish between a system failing due to a difficult task (e.g., long lists) versus a perception issue (e.g., poor scan quality).

Methodology: The ExtractBench Framework

ExtractBench is a comprehensive benchmark designed to evaluate schema-guided extraction across value accuracy, grounding, and cost simultaneously.

1. Corpus and Taxonomy

The benchmark consists of 370 documents spanning 4,869 pages, covering 8 business domains (Finance, Energy, Government, etc.) and 67 document types.

• Sources: The corpus includes real born-digital documents, synthetic long lists based on real layouts, and real regulatory/tax forms.
• Challenge Tagging: Each document is tagged along five independent axes to enable granular failure analysis:
  • Task Challenge: Long-list completeness (T1), needle-in-haystack (T2), and dense documents (T3).
  • Perception Challenge: Rotated/image-only (P1), scanned (P2), and handwriting (P3).
  • Table Structure: Merged headers, cross-page tables, enormous tables, etc.
  • Length: Short (≤10pp), Medium (11–50pp), Long (>50pp).
  • Domain: Specific industry verticals.

2. Ground-Truth Curation Pipeline

To generate high-quality ground truth without prohibitive manual labeling costs, the authors employed a scalable, source-specific pipeline:

• Real Documents: Schemas are drafted from samples. Multiple extraction systems (frontier models) run against the schema. Values where all systems agree become candidate ground truth; disagreements are adjudicated by humans or schema refinement.
• Synthetic Long Lists: Built "backwards" (data first, document second). Records are generated or parsed from real filings, then rendered into PDFs by coding agents. Ground truth is exact by construction, with boxes read directly from the render.
• Scanned Forms: Schemas are authored against blank templates. An ensemble of systems votes on values, with contested fields adjudicated by humans who also verify and redraw bounding boxes.

3. Evaluation Metrics

• Value Accuracy: Measured via order-insensitive Value F1. Extracted JSON is flattened into cells. Arrays are aligned using the Hungarian algorithm to minimize mismatched records. Values are normalized (e.g., dates to ISO) before comparison.
• Grounding:
  • Word-level F1: Requires the value to be correct and the predicted bounding box to overlap the ground-truth box (IoU ≥ 0.5).
  • Page-level F1: Requires the correct value and the correct source page.
• Cost: Measured in cents per page based on actual token/credit consumption.

4. Systems Evaluated

The benchmark evaluated 14 frontier systems across three categories:

• Commercial VLMs: (e.g., Gemini 3.5 Flash, GPT-5.4 Nano).
• Coding Agents: (e.g., Claude Code Opus 4.8, Codex GPT-5.5) which use iterative tool loops to parse documents.
• Specialized APIs: (e.g., LlamaExtract tiers, Reducto, Extend, Datalab).

Key Results

Quality–Cost Frontier

The evaluation reveals a distinct tradeoff landscape:

• Commercial VLMs: Offer the lowest cost (≤1.0 ¢/page) but struggle with accuracy, particularly on long documents (F1 drops from ~88% on short docs to ~28% on long docs for Gemini 3.5 Flash). They frequently truncate long record lists.
• Coding Agents: Achieve high accuracy (87–94% F1) and handle long documents well but at a significantly higher cost (>15 ¢/page).
• Specialized APIs: Span the widest range. LlamaExtract Agentic Plus emerges as the leader, achieving 95.6% F1 at 8.1 ¢/page. This outperforms coding agents in accuracy while costing less than half as much.

Dimension-Specific Performance

• Document Length: Most systems degrade significantly on documents >50 pages. LlamaExtract Agentic Plus is the only system to maintain >90% F1 across all length buckets.
• Table Structure: "Enormous tables" (>1,000 rows) cause catastrophic failure for most VLMs (F1 < 10%). Specialized APIs and coding agents perform significantly better here.
• Perception: Systems show specific blind spots; for instance, Codex GPT-5.5 struggles with rotated images, while some APIs struggle with scanned pages but handle handwriting well.

The Grounding Gap

A critical finding is the lack of source traceability in general models:

• Standard VLMs and coding agents do not return word-level boxes by default, scoring 0.0% on word-level grounding.
• Even specialized systems that do return boxes struggle with precision: the best word-level grounding F1 is 46.4% (LlamaExtract Agentic Plus), while page-level grounding is much higher (84.9%). This indicates that while systems can find the right page, reliably connecting a value to its exact word-level location remains an open challenge.

Significance and Claims

The paper claims to provide the first evaluation that jointly reports value accuracy, record completeness, grounding, and measured cost at scale for schema-guided extraction.

• Diagnostic Power: By using challenge tags, ExtractBench moves beyond aggregate scores to diagnose why systems fail (e.g., distinguishing between a perception failure and a long-list truncation).
• Production Relevance: The inclusion of cost and grounding metrics addresses the specific requirements of enterprise deployment, where financial viability and auditability are as important as raw accuracy.
• State of the Art: The results suggest that while general-purpose VLMs are cost-effective for simple tasks, specialized agentic APIs currently offer the best balance of high accuracy, robustness to document complexity, and cost-efficiency for enterprise workflows.

The authors conclude that reliable extraction at scale requires systems that can produce complete outputs, provide traceable evidence, and operate at a sustainable cost, a combination currently best achieved by specialized agentic solutions like LlamaExtract Agentic Plus.