ExStrucTiny: A Benchmark for Schema-Variable Structured Information Extraction from Document Images

1. Problem Statement

Enterprise documents (forms, reports, receipts) contain critical information requiring extraction for downstream applications like analytics and automation. While Vision Language Models (VLMs) have advanced, their ability to perform holistic, fine-grained structured extraction across diverse document types and flexible schemas remains under-studied.

Existing benchmarks suffer from three main limitations:

• Narrow Ontologies: Key Entity Extraction (KEE) datasets (e.g., CORD, FUNSD) rely on fixed, domain-specific entity sets, failing to test generalization to new entities or user requirements.
• Simple Queries: Visual Question Answering (VQA) datasets often use single-span, closed queries with high lexical overlap with the document text, allowing models to succeed via string matching rather than semantic understanding.
• Lack of Structural Complexity: Most datasets do not require models to extract multiple related entities, handle missing information (unanswerable queries), or adapt to variable schemas (e.g., JSON structures) in a zero-shot or few-shot setting.

2. Methodology

A. Dataset Construction (EXSTRUCTINY)

The authors introduce EXSTRUCTINY, a benchmark comprising 304 query-answer (QA) pairs across 110 multi-page documents (forms, financial reports, slide decks, and web screenshots). The dataset unifies KEE, Relation Extraction (RE), and VQA tasks.

Data Generation Pipeline:

1. Manual Annotation: A small subset (102 QA pairs) was manually annotated by experts to define high-quality ground truth.
2. Synthetic Generation: A large-scale synthetic dataset (202 QA pairs) was generated using Gemini-2.5-Flash-Thinking.
   • Prompting Strategy: The model was provided with few-shot examples and Chain-of-Thought (CoT) reasoning to ensure adherence to complex output structures.
   • Augmentation: The pipeline included specific augmentation streams to:
     • Reformulate: Reduce lexical overlap between queries and document text.
     • Introduce Missing Entities: Simulate real-world scenarios where requested data is absent.
     • Generate Unanswerable Queries: Create fully unanswerable cases.
3. Human Validation: A rigorous manual validation process corrected errors in text extraction, page references, bounding boxes, and schema structuring. Only 2 samples were rejected; the rest were repaired.

Task Formulation:
The task requires models to extract information based on three query types:

• Closed IE (Plain Text): Explicitly requests specific entities.
• Closed IE (Schema): Provides a JSON schema to populate.
• On-Demand IE: Implicitly requests a parent entity, requiring the model to infer and extract child attributes without a predefined schema.

Output Format:
All answers must be JSON-formatted objects containing:

• text: The verbatim extracted value.
• page: The page index.
• bbox: Normalized bounding box coordinates.
• answer_context: Classification of the source (e.g., free text, table, chart).

B. Evaluation Framework

To address the challenge of varying output structures (different key names or nesting), the authors propose a semantic mapping approach:

1. Schema Mapping: A text-only LLM (specifically gpt-oss-20b) is used to align ground-truth keys with predicted keys, handling paraphrasing and structural differences.
2. Metrics:
   • Structure Prediction: Tree-edit distance and schema mapping Recall/Precision/F1.
   • Text Extraction: ANLS (Average Normalized Levenshtein Similarity).
   • Localization: Page accuracy, Bounding Box IoU, and Normalized Proximity.

3. Key Contributions

1. EXSTRUCTINY Benchmark: The first dataset to unify closed, schema-based, and on-demand IE tasks with variable schemas and multi-entity extraction requirements across diverse document types.
2. Novel Generation Pipeline: A hybrid approach combining manual annotation with LLM-generated, human-validated synthetic data to scale dataset creation while maintaining high quality and diversity.
3. Evaluation Framework: A robust metric system that decouples structural alignment from content extraction, enabling fair evaluation of models with different output formats.
4. Comprehensive Benchmarking: Extensive analysis of both open-source and closed-source VLMs, highlighting specific failure modes in current state-of-the-art models.

4. Results and Findings

Model Performance:

• Closed vs. Open: Closed-source models (e.g., Gemini-2.5-Pro) significantly outperform open-source models, with a gap of 18+ ANLS points.
• Scale Correlation: Performance strongly correlates with model size. For example, within the Qwen2.5-VL family, performance jumps from 38.5 (3B) to 61.4 (72B).
• Query Difficulty:
  • Schema & On-Demand: Models struggle significantly with schema-based and on-demand queries compared to plain-text closed queries. On-demand queries are the most difficult due to underspecification.
  • Extraction Length: Open-source models show a sharp decline in performance as the number of values to extract increases, whereas closed models remain stable.
• Localization: All models struggle with spatial grounding. The best model (Gemini-2.5-Pro) achieves only 14.4% IoU for bounding boxes and 84.3% page accuracy, indicating a gap between text extraction and evidence localization.
• Context Sensitivity: Performance drops significantly in charts and free-text contexts, where visual distractors and dense text hinder extraction.

Data Quality:

• Models perform slightly worse on manually annotated data (13.6% lower ANLS) than synthetic data, suggesting manual queries are harder.
• There is no bias toward the model used for generation (Gemini), validating the effectiveness of the human validation pipeline.

5. Significance and Future Work

• Real-World Relevance: EXSTRUCTINY addresses the gap between academic benchmarks and real-world enterprise needs, where schemas are dynamic, queries are underspecified, and data is often missing.
• Research Direction: The results highlight that current VLMs lack robustness in schema adaptation, query under-specification, and answer localization.
• Limitations: The dataset is currently English-only, and the ANLS metric may be suboptimal for numerical/date extraction. The schema mapping step relies on an external LLM, which adds latency.

Conclusion:
EXSTRUCTINY serves as a critical "bedrock" for advancing generalist VLMs. It demonstrates that while large models are improving, significant challenges remain in achieving reliable, structured, and grounded information extraction from visually rich documents without domain-specific fine-tuning.