LLMStructBench: Benchmarking Large Language Model Structured Data Extraction

1. Problem Statement

Extracting structured data (specifically JSON) from unstructured natural language text is a critical requirement for automation in domains like HR, finance, and logistics. While Large Language Models (LLMs) are increasingly used for these Extract, Transform, Load (ETL) tasks, their reliability remains insufficiently understood.

• The Gap: Existing models often fail to produce outputs that are both syntactically valid (parseable JSON) and semantically correct (accurate values matching the source text).
• The Constraint: Many enterprise workflows require locally deployable, privacy-preserving solutions (open-source models) rather than proprietary cloud APIs, yet benchmarks often focus on closed models or constrained decoding engines rather than open-weight model performance under realistic prompting.
• The Challenge: There is a lack of systematic evaluation on how prompting strategies and model size trade off between structural validity (getting the JSON format right) and semantic accuracy (getting the data values right).

2. Methodology

A. Dataset: LLMStructBench

The authors created a novel, open dataset comprising 995 manually verified test cases across 5 distinct use cases (Support tickets, Sick leave, Project extension, Conference registration, Loan requests).

• Generation: Synthetic data was generated using GPT-4o to create realistic natural language messages (e.g., emails) paired with ground-truth (GT) JSON objects and schemas.
• Verification: All samples were manually reviewed to ensure the text contained all information required by the schema and that the GT JSON was perfectly aligned with the text.
• Complexity: The dataset ranges from flat key-value pairs to deeply nested structures (up to 4 levels deep) with arrays of objects.

B. Evaluation Metrics

The paper introduces a dual-metric approach to capture both token-level accuracy and document-level validity:

1. Micro-averaged F1 ($F1_{micro}$): Measures semantic extraction accuracy. It weights Key presence (25%) and Value correctness (75%). It uses a nuanced credit system for partial matches (e.g., string vs. integer coercion, Levenshtein distance for string deviations) while penalizing missing keys or type errors heavily.
2. Document Success ($DOC_{micro}$): Measures the percentage of tests where the model returned a perfectly valid JSON object with no parsing errors, empty outputs, or fatal schema violations. This captures "brittleness" that token-level metrics might hide.
3. Composite Score: A weighted combination ($Score = 0.5 \times F1_{micro} + 0.5 \times DOC_{micro}$) to rank models holistically.

C. Experimental Setup

• Models: Evaluated 22 open-source LLMs (families: Qwen3, Phi, Gemma3, Deepseek-R1, Llama3) ranging from 0.6B to 70B parameters. GPT-4o was included as a proprietary reference.
• Prompting Strategies: Five distinct strategies were tested to isolate the impact of API settings vs. prompt instructions:
  • J: API format set to "json" only.
  • P: System prompt explicitly requests JSON with schema/example (No API format setting).
  • PJ: Combination of J and P.
  • J+: API format set to full JSON Schema object.
  • PJ+: Combination of J+ and P (Schema in API + Schema in Prompt).

3. Key Results

A. Prompting Strategy Dominance

• Strategy > Size: The choice of prompting strategy was found to be more critical than model size.
• Best Performers: The P (Prompt-only with schema) and PJ+ (API Schema + Prompt Schema) strategies generally yielded the best results.
  • P often provided the best balance of accuracy and validity for well-aligned models.
  • PJ+ was the "safest" strategy for ensuring parseable outputs (high $DOC_{micro}$), particularly for smaller or less reliable models, though it sometimes increased semantic errors.
• Failure Cases: The Phi family (specifically 3.8B variants) struggled significantly with the P strategy, resulting in high failure rates, highlighting that not all models respond well to the same prompting techniques.

B. Model Size vs. Performance

• Non-Monotonic Scaling: There is no strict linear relationship between parameter count and performance.
  • Small Models Outperforming Large Ones: Gemma3-12B outperformed several 70B models. Qwen3-1.7B ranked mid-table, outscoring many 14–32B models.
  • Diminishing Returns: Increasing size beyond ~14B–32B offered marginal gains in value accuracy ($F1_{micro}$) but did not guarantee better structural validity ($DOC_{micro}$).
• Error Distribution:
  • Small Models (<7B): Prone to Missing Key (MK) errors (structural failures).
  • Large Models (>14B): Structural failures (MK) largely disappear, but Wrong Value (WV) errors become the dominant bottleneck. Even the largest models (70B) produced incorrect values in roughly 50% of test runs where the structure was correct.

C. Error Analysis

• Semantic Errors are Persistent: While larger models fix structural issues, they do not automatically fix semantic hallucinations. The paper notes that Wrong Value (WV) errors account for 97–98% of remaining errors in larger models.
• Prompt-Model Interaction: The variance in $DOC_{micro}$ scores increased with model size for some families (e.g., Deepseek-R1), suggesting that larger models are more sensitive to the specific combination of prompt and architecture.

4. Key Contributions

1. LLMStructBench Dataset: A high-quality, manually verified benchmark of 995 diverse scenarios specifically designed for JSON extraction from natural language.
2. Novel Metrics: Introduction of a composite scoring system that separates and weights document-level validity (can the parser read it?) from semantic accuracy (is the data right?).
3. Empirical Insights:
   • Demonstrated that prompting strategy is a more significant lever for performance than model scale for structured extraction.
   • Identified that Wrong Value errors are the primary bottleneck for state-of-the-art models, suggesting that future improvements require better semantic validation or fine-tuning rather than just scaling parameters.
   • Showed that smaller, well-tuned models (e.g., Gemma3-12B) can compete with or exceed massive proprietary models in specific structured tasks.

5. Significance and Implications

• For Practitioners: The paper provides actionable guidelines for deploying LLMs in ETL pipelines. It suggests that investing in the right prompting strategy (e.g., PJ+ for reliability, P for accuracy) is more cost-effective than simply purchasing larger models.
• For Open-Source Ecosystem: It validates the viability of open-weight models for production-grade data extraction, showing that models like Gemma3 and Qwen3 can achieve performance comparable to GPT-4o when paired with the correct prompting techniques.
• Future Research: Highlights the need for research into post-generation semantic validation and fine-tuning for value fidelity, as scaling alone has reached diminishing returns for reducing semantic errors. The benchmark serves as a standard for evaluating future models in the context of Extract, Transform, Load (ETL) applications.