💬 NLP

ScrapeGraphAI-100k: Dataset for Schema-Constrained LLM Generation

This paper introduces ScrapeGraphAI-100k, a large-scale, real-world dataset of 93,695 schema-constrained extraction events derived from practitioner telemetry, which enables the training and benchmarking of large language models on structured generation tasks where prior synthetic or text-only corpora were insufficient.

Original authors: William Brach, Francesco Zuppichini, Marco Vinciguerra, Lorenzo Padoan

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: William Brach, Francesco Zuppichini, Marco Vinciguerra, Lorenzo Padoan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but sometimes overly chatty, robot assistant (a Large Language Model). You ask it to read a messy webpage and pull out specific details, like the price of a shoe or the author of an article. You want the answer in a perfect, neat box (a JSON schema), not a rambling paragraph.

The problem is that while we have lots of data on how these robots talk, we don't have enough data on how they extract information from real websites when you give them strict rules. Most existing datasets are like practice drills made up in a lab—they don't look like the messy, real-world websites people actually use.

Enter "ScrapeGraphAI-100k."

Think of this paper as the introduction of a massive, real-world training manual for these robots. Here is the breakdown in simple terms:

1. The "Real-World Gym"

The authors didn't make up fake websites. They collected 93,695 real examples of people using their software to scrape data from the internet.

The Source: They asked 9 million users of their open-source tool if they could anonymously share what they were doing. About 93,000 of those interactions were kept after cleaning up duplicates and noise.
The "Four-Tuple": Every example in this dataset is a complete story:
1. The Content: The webpage text (converted to Markdown, like a clean document).
2. The Request: The human's prompt (e.g., "Get the price and size").
3. The Rules: The strict "box" (JSON schema) the robot had to fill.
4. The Result: What the robot actually wrote back.

2. The "Complexity Trap"

The researchers analyzed these examples and found a fascinating pattern, like a speed limit sign for robots.

Simple Tasks: If the rules (schema) are simple (like asking for just a name and price), the robots are great at following them.
The Cliff: As the rules get more complex (more nested folders, more fields, more "if/then" logic), the robots start to crash.
The Finding: There is a sharp "failure threshold." Once a rule set gets too deep or has too many keys, the success rate drops like a stone. It's like asking a human to fill out a form with 500 fields; eventually, they just give up or make mistakes.

3. The "Student vs. Teacher" Experiment

To prove this dataset is useful, the authors ran a small experiment, like a science fair project:

The Teacher: A very smart, expensive model (GPT-5-nano) that generated perfect answers for the dataset.
The Student: A tiny, cheap model (1.7 billion parameters) that was "taught" using this new dataset.
The Result: The tiny student learned so well from the real-world examples that it started acting almost as good as a much larger, more expensive model (30 billion parameters) when it came to following the strict rules.
The Catch: The student was great at drawing the outline of the box correctly (structural accuracy), but sometimes still struggled to get the exact words inside the box perfect (semantic correctness).

4. What's Inside the Box?

Languages: It's mostly English and Traditional Chinese (about 88% of the data), covering 18,000 different types of "rule sets."
What's Missing: The paper admits they didn't include the raw, messy HTML code of the websites (just the clean text version) and they don't have a "human-verified" gold standard for every single answer yet. They are saving that for a future update (Version 2.0).

The Bottom Line

This paper says: "We built a massive library of real-world examples where robots tried to follow strict rules to extract data. We found that robots get confused when rules get too complex, but if you train a small robot on this real data, it can learn to follow the rules almost as well as a giant robot."

It's a tool for researchers to build better, smaller, and more efficient AI tools that can read websites without getting lost in the details.

Technical Summary: ScrapeGraphAI-100k

Problem Statement

Modern large language models (LLMs) require the ability to produce output conforming to specified JSON schemas for critical tasks such as tool use, structured extraction, and knowledge base construction. Despite the centrality of this capability, the research community lacks a public, large-scale dataset that captures the full "extraction four-tuple": unstructured web content, natural-language prompts, target schemas, and resulting model outputs. Existing resources suffer from significant gaps:

Pre-training corpora (e.g., C4, FineWeb) provide volume but strip structural signals like HTML tags and hierarchical metadata during preprocessing.
Content extraction benchmarks (e.g., WebMainBench) focus on isolating main content from HTML to Markdown but do not address mapping that content to structured schemas.
Structured extraction datasets (e.g., SWDE, OpenCeres) are often small-scale, outdated, restricted to English, or operate on clean text rather than raw web inputs.
Generative benchmarks often rely on synthetic prompts or clean text, failing to reflect the complexity of real-world production pipelines.

Consequently, it is difficult to study end-to-end extraction quality, failure modes, or the scaling of reliability with schema complexity under realistic input distributions.

Methodology

The authors introduce ScrapeGraphAI-100k, a dataset derived from real-world production usage of the ScrapeGraphAI library.

Data Collection

Source: The dataset originates from approximately 9 million opt-in telemetry events collected via PostHog during Q2–Q3 2025. Users explicitly consented to share anonymized execution data.
Content: Each event captures the full extraction lifecycle: the input prompt, the target JSON schema, the source website content (converted to Markdown), the specific LLM used, and the generated response.
Privacy: The collection process excludes personal identifiers (PII) and user IDs.

Data Processing Pipeline

To transform the raw telemetry into a high-quality training resource, the authors applied a multi-stage cleaning and balancing pipeline:

Flattening and Validation: Events were flattened into a structured format. Every response was validated against its corresponding schema using the jsonschema-rs library. The overall validity rate in the raw corpus was 93%. Invalid responses are retained with a response_is_valid=false flag to enable failure mode analysis.
Noise Identification: Failures were categorized into upstream LLM errors (truncation, refusals), fetch-time failures (HTTP errors, CAPTCHAs), and non-textual content (image galleries).
Complexity Metrics: Five structural complexity metrics were computed for each schema: depth, key count, element count, cyclomatic complexity, and a composite score.
Balancing and Deduplication: The raw data exhibited extreme class imbalance (e.g., two schemas for Chinese e-commerce sites accounted for ~5M events). The authors grouped events by a SHA256 hash of the JSON schema and applied a sampling strategy retaining at most 5 examples per unique schema. To ensure diversity, selected examples were required to originate from distinct root domains.
Final Corpus: The process reduced the dataset from 9 million events to 93,695 balanced examples.

Dataset Characteristics

Scale: 93,695 extraction events.
Languages: 15 named languages (English and Traditional Chinese cover 88% of content) plus a long-tail "Other" category.
Schema Diversity: Spans over 18,000 unique schemas.
Complexity Distribution: Schema depth clusters around 5 levels, but key counts are heavily right-skewed (median 37, max 3,060). The composite complexity score shows a bimodal distribution: 90% of schemas are routine templates, while 10% represent complex edge cases (scores up to 7,935).
Response Sizes: Heavy-tailed distribution ranging from <1 KB to nearly 1 MB.

Key Contributions

Release of ScrapeGraphAI-100k: The first real-usage dataset exposing the full extraction four-tuple (content, prompt, schema, output) at scale, paired with per-example structural validation labels.
Schema-Centric Diagnostics: The provision of structural complexity metadata and validation labels that expose sharp failure thresholds as task complexity increases.
Case Study in Knowledge Distillation: A demonstration that a small language model (1.7B parameters) fine-tuned on this data can closely track the output distribution of a significantly larger teacher model (30B-A3B MoE) regarding schema conformance.

Results and Analysis

Failure Thresholds

The analysis reveals non-linear failure thresholds as schema complexity grows:

Depth: Validation rates drop from 98% to 20% when schema depth reaches $\ge$ 7.
Key Count: Rates decline to ~65% when key counts exceed 200.
Composite Score: Rates fall to ~55% when the complexity score exceeds 1,000.
These findings suggest LLMs handle incremental complexity gracefully until hitting structural limits, after which failures accelerate rapidly.

Fine-Tuning Case Study

The authors fine-tuned a Qwen3-1.7B model using QLoRA on a filtered subset of the data (25,244 training / 2,808 test examples). The training targets were regenerated by GPT-5-nano to ensure schema consistency (pseudo-labels).

Performance: The fine-tuned 1.7B model achieved a Key F1 of 0.8866, which is within 0.005 of the 30B-A3B reference model (0.8915).
Improvement: The fine-tuned model gained +25.7% in Overall BLEU over the base 1.7B and surpassed the 4B baseline on all key-based metrics.
Schema Compliance: Compliance rose from 0.7954 (base) to 0.9167 (fine-tuned), a 15% absolute increase.
Hallucination Reduction: Extra key generation dropped significantly from 2.64 to 0.33 per sample, outperforming the 30B-A3B baseline (0.5679).
Limitation: While structural accuracy improved, value extraction (Value Score) remained lower (0.4624) compared to the 30B-A3B (0.4797), indicating that identifying which fields to extract is easier than extracting the precise values.

Significance and Claims

The paper positions ScrapeGraphAI-100k as a foundational resource for bridging the gap between synthetic benchmarks and real-world extraction workloads.

Grounding in Reality: By pairing Markdown-converted page content with actual prompts and schemas used in production, the dataset enables training and benchmarking that prior synthetic or text-only corpora could not support.
Small Model Viability: The case study provides preliminary evidence that grounding schema-constrained generation in real practitioner workloads allows small models to approach the structural accuracy of frontier models through targeted fine-tuning.
Failure Analysis: The dataset's inclusion of invalid responses and complexity metadata allows researchers to systematically study extraction failure modes (syntactic, structural, and semantic) and predict failures based on schema complexity.

The authors note that the dataset is a proxy for typical LLM-based extraction workloads, currently skewed toward e-commerce schemas and GPT-4o-mini usage. They explicitly state that the dataset does not claim semantic correctness of individual extractions and that raw HTML/DOM analysis is deferred to version 2.0.