Technical Summary: WCXB – A Multi-Type Web Content Extraction Benchmark

Problem Statement

Web content extraction—the process of isolating a page's main content from surrounding boilerplate (navigation, ads, footers)—is a critical prerequisite for search indexing, retrieval-augmented generation (RAG), NLP dataset construction, and large language model (LLM) training. Despite its importance, progress in this field has been constrained by the limitations of existing evaluation benchmarks. Current datasets are typically small (100–800 pages), restricted almost exclusively to news articles, or based on HTML structures from over a decade ago.

Crucially, existing benchmarks fail to account for the structural diversity of the modern web. A web crawler encounters product pages with structured specifications, forums with nested user posts, documentation with code blocks, and service pages with distributed content sections. Heuristics optimized for article extraction fail predictably on these other types, yet no existing benchmark evaluates extraction performance across these distinct structural categories. This "article-only" bias obscures significant performance gaps and blind spots in current extraction systems.

Methodology

The WCXB Dataset

The authors introduce the Web Content Extraction Benchmark (WCXB), a dataset comprising 2,008 web pages from 1,613 unique domains. The dataset is split into a 1,497-page development set and a 511-page held-out test set, with matched page type distributions to ensure generalization validation.

Page Type Taxonomy:
Unlike prior work that classifies pages by semantic content, WCXB defines seven page types based on HTML structural characteristics that dictate extraction strategy:

1. Article: Single content container with sequential paragraphs.
2. Forum: Multiple user posts with voting controls and metadata.
3. Product: Structured data (JSON-LD), specifications, and pricing.
4. Collection: Product grids with filter panels and pagination.
5. Listing: Repeated card elements with summaries.
6. Documentation: Code blocks, sidebar navigation, and versioned content.
7. Service: Multi-section layouts (hero, features, testimonials) distributed across independent <section> elements.

Annotation Pipeline

Ground truth annotations were generated through a rigorous five-stage pipeline to ensure high quality:

1. LLM-assisted drafting: An LLM (Claude) generates initial extractions, excluding navigation and boilerplate.
2. Automated verification: Scripts check encoding, JSON schema validity, and content bounds.
3. Frontier model review: Four independent passes by Claude Opus agents review content completeness, boundaries, metadata, and consistency.
4. Snippet and quality verification: Automated scripts verify that required "with" snippets appear in the extraction and "without" boilerplate snippets are absent.
5. Human review: A human annotator adjudicates disagreements and performs final quality assurance.

Annotations include the main content as plain text, along with specific "with" (must-include) and "without" (must-exclude) text snippets to evaluate precision and recall at the phrase level.

Evaluation Systems and Metrics

The paper evaluates 13 extraction systems: 11 heuristic (including rule-based and hybrid rule+ML systems like rs-trafilatura, Resiliparse, Trafilatura) and 2 neural systems (MinerU-HTML and ReaderLM-v2).

Metrics:

• Word-level F1: Calculated on tokenized word bags (lowercased, non-alphanumeric split) to measure content overlap, avoiding penalties for whitespace or paragraph ordering differences.
• Snippet-based metrics: "With recall" (content completeness) and "Without inclusion" (boilerplate contamination).
• Speed: Measured in milliseconds per page.

Key Results

Overall Performance

On the development set, rs-trafilatura (a hybrid rule+ML system) achieved the highest overall F1 score of 0.859, processing pages in 44ms. The neural system MinerU-HTML followed closely at 0.827, while ReaderLM-v2 (1.5B parameters) underperformed several heuristic systems with an F1 of 0.741.

The "Article vs. Structure" Gap

The most significant finding is the divergence in performance based on page type:

• Articles: Performance is saturated. All top systems converge with F1 scores between 0.871 and 0.932. The gap between the best heuristic and best neural system is negligible (0.4 points).
• Structured Types: Performance diverges sharply.
  • Forums: F1 ranges from 0.520 to 0.794 (27.4-point spread).
  • Collections: F1 ranges from 0.415 to 0.713 (29.8-point spread).
  • Products: F1 ranges from 0.464 to 0.670 (20.6-point spread).
  • Service Pages: F1 ranges from 0.708 to 0.843.

Generalization and Cross-Benchmark Validation

• Generalization: When evaluated on the held-out test set, all systems showed a uniform upward shift in performance (likely due to cleaner test annotations and a slightly higher proportion of articles), but the relative ranking of top systems remained stable.
• Cross-Benchmark: On the article-only ScrapingHub benchmark, top systems achieved F1 scores near 0.96. However, on WCXB, the overall F1 dropped to 0.859. The authors note that the 7.3-point gap between article-only performance and multi-type performance quantifies the "blind spot" of existing benchmarks.

Neural vs. Heuristic

Contrary to the assumption that neural models generalize better, the results show that neural systems inherit the same article bias as heuristic systems. MinerU-HTML trailed the best heuristic on 6 of 7 page types, and ReaderLM-v2 underperformed three heuristic systems overall. Larger model size did not correlate with better performance on structured types.

Significance and Claims

The paper claims that article extraction is largely a solved problem, with performance ceilings reached for current architectures. Consequently, future progress in web content extraction must come from addressing structured page types (forums, products, collections, etc.), which remain open problems with 20–30 point performance gaps between systems.

The authors assert that:

1. Existing benchmarks are insufficient: They mask critical failures on non-article pages, leading to a false sense of progress.
2. Neural approaches have not solved diversity: Current LLM-based extractors do not generalize better to structural variations and often underperform faster heuristic methods.
3. Structural taxonomy is essential: Extraction strategies must be driven by HTML structural properties (e.g., JSON-LD handling, multi-section merging) rather than semantic content categories.

The WCXB dataset, released under CC-BY-4.0, provides the necessary infrastructure (diverse page types, ground truth, and baselines) to drive research toward type-aware extraction systems that can handle the full complexity of the modern web.