💬 NLP

LiveWeb-IE: A Benchmark For Online Web Information Extraction

This paper introduces LiveWeb-IE, a novel benchmark for evaluating web information extraction on live, evolving websites, and proposes Visual Grounding Scraper (VGS), a multi-stage agentic framework that mimics human visual cognition to robustly extract data from dynamic web pages.

Original authors: Seungbin Yang, Jihwan Kim, Jaemin Choi, Dongjin Kim, Soyoung Yang, ChaeHun Park, Jaegul Choo

Published 2026-03-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Seungbin Yang, Jihwan Kim, Jaemin Choi, Dongjin Kim, Soyoung Yang, ChaeHun Park, Jaegul Choo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

🌐 The Problem: The "Frozen Photo" vs. The "Live Stream"

Imagine you are trying to teach a robot how to find specific items in a library.

The Old Way (Offline Benchmarks):
Previously, researchers taught robots using a frozen photograph of a library shelf taken five years ago. They asked the robot, "Find the red book on the second shelf." The robot learned the rules perfectly based on that photo.

The Catch: In the real world, libraries are messy and constantly changing. Books get moved, shelves are rearranged, and new books arrive. If you take that same robot to a real, live library today, it will get confused because the "red book" might now be on the fourth shelf, or the shelf might look completely different. The robot fails because it was trained on a static snapshot, not the living, breathing reality.

The New Solution (LIVEWEB-IE):
This paper introduces LIVEWEB-IE, a new "training ground" where robots must find information on live websites right now. It's like moving the test from a photo to a live video stream. The robot has to navigate a website that might have changed its layout since the test started.

🕵️‍♂️ The New Method: VGS (The "Visual Detective")

The authors also built a new robot brain called VGS (Visual Grounding Scraper). To understand how it works, let's compare it to how a human finds a specific item in a messy room.

The Old Robot (HTML-Only):
Old methods tried to read the website like a giant, confusing instruction manual written in code (HTML).

Analogy: Imagine trying to find your friend's face in a crowd by reading a 50-page list of everyone's height, weight, and shoe size. It's slow, confusing, and if one person changes their shirt, the list becomes useless.

The VGS Robot (Visual Grounding):
VGS mimics how a human actually looks for something. It uses a four-step detective process:

The Question (Attribute Identification):
- Human: "I need the picture of the cat."
- VGS: Breaks the request down. "Okay, I'm looking for an image of a cat."
The Scan (Visual Grounding):
- Human: Instead of reading the whole room, you glance around and say, "The cat is definitely in the living room, not the kitchen."
- VGS: It looks at the screenshot of the webpage and ignores the code. It finds the specific area (region) where the cat picture is likely to be. It ignores the rest of the page noise.
The Pinpoint (Element Pinpointing):
- Human: Now that you are in the living room, you zoom in. "Ah, there it is! It's the photo on the coffee table, not the one on the wall."
- VGS: It puts a little "target" box around the exact image it needs, ignoring other pictures nearby.
The Map (XPath Synthesis):
- Human: You draw a map for your robot friend: "Go to the living room, look at the coffee table, and grab the photo."
- VGS: It writes a precise set of instructions (an XPath) to grab that specific image. Because it used the visual map first, these instructions are much more accurate and less likely to break if the website changes slightly.

🏆 Why This Matters

The paper tested this new "Visual Detective" (VGS) against other methods on the new "Live Website" test (LIVEWEB-IE).

The Result: The old methods (reading the code manual) struggled badly, especially when the websites were complex or had changed. They got lost in the noise.
The Winner: VGS performed significantly better. By looking at the visual picture first to narrow down the search, it could find the data even when the website's code was messy or had changed.

📝 The Big Takeaway

The internet is a living, changing thing. You can't build a reliable tool for extracting data by only looking at old, frozen snapshots.

LIVEWEB-IE is the new, tougher test that forces robots to deal with the real, changing web.
VGS is the new strategy that teaches robots to "look before they leap," using visual clues to find information just like humans do, rather than getting lost in the code.

This research paves the way for smarter, more robust tools that can automatically gather information from the internet without breaking every time a website updates its design.

1. Problem Statement

Web Information Extraction (WIE) aims to automatically extract structured data from web pages. While significant progress has been made, current evaluation paradigms suffer from a critical limitation: they rely on static HTML snapshots taken at a single point in time.

The Gap: Real-world websites are dynamic; their layouts and DOM structures evolve over time. Systems trained or evaluated on static snapshots often fail to generalize to live, evolving web pages.
The Consequence: Empirical analysis in the paper shows that LLM-based wrapper generation methods suffer an average F1 score degradation of over 15% when applied to the structurally evolved versions of the same websites compared to their static snapshots.
Existing Limitations: Current benchmarks (e.g., SWDE, WEIR) focus on text-only extraction from frozen HTML. They do not account for non-textual data (images, hyperlinks) or the visual context required to navigate complex, changing layouts.

2. Methodology

The paper introduces two core components: a new benchmark (LIVEWEB-IE) and a novel extraction framework (VGS).

A. LIVEWEB-IE Benchmark

LIVEWEB-IE is designed to evaluate WIE systems directly against live websites.

Construction: Built from 15 permission-granted websites across 8 diverse domains (Academic, Automotive, Sports, Library, Food, Book, E-commerce, Others).
Stability Strategy: To ensure valid ground truth despite layout changes, the benchmark curates queries for factually stable information (e.g., "2022 World Cup final results") that remains constant even as the visual layout and DOM structure change.
Task Complexity: Tasks are categorized into four types based on attribute count and value cardinality:
- Type I: Single attribute, single value.
- Type II: Multiple attributes, single values.
- Type III: Single attribute, list of values.
- Type IV: Multiple attributes, lists of values.
Data Modalities: Unlike previous benchmarks limited to text, LIVEWEB-IE includes Text, Images, and Hyperlinks.
Scale: 342 natural language queries, 97 unique attributes, and 46 distinct page groups.

B. Visual Grounding Scraper (VGS) Framework

To address the failure of HTML-only parsing on live sites, the authors propose VGS, a multi-stage agentic framework that mimics human cognitive processes by visually narrowing down content. It operates in four sequential stages:

Attribute Identification: An LLM decomposes a natural language query into a structured set of target attributes ( $\hat{A}$ ).
Visual Grounding: A Vision-Language Model (VLM) analyzes screenshots of the rendered webpage, divided into vertical regions. It identifies the specific region ( $r'$ ) containing the target attribute, effectively pruning the observation space and ignoring irrelevant DOM noise.
Element Pinpointing: Within the grounded region, the VLM uses Set-of-Mark (SoM) prompting (overlaying bounding boxes with IDs) to pinpoint the exact elements corresponding to the attribute values. This handles both text and non-textual elements (e.g., identifying an <img> tag for an image attribute).
XPath Synthesis: Using the localized HTML segments surrounding the pinpointed elements and the visual context, the VLM synthesizes robust, generalizable XPaths. These XPaths are then used to extract data from the live page.

3. Key Contributions

LIVEWEB-IE Benchmark: The first benchmark specifically designed for online WIE evaluation. It shifts the paradigm from static snapshots to live web evaluation, featuring diverse data modalities (text, image, link) and multi-dimensional complexity levels.
VGS Framework: A novel vision-grounded approach that moves away from raw HTML parsing. By simulating human visual scanning (Grounding $\to$ Pinpointing $\to$ Synthesis), VGS generates wrappers that are robust to structural drift.
Comprehensive Evaluation: Extensive experiments demonstrating that VGS outperforms state-of-the-art baselines (CoT, Reflexion, AutoScraper) on both the new LIVEWEB-IE benchmark and existing benchmarks (SWDE, Expanded SWDE).

4. Experimental Results

The authors evaluated VGS against baselines (CoT, Reflexion, AutoScraper) using various backbone models (Gemini-2.5-Flash, GPT-4o, Qwen-2.5, Gemma-3).

Performance on LIVEWEB-IE:
- VGS consistently outperformed all baselines across all task types.
- Type III & IV (Complex Tasks): The performance gap was most significant here. For example, using GPT-4o, VGS outperformed AutoScraper by 36.28% (Type III) and 34.58% (Type IV) in F1 score.
- Human Gap: While VGS is the best automated method, it still lags behind human performance (approx. 38-48% lower F1), highlighting the difficulty of the task.
Robustness to Structural Evolution:
- In a comparative study between static snapshots and live pages, baselines suffered an average F1 drop of 15.14%, whereas VGS only dropped by 8.15%, proving its superior robustness to layout changes.
Existing Benchmarks: VGS also achieved State-of-the-Art (SOTA) results on traditional benchmarks like SWDE and Expanded SWDE, validating that visual grounding is beneficial even for static data.
Efficiency: While VGS has higher initial latency on simple pages, it maintains superior performance on complex, content-heavy pages where iterative HTML-based baselines suffer from context length limits and noise.

5. Significance

Paradigm Shift: The paper argues that the field must move from "offline" (snapshot-based) to "online" (live) evaluation to build truly practical WIE systems.
Visual Grounding Necessity: It demonstrates that relying solely on HTML structure is insufficient for modern web scraping. Integrating visual context (screenshots) is crucial for filtering noise and identifying targets in evolving layouts.
Future Direction: By providing a challenging, ethically sourced, and diverse benchmark, LIVEWEB-IE sets a new standard for developing and evaluating robust, real-world web extraction agents.

In conclusion, LIVEWEB-IE and VGS represent a significant step forward in making web information extraction reliable in the face of the web's inherent dynamism, bridging the gap between academic benchmarks and real-world application.