🤖 AI

Making Failure Safe: A Constrained, Verifiable Agent Framework for Open-Web Data Collection

This paper proposes a constrained, verifiable agent framework that replaces unreliable free-form LLM code generation with typed JSON collector configurations and static execution pipelines to achieve deterministic, low-cost, and reusable open-web data collection.

Original authors: Bo Chen

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bo Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you need to hire a robot to go out and gather specific information from thousands of different websites every day. You could just tell the robot, "Go get the news," and hope it figures out how to do it. But as the paper explains, this "free-for-all" approach is like sending a child into a library with no instructions; they might grab the wrong books, trip over chairs, or come back with a jumbled mess of pages.

This paper proposes a new way to build these data-gathering robots (called "agents") that makes failure safe, predictable, and easy to fix. Here is how it works, broken down into simple concepts:

1. The Problem: The "Wild West" of Web Scraping

Currently, if you ask an AI to write code to scrape a website, it often tries to write a brand-new script from scratch every time.

The Issue: Websites are messy and change often. If the AI guesses wrong about where a price tag is located on a page, the whole script breaks.
The Result: You get errors, broken data, or scripts that stop working the moment a website updates its layout. It's like trying to build a house by guessing where the bricks go every single time you lay one.

2. The Solution: The "LEGO Kit" Approach

Instead of letting the AI write free-form code (like writing a novel), the authors force the AI to fill out a structured form (like filling out a LEGO instruction sheet).

The Taxonomy (The 6 Types): The system first asks, "What kind of job is this?" It categorizes tasks into six specific types, like a menu:
1. Search: Finding links based on keywords.
2. List: Walking through pages of items (like a news archive).
3. Detail: Reading the full content of a single page.
4. API: Asking a computer directly for data (like ordering from a menu).
5. Interactive: Clicking buttons or typing on dynamic pages.
6. File: Downloading PDFs or Excel sheets.
- Analogy: Instead of telling a chef to "make dinner," you tell them, "You are making a soup," and they only use the tools and recipes for soup. This stops them from trying to bake a cake when you wanted soup.
The Constraints (The Safety Rails): The AI isn't allowed to invent new code. It must pick from a pre-approved library of "utility functions" (pre-built tools) and fill in the blanks on a template.
- Analogy: Think of it like a "Mad Libs" game where the AI can only fill in the specific blanks provided, rather than writing the whole story itself. This ensures the output is always in a format the computer can understand.

3. The Process: The "Test Drive" Loop

The framework doesn't just send the robot out immediately. It uses a strict "Generate → Check → Fix" loop:

Generate: The AI creates a configuration file (a JSON plan) based on the user's request.
Test Drive (Validation): Before running the full job, the system runs a tiny, cheap test on just a few pages.
The Quality Check (The Referee): A rule-based system (not an AI) checks the results. It asks: "Did we get the right fields? Is the data empty? Did we crash?"
- Crucial Point: If the test fails, the system doesn't just say "try again." It creates a specific "blacklist" of what not to do (e.g., "Do not look for the price in the footer").
Fix: The AI tries again, but this time it is forced to avoid the mistakes it just made.
Scale Up: Only when the test drive passes does the system run the full collection job.

4. The Results: Speed vs. Perfection

The authors tested this on 138 different data collection tasks. Here is what they found:

One-Shot Quality: If you just want to grab data once right now, other methods that let the AI write free code might get slightly better results immediately (about 70% success vs. 50% for this method).
The Trade-Off: However, the authors' method is much faster and much cheaper to run repeatedly.
- The Magic: Once the plan is made, the actual collection runs without using any AI at all. It just executes the pre-made plan.
- Analogy: Other methods are like hiring a human translator for every single sentence you read. This method is like hiring a translator once to write a dictionary, and then using that dictionary forever.
Reliability: When the system failed, it didn't fail silently. It produced a clear error report, allowing the "Fix" loop to correct it. In their tests, this feedback loop turned a failing system (0% pass rate) into a perfect one (100% pass rate).

Summary

This paper argues that we shouldn't try to make AI "perfect" at guessing how to scrape the web. Instead, we should constrain the AI to follow strict rules, use pre-built tools, and run a "test drive" before doing the real work.

By trading a little bit of initial perfection for a system that is verifiable, reusable, and cheap to run, this framework makes automated data collection reliable enough for real-world, scheduled use (like gathering news or government data every morning) without needing a human to fix the code every time a website changes.

Technical Summary: Making Failure Safe: A Constrained, Verifiable Agent Framework for Open-Web Data Collection

1. Problem Definition

The paper addresses the unreliability of using Large Language Models (LLMs) and agents for direct open-web data collection. While LLMs can generate scraper code from natural language, direct application suffers from structural brittleness due to heterogeneous web pages and the stochastic inconsistency of LLM code generation. Common failure modes include dependency errors, broken selectors, schema mismatches, and inconsistent output formats. Traditional pipelines rely on manual effort, while unconstrained agent approaches lack verifiability, making it difficult to audit, measure, or correct failures in a production environment.

The core challenge is to shift from "perfect agent reasoning" to a "verifiable collection pipeline" where every stage's inputs and outputs can be checked, measured, and audited.

2. Methodology

The authors propose a constrained, verifiable agent framework that replaces free-form code generation with the generation of typed JSON collector configurations. The system operates on a "generate–execute–check–fix" closed loop, decoupling the LLM's role (requirement understanding and configuration generation) from the execution engine (static Airflow DAGs).

2.1 Six-Type Collector Taxonomy

To enforce structural boundaries, the framework defines a taxonomy of six collector types, each with explicit functional boundaries and composition rules:

Search: Discovers candidate URLs via keywords (no content extraction).
List: Traverses paginated lists to find detail links (no deep content extraction).
Detail: Extracts structured fields from a single page.
API: Calls public APIs and maps response fields (no browser interaction).
Interactive: Executes clicks and inputs on dynamic pages (no bypassing auth/CAPTCHAs).
File: Downloads and parses public files (PDF, Excel).

2.2 Constrained Agent Framework

The framework consists of five modules operating under four constraint categories:

Requirement Understanding Agent: Converts natural language descriptions ( $d$ ) into a full task representation ( $R$ ) by proactively probing the target site (homepage, sitemap, sample pages) to extract source context ( $x$ ).
Collector Instantiation Agent: Generates JSON configurations ( $M$ $M$ ) using a "reference coding + template slot-filling" approach. Instead of writing code, the agent fills slots in pre-built templates constrained by:
- Task-type constraints: Ensuring the correct collector type is selected.
- Configuration-structure constraints: Enforcing JSON Schema compliance.
- Tool-reuse constraints: Reusing pre-built utility functions rather than generating new code.
Airflow Validation & Scale Execution: A static Airflow DAG executes the JSON configurations. Validation runs use small-sample parameters to shift errors to a low-cost pre-production stage, while scale runs use production parameters. This ensures zero LLM tokens are consumed during execution.
Quality Check Module: A rule-based, deterministic evaluator (no LLM) scores results across five dimensions: Structure, Content, Relevance, Attachment, and Execution. It produces a composite score ( $S$ ) and an error report.
Feedback Correction Module: Converts quality reports into structured feedback constraints (e.g., failure blacklists, prohibited hypotheses). The instantiation agent regenerates the configuration under these new constraints, forming a closed loop.

3. Key Contributions

Six-Type Collector Taxonomy: A formal classification system with explicit boundaries and composition rules to standardize collection tasks.
Constrained Agent Framework: A system that shifts LLM output from free-form code to typed JSON configurations under a four-tier nested constraint hierarchy (type $\to$ template $\to$ utility $\to$ validation).
Verifiable Closed Loop: A "generate–execute–check–fix" mechanism using rule-based quality gating and structured feedback correction, enabling auditable failure signals.
Multi-Level Experimental Validation: Comprehensive testing covering requirement classification, collector instantiation, quality-cost trade-offs against runtime LLM baselines, and Airflow compatibility.

4. Experimental Results

The framework was evaluated on a dataset of 138 collection tasks (80 source-verified).

Requirement Classification: The system achieved a Macro-F1 of 0.7793 in classifying collector types from descriptions, outperforming a rule-based keyword baseline (0.7562). However, the authors note that descriptions alone are insufficient for ambiguous tasks (e.g., "detail" vs. "list"), necessitating proactive probing.
Collector Instantiation: The constrained framework achieved an average quality score of 0.5369 and a pass rate of 55.07%, significantly outperforming unconstrained LLM generation (0.4876 quality, 18.84% type match) and rule-based baselines.
Quality-Cost Trade-off (80 Source-Verified Tasks):
- Runtime LLM Baselines (e.g., Schema-constrained LLM, Crawl4AI): Achieved higher one-shot quality (0.7051 and 0.6996) and pass rates (81.25% and 80.00%) but incurred high costs (3,400–4,200 tokens per run) and longer wall-clock times (14–16s).
- Proposed Framework: Achieved lower one-shot quality (0.5631) and pass rate (50.00%) but utilized zero execution-stage LLM tokens and achieved the lowest wall-clock time (5.052s).
Feedback Correction: When applied to intentionally flawed configurations, feedback correction improved the average final quality from 0.5087 to 0.9007 and the pass rate from 0.72% to 100%, demonstrating the efficacy of the repair mechanism for known defect types.
Engineering Verification: All 138 generated configurations were grounded in shared templates (100% reuse rate) and successfully executed within Airflow 2.8.0.

5. Significance and Claims

The paper positions the framework not as a general replacement for runtime LLM extraction, but as a reusable, low-cost, and verifiable execution path suited for repeated, scheduled data collection.

Philosophy of "Making Failure Safe": The authors argue that controlled, explicit failure (via type constraints, schema validation, and rule-based gating) is preferable to uncontrolled, silent success. Failures become auditable, traceable signals rather than silent data corruption.
Trade-off Acknowledgement: The paper modestly acknowledges that while runtime LLM baselines offer superior one-shot quality, the proposed framework offers a deterministic, amortized cost model. By shifting LLM costs to a one-time configuration generation step, the framework becomes economically viable for high-frequency, scheduled collection tasks where execution speed and token cost are critical.
Limitations: The authors explicitly state limitations, including the use of constructed requirements rather than raw user requests, the lack of long-horizon significance tests, and the sensitivity of the system to complex interactive pages and authentication boundaries. The framework is designed for compliant, publicly accessible data scenarios.