Coverage-Aware Web Crawling for Domain-Specific Supplier Discovery via a Web--Knowledge--Web Pipeline

This paper proposes a Coverage-Aware Web Crawling framework utilizing a Web--Knowledge--Web pipeline and ecological species-richness estimators to iteratively discover and map under-represented SME suppliers in specialized sectors, achieving superior precision and efficiency compared to baseline methods in the semiconductor equipment manufacturing industry.

The Gist

Imagine you are trying to map a hidden city of small businesses (like a specific type of factory) to make sure your supply chain doesn't break. You know the city exists, but the official maps (databases) are incomplete. They show the big, famous landmarks but miss the tiny alleyways where the real work happens.

This paper proposes a new, smarter way to explore this city. Instead of just walking down every street randomly, they built a "Web-Knowledge-Web" (W→K→W) pipeline. Think of it as a self-correcting treasure hunt.

Here is how it works, broken down into simple steps:

1. The Problem: The "Blind Search"

Traditional web crawlers are like a person walking through a city with a blindfold, grabbing every flyer they see. They waste time on irrelevant shops (like a bakery when you're looking for a chip manufacturer) and miss hidden gems because they don't know what they are looking for. Existing databases are like old, incomplete maps that miss 80% of the small suppliers.

2. The Solution: The "Smart Detective" Loop

The authors created a three-step loop that acts like a detective who learns from every clue they find.

Step A: Web → Knowledge (The "Note-Taker")

The system starts by visiting a few known websites (like industry directories). It uses a powerful AI (a Large Language Model) to read these pages and pull out specific facts: Who makes what? Who supplies whom? Where are they located?

• The Analogy: Imagine a detective reading a newspaper and writing down names and connections in a notebook. But this notebook isn't just a list; it's a Knowledge Graph—a web of sticky notes connected by strings. If "Company A" makes "Part B," there is a string connecting them.
• The Trick: To make sure the AI doesn't get confused, the researchers gave it a "cheat sheet" (a glossary of industry terms) and specific rules (e.g., "If a company makes a product, that's a 'produces' link, not a 'supplies' link"). This ensures the notes are perfectly organized.

Step B: Knowledge → Web (The "Gap Finder")

This is the magic part. The system looks at its notebook (the Knowledge Graph) and asks: "Wait, if Company A supplies Company B, and Company B is in the 'Vacuum Systems' sector, where is the company that supplies Company A? It's missing!"

• The Analogy: Imagine looking at a puzzle. You see a gap where a piece should be. Instead of randomly grabbing puzzle pieces from the box, the system looks at the shape of the hole and says, "I need a piece that looks like this."
• The Action: It uses these "holes" to generate new search queries. It goes back to the web and specifically hunts for the missing pieces (the under-represented suppliers).

Step C: Coverage Estimation (The "Population Counter")

How do you know when to stop searching? You don't want to search forever, but you don't want to stop too early.

• The Analogy: The researchers borrowed a method from ecologists who count animals in a forest. If you catch 100 fish, and 50 of them are ones you've never seen before (singletons), you know the lake is huge. If you catch 100 fish and 90 are repeats, you've probably found most of the fish.
• The Result: They use this math to estimate how many total suppliers exist and tell you, "You've found about 16% of the total population. Keep going until you hit 85%."

3. The Results: Efficiency Wins

They tested this on the semiconductor equipment industry (making the machines that build computer chips).

• The Old Way (Baselines): Tried to find companies by crawling 213 pages. Found about 18–20 real companies, but also a lot of junk (low precision).
• The New Way (W→K→W): Crawled only 144 pages (32% less effort!).
• The Outcome: It found the same number of real companies but with much higher accuracy. It built a map of 664 entities (companies, products, locations) with zero logical errors in how they were connected.

Why This Matters

This isn't just about finding more companies; it's about resilience.

• Real World Impact: When supply chains break (like during the chip shortage), big companies often don't know who the small, hidden suppliers are.
• The Benefit: This system helps governments and companies find those "invisible" small businesses, ensuring that if one supplier fails, there are others ready to step in.

Summary Metaphor

Imagine trying to find every hidden coffee shop in a city.

• Old Method: Drive around randomly, stopping at every building. You waste gas and miss the shops tucked behind warehouses.
• New Method: You drive a few blocks, draw a map of the shops you found, notice a whole neighborhood has no coffee shops on the map, and then drive specifically to that neighborhood to find the missing ones. You stop when your map looks complete.

This paper proves that by using a "smart map" to guide your search, you can find more hidden treasures with less effort.

Technical Summary

1. Problem Statement

Modern supply chain resilience relies on visibility into the entire supplier ecosystem, including Small and Medium-sized Enterprises (SMEs) that often operate below the radar of major commercial databases (e.g., Dun & Bradstreet). Existing databases suffer from significant coverage gaps, particularly for sub-tier suppliers and firms in niche emerging markets.

Key Challenges:

• Incompleteness: General-purpose web crawlers lack domain awareness, cannot estimate how much of the target population remains undiscovered, and waste resources on irrelevant pages.
• Lack of Structure: Focused crawlers improve relevance but fail to maintain structured representations of discovered entities or reason about coverage completeness.
• Stopping Criteria: There is no principled method to determine when a crawling campaign has achieved sufficient coverage of a specific domain.

2. Methodology: The Web–Knowledge–Web (W→K→W) Pipeline

The authors propose an iterative framework that closes the loop between data acquisition and structured knowledge representation. The pipeline operates in three phases per iteration:

Phase 1: Web → Knowledge (Extraction)

• Crawling: An ethical crawler fetches pages from seed URLs and outbound links, adhering to robots.txt, rate limits (1.5s delay), and using a self-identifying user-agent.
• Text Processing: Raw HTML is cleaned using Trafilatura (with BeautifulSoup fallback) to isolate main content.
• Domain-Adapted Few-Shot Extraction:
  • Model: Uses GPT-4o-mini for joint Named Entity Recognition (NER) and Relation Extraction.
  • Prompt Engineering: The prompt is augmented with a curated domain glossary (~80 semiconductor terms), formal type constraints (defining valid source/target types for relations), and few-shot examples to disambiguate complex cases (e.g., distinguishing "Company supplies Product" from "Company supplies Company").
  • Output: Structured triples populating a heterogeneous Knowledge Graph (KG) with nodes (Companies, Products, Sectors, Locations) and edges (e.g., supplies_to, produces, located_in).
  • Post-Processing: Applies schema-level type filtering and Entity Resolution (Jaro-Winkler similarity) to merge duplicates.

Phase 2: Knowledge → Web (Gap-Guided Seed Generation)

The KG is analyzed to identify "structural holes" (missing entities or relations) to guide the next crawl cycle:

• Structural Hole Detection: Identifies sectors with low company degrees, missing intermediate suppliers in supply chains, or geographic gaps.
• Query Expansion: Generates targeted search queries by combining neighboring entity names, sector keywords, and location constraints.
• Link Prediction: Uses a DistMult embedding model trained on the current KG to predict missing supplies_to links, serving as a supplementary signal for gap detection.
• Seed Generation: These gaps are translated into new seed URLs and search queries for the next iteration.

Phase 3: Coverage Estimation

• Ecological Analogy: Adapts Chao1 and ACE species-richness estimators from ecology to web crawling.
• Mechanism: Treats each crawl iteration as a "capture occasion." It counts singletons (entities seen once) and doubletons (entities seen twice) to estimate the total population size ($|E^*|$).
• Stopping Criterion: The process stops when the estimated coverage ratio ($\hat{C} = |\hat{E}| / \hat{|E^*|}$) exceeds a threshold (e.g., 85%) or the marginal discovery rate drops.

3. Key Contributions

1. Novel Pipeline: A unified W→K→W iterative framework that combines focused crawling with KG construction for domain-specific discovery.
2. Domain-Adapted Extraction: A few-shot LLM extraction strategy integrating glossaries and type constraints, achieving 100% relation type-consistency and 66.2% precision without labeled training data.
3. Coverage Estimation Framework: The first adaptation of ecological richness estimators (Chao1) to web-entity populations, providing calibrated completeness scores and principled stopping criteria.
4. Empirical Validation: Demonstrated superior performance in the semiconductor equipment sector (NAICS 333242), achieving higher precision and F1 scores with significantly fewer resources than baselines.

4. Experimental Results

Domain: Semiconductor Equipment Manufacturing (NAICS 333242).
Ground Truth: A curated list of 195 companies.
Baselines: BFS Crawler, Focused Crawler, and a non-iterative W→K approach (all using a 213-page budget).

Metric	BFS Crawler	Focused Crawler	W→K (No Feedback)	W→K→W (Ours)
Pages Crawled	213	213	213	144 (32% fewer)
Precision	0.125	0.076	0.125	0.165
Recall	0.103	0.092	0.103	0.097
F1 Score	0.113	0.084	0.113	0.123
Unique Companies	160	236	160	115

Key Findings:

• Efficiency: W→K→W achieved the highest F1 score (0.123) and Precision (0.165) while using 32% fewer pages than the baselines.
• Quality: The pipeline built a KG with 664 entities and 542 relations, all of which satisfied schema constraints (100% type-consistency).
• Coverage Estimation: The Chao1 estimator provided a lower-bound estimate of the total population. While it slightly overestimated coverage relative to the curated ground truth (due to the inclusion of non-ground-truth entities), it successfully tracked the diminishing returns of discovery.
• Scalability: A Michaelis-Menten projection suggests diminishing returns after ~20 iterations, with the total discoverable entity population estimated around 1,136.
• Cost: The LLM API cost for the entire experiment (144 pages) was approximately $0.04 USD, demonstrating high economic viability.

5. Significance and Implications

• Supply Chain Resilience: The framework addresses critical needs in sectors like semiconductors by uncovering hidden sub-tier suppliers that traditional databases miss, aiding in risk mitigation and procurement strategy.
• Methodological Advancement: It shifts the paradigm from "crawling to retrieve documents" to "crawling to fill knowledge gaps," using the KG itself as a guide for future data acquisition.
• Scalable Domain Adaptation: The study proves that lightweight domain adaptation (glossaries + few-shot examples) can drastically improve LLM extraction quality without the need for expensive fine-tuning or large labeled datasets.
• Ethical Crawling: The system emphasizes responsible data collection, strictly adhering to robots.txt, rate limiting, and avoiding access control bypass, setting a standard for academic and industrial web scraping.

In conclusion, the W→K→W pipeline offers a robust, cost-effective, and measurable approach to discovering domain-specific entities, transforming web crawling from a passive retrieval task into an active, hypothesis-driven exploration of the digital landscape.