🤖 AI

Identifying AI Web Scrapers Using Canary Tokens

This paper proposes a novel technique using dynamic canary tokens to automatically and accurately identify which web scrapers feed data to specific large language models, enabling website owners to better control unwanted scraping without relying on voluntary corporate disclosures.

Original authors: Steven Seiden, Triss Ren, Caroline Zhang, Taein Kim, Enze Liu, Emily Wenger

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Steven Seiden, Triss Ren, Caroline Zhang, Taein Kim, Enze Liu, Emily Wenger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a baker who wants to know exactly which delivery trucks are bringing ingredients to a giant, hungry robot chef. You suspect the robot is using these ingredients to cook up new recipes (answers) for customers. But the trucks are sneaky: they wear disguises, they might be borrowing trucks from other companies, and the robot chef doesn't always tell you where it got its food.

This paper is about a clever trick the researchers used to catch these sneaky trucks in the act. They call their trick "Canary Tokens."

The Setup: The "Trap" Websites

Instead of trying to spy on the robot directly, the researchers built 20 fake websites. Think of these websites as customized bait.

The Bait: They created a template for a website (like a fake person's profile or a fake company page).
The Canary Token: Before the website loaded, the researchers gave every single visitor a unique, invisible "stamp" or "token."
- If a truck wearing a "Google" hat visited, the website might say: "My favorite color is Blue."
- If a truck wearing a "Bing" hat visited, the website might say: "My favorite color is Purple."
- If a truck wearing a "Chrome" hat visited, the website might say: "My favorite color is Green."

The researchers kept a secret list: "Blue" = Google Truck, "Purple" = Bing Truck, etc.

The Test: Asking the Robot Chef

After letting these websites sit online for two months (giving the trucks plenty of time to visit and memorize the content), the researchers asked 22 different AI chatbots (like ChatGPT, Claude, and others) questions about these fake websites.

They asked things like: "Tell me about this fake person. What is their favorite color?"

The Results: Who is Eating What?

The researchers looked at the answers the robots gave. If the robot said, "Their favorite color is Blue," the researchers knew for a fact: "The Google Truck delivered this information to the robot."

Here is what they discovered:

1. The "Disguise" Problem
Many of the AI robots didn't show up in their official uniforms.

The Official Uniform: Some robots admitted, "I am the OAI-SearchBot" (OpenAI's official truck).
The Disguise: Others showed up wearing generic clothes, like a regular web browser (e.g., "Chrome" or "Safari"). It's like a delivery driver pretending to be a tourist so no one stops them.
The Borrowed Truck: Surprisingly, many robots didn't visit the websites directly at all. Instead, they asked a search engine (like Google or Bing) to do the visiting for them. The robot would say, "I found this info on Google," even if the website owner never gave Google permission to share it with the robot.

2. The "Memory" Problem (Caching)
The researchers tried to stop the robots by taking the websites offline (turning off the lights).

The Result: Even after the websites were gone for a week, the robots still knew the "favorite colors."
The Metaphor: It's like the robot chef memorized the recipe from a cookbook it stole earlier. Even if you burn the cookbook later, the chef still remembers the ingredients. The robots were holding onto old data they had scraped previously.

3. The "Do Not Enter" Sign Problem (Robots.txt)
Website owners often put up a sign called robots.txt that says, "Please do not enter."

The Result: The researchers put up these signs, but the robots mostly ignored them. They kept visiting and memorizing the content.
The Exception: Only one robot, Duck.ai, actually respected the sign and stopped visiting. The rest kept going, either because they didn't see the sign or because they decided to ignore it.

The Big Takeaway

The paper proves that:

You can't always trust what a robot says about itself. They often hide their true identity or use other people's trucks (search engines) to get data.
Simple blocking doesn't work. Turning a website off or putting up a "Do Not Enter" sign doesn't guarantee the robot will forget what it already learned.
The "Canary Token" trick works. By serving unique, random facts to different visitors, you can trace exactly which "truck" fed information to which robot, even if the robot tries to hide.

In short, the researchers built a digital fingerprinting system to see exactly how AI chatbots are stealing (or borrowing) information from the web, revealing that the process is much messier and less transparent than the companies claim.

Technical Summary: Identifying AI Web Scrapers Using Canary Tokens

Problem Statement

Large Language Models (LLMs) rely on web-scraped data for both pre-training and query-time augmentation (e.g., Retrieval Augmented Generation or live web search). While this enhances model performance, it raises significant concerns regarding site stability, privacy, copyright, and ethics. Website owners often wish to restrict access to their content by specific scrapers, typically using mechanisms like the Robots Exclusion Protocol (robots.txt). However, the efficacy of these controls is compromised by a lack of reliable knowledge regarding which specific web scrapers (identified by User-Agent strings) feed data to which AI chatbots.

Existing methods for identifying AI-related scrapers rely on voluntary corporate disclosure, one-off researcher experiments, or crowd-sourced reports. These methods are neither reliable nor scalable. Furthermore, it remains unclear whether AI chatbots retrieve content live, rely on cached data from third-party search engines, or spoof their identities to bypass access controls. This opacity prevents site owners from effectively opting out of the AI ecosystem.

Methodology

The authors propose a novel technique using Canary Tokens to accurately and automatically infer the relationship between web scrapers and AI chatbots. The approach involves a three-stage pipeline:

Deployment of Canary Token Websites:
- The authors deployed 20 unique websites mimicking common content types (e.g., personal portfolios, company sites).
- Each website template contained placeholders for canary tokens (unique, randomized strings such as names, locations, or numbers).
- Dynamic Serving: When a scraper accessed a site, the system identified the visitor based on a unique combination of User-Agent and Autonomous System Number (ASN).
- Token Assignment: Each unique scraper was served a distinct set of canary tokens embedded in the page content. If a scraper visited multiple times, it received the same tokens; different scrapers received different tokens.
AI Chatbot Querying:
- After a two-month propagation period, the authors queried 22 production AI chatbots (including ChatGPT, Claude, Copilot, Gemini, etc.) using specific prompts.
- Prompts were designed to elicit responses containing information about the deployed websites, specifically asking for details that would correspond to the canary tokens (e.g., "What is the hometown of Alice?").
- A follow-up query was used to check for variant information or discrepancies.
Scraper Inference:
- The authors extracted canary tokens from the chatbot responses.
- Matching Algorithm: A match was established if a chatbot response contained a token uniquely assigned to a specific scraper.
- Thresholds: To mitigate false positives (e.g., hallucinations or coincidental matches), a scraper was only inferred to feed a chatbot if it was associated with at least 2 tokens ( $t=2$ ) or observed in 1 distinct website interaction ( $w=1$ ) across the dataset.
- Ambiguous matches (e.g., subset tokens or numerical coincidences) were discarded to ensure reliability.

Experimental Setup

Subjects: 22 production AI chatbots from major providers (OpenAI, Anthropic, Google, Microsoft, etc.) and search engine providers.
Conditions: The study ran over several months under three accessibility conditions:
1. Fully Accessible: All sites online (Baseline).
2. Offline: 10 sites taken down to test caching.
3. Restricted: 10 sites remained online but blocked via robots.txt to test protocol adherence.
Metrics: The presence of specific User-Agent strings in the inferred data source and the persistence of content after blocking.

Key Results

1. Mapping Scrapers to Chatbots (RQ1)

The study successfully identified the User-Agents used by 18 of the 22 chatbots.

Declared vs. Actual: While some chatbots used their self-declared User-Agents (e.g., OAI-SearchBot for ChatGPT), many utilized Generic Browser Agents (e.g., Chrome, Safari) or Third-Party Search Agents (e.g., Googlebot, Bingbot, Bravebot).
Identity Rotation: Some systems, notably Kimi, appeared to rotate through a large list of User-Agents, making identification via static strings difficult.
Third-Party Reliance: Many chatbots returned content associated with search engine crawlers (e.g., Googlebot) rather than their own direct crawlers, suggesting they rely on search engine indexes rather than direct scraping.

2. Caching and Persistence (RQ2)

Caching is Common: When websites were taken offline, the majority of chatbots continued to return content associated with those sites.
Search Engine Caching: Chatbots relying on third-party search agents (Googlebot, Bingbot) showed high persistence, consistent with search engine indexing.
Direct Caching: Even chatbots using generic browser agents (e.g., Grok, Solar) continued to serve content after the source was offline, indicating they maintain their own caches.

3. Efficacy of Blocking (RQ3)

Ineffectiveness of Standard Controls: Taking sites offline or implementing robots.txt restrictions largely failed to prevent chatbots from returning previously scraped content.
Robots.txt Ignored: In the robots.txt condition, many chatbots continued to return content, suggesting they either ignore the protocol or rely on cached data that predates the restriction.
Exception: Duck.ai was the only system observed to stop returning content in both the offline and robots.txt conditions, suggesting it actively evaluates availability and protocol directives.

Significance and Claims

The paper claims to provide the first critical look at the infrastructure underpinning real-time content retrieval in AI chatbots. Its primary contributions are:

A Novel Measurement Technique: The canary token approach offers a reliable, automated method for unprivileged third parties to infer which scrapers feed which LLMs without relying on voluntary disclosure.
Revealing Opacity: The study exposes that the data supply chain is more complex than public disclosures suggest. AI chatbots often rely on third-party search engines or generic browser agents, and they frequently cache content, rendering standard blocking mechanisms like robots.txt ineffective for removing data from AI outputs.
Implications for Control: The findings suggest that website owners wishing to opt out of AI scraping cannot rely solely on robots.txt or taking sites offline. The paper highlights the need for new mechanisms to signal preferences and enforce them across complex, multi-party data supply chains.

The authors conclude that while their approach enables better identification of scrapers, the current landscape of AI web scraping remains largely opaque and resistant to traditional access control measures.