Technical Summary: Is Misinformation More Open? A Study of robots.txt Gatekeeping on the Web

Problem Statement

Large Language Models (LLMs) increasingly rely on web crawling to update their training corpora, a practice that has intensified concerns regarding data quality and the ingestion of misinformation. While the Robots Exclusion Protocol (REP) serves as a voluntary mechanism for website owners to signal access preferences to automated agents, it remains unclear how different categories of content providers—specifically reputable news outlets versus misinformation sites—utilize robots.txt to manage AI crawlers. The core problem addressed is whether a disparity exists in how these two groups configure access controls, potentially creating an asymmetry where misinformation content remains more accessible to AI training data than reputable content.

Methodology

The study employs a longitudinal, multi-vantage point analysis of robots.txt configurations and active blocking behaviors.

• Dataset Construction: The authors curated a dataset of 4,079 websites based on classifications from Media Bias/Fact Check (MBFC). This included 3,369 "reputable" news websites (rated High/Very High factual reporting) and 710 "misinformation" websites (rated Low/Very Low factual reporting, labeled as Questionable Source or Conspiracy-Pseudoscience).
• AI Agent Identification: A comprehensive list of 63 unique AI-related user agents was compiled from four sources: Dark Visitors, Cloudflare Radar, prior academic studies (Liu et al.), and the ai.robots.txt repository.
• Data Collection:
  • Cross-Sectional Crawl: robots.txt files and homepage responses were retrieved from seven geographically distributed AWS vantage points (Germany, Sweden, USA, Brazil, Africa, Asia, Australia) in May 2025. To ensure robustness, the most permissive valid result was retained.
  • Longitudinal Analysis: Historical snapshots were retrieved from the Internet Archive at six 4-month intervals between September 2023 and May 2025 to track the evolution of exclusion directives.
  • Active Blocking Measurement: To detect blocking mechanisms beyond robots.txt, the authors performed sequential HTTP requests to homepages, varying only the User-Agent header to emulate specific AI crawlers (e.g., ClaudeBot, Anthropic-AI). Responses were compared against control crawls to identify active denial of service based on user-agent strings.
• Validation: The study distinguishes between sites that are non-functional and those that genuinely lack a robots.txt file by analyzing HTTP status codes and redirections.

Key Contributions and Results

1. Disparity in robots.txt Configuration

The study reveals a stark contrast in how reputable and misinformation sites configure access for AI agents:

• Prevalence of Restrictions: Among sites with valid robots.txt files, 60.0% of reputable news sites include a DisallowAll directive for at least one AI agent, compared to only 9.1% of misinformation sites.
• Scope of Exclusion: Reputable sites block an average of 15.5 distinct AI agents, whereas misinformation sites block an average of 0.77 distinct agents.
• Specific Agents: Over 50% of reputable sites disallow GPTBot, and 40–50% restrict major agents like CCBot, ClaudeBot, and Google-Extended. In contrast, misinformation sites rarely prohibit specific agents, with DisallowAll rates for any single agent remaining below 5%.
• Popularity Factor: The disparity persists even when controlling for site popularity (Tranco rank), indicating that the difference is driven by the nature of the content source rather than resource availability alone.

2. Active Blocking Mechanisms

The study measured active blocking (denying access via HTTP response based on User-Agent headers):

• Adoption: Both categories utilize active blocking, but the correlation with declared policies differs.
• Correlation: For reputable sites, active blocking behavior aligns closely with their robots.txt directives. For example, 16.9% of reputable sites actively block both ClaudeBot and Anthropic-AI, and nearly half of these also have corresponding robots.txt rules.
• Misinformation Behavior: Misinformation sites rely relatively more on active blocking without corresponding robots.txt declarations (e.g., 9.8% block both agents actively, but fewer than 2% have robots.txt rules for them).

3. Longitudinal Trends (2023–2025)

Analysis of six snapshots reveals a widening gap over time:

• Reputable Sites: The percentage of reputable sites disallowing at least one AI agent rose from 23% in September 2023 to nearly 60% by May 2025. The median number of disallowed agents exceeded 25 by early 2025.
• Misinformation Sites: These sites remained largely passive, with disallow rates staying below 10% and the median number of blocked agents remaining near zero throughout the period.
• Triggers: The increase in restrictions among reputable sites correlates with public announcements regarding AI data usage, specific crawler behaviors (e.g., PerplexityBot ignoring directives), and legislative developments like the EU AI Act.

Significance and Claims

The paper claims that its findings highlight a growing asymmetry in content accessibility that may fundamentally shape the training data available to LLMs.

• Data Environment Shift: As reputable news sources systematically opt out of AI crawling via robots.txt and active blocking, while misinformation sites remain permissive, the relative proportion of low-quality or false information in the data available for AI training may increase.
• Self-Reinforcing Loop: The authors suggest a risk where permissive misinformation sites (some potentially AI-generated) are continuously scraped, redistributed, and re-ingested into future training corpora, potentially amplifying the reach and perceived legitimacy of misinformation.
• Ethical and Transparency Implications: The study underscores the critical role of robots.txt as a governance tool for web transparency and data ethics. It suggests that the voluntary nature of the protocol is currently creating an uneven playing field where the "openness" of the web is inversely correlated with the credibility of the source.

The authors maintain modesty regarding their claims, noting that they do not verify the actual behavior of AI crawlers (only the intended restrictions) and cannot establish a direct causal link between specific exclusion policies and the final composition of proprietary LLM training datasets. However, they argue that the observed trends in access control mechanisms are a significant indicator of the evolving data landscape for AI.