Temporal Leakage in Search-Engine Date-Filtered Web Retrieval: A Retrospective Forecasting Case Study

1. Problem Statement

Retrospective Forecasting (RF) evaluates forecasting systems on questions with known outcomes. To ensure validity, the system must only access information available before the question's resolution date (the "cutoff"). If post-cutoff information leaks into the retrieved documents, the evaluation is compromised, artificially inflating accuracy.

Current RF pipelines typically rely on search-engine date filters (e.g., Google's before: operator or DuckDuckGo's date-range filter) to enforce this cutoff. The paper argues that this assumption is flawed: date filters often fail to exclude documents containing post-cutoff information because web pages are dynamic, metadata is unreliable, and search engines index updated content without respecting the original publication date.

2. Methodology

The authors conducted a systematic audit of two major search engines using a large-scale dataset of resolved forecasting questions.

• Dataset: 393 resolved forecasting questions from the Metaculus platform (resolution dates spanning 2021–2025).
• Retrieval Process:
  • For each question, an LLM generated 10–20 search queries.
  • Queries were run on Google Search (using the before: filter) and DuckDuckGo (using date-range filters) with the cutoff set to the question's opening date.
  • Approximately 100 unique URLs were retrieved per question, totaling 38,879 URLs from Google and 34,454 from DuckDuckGo.
• Leakage Scoring System:
  • A 0–4 severity scale was developed to quantify leakage:
    • 0: No post-cutoff info.
    • 1: Topical but uninformative.
    • 2: Weak directional signal.
    • 3: Major signal enabling strong inference.
    • 4: Directly reveals the answer.
  • LLM-as-a-Judge: The model gpt-oss-120b was used to score the content of retrieved pages against the rubric. Human annotators validated the system, achieving 76.1% exact accuracy and 0.85 Quadratic Weighted Kappa agreement with the LLM.
• Downstream Impact Experiment: The authors tested gpt-oss-120b on binary forecasting questions (opened in 2025) to measure how leakage affects prediction accuracy (Brier score) when provided with leaky vs. leak-free documents.

3. Key Contributions

1. Leakage Audit: The first systematic, quantitative study demonstrating that date-filtered search is unreliable.
2. Downstream Impact Analysis: Quantifying how leakage artificially inflates forecasting performance metrics.
3. Mechanism Characterization: Identifying and categorizing four specific technical pathways through which temporal leakage occurs.

4. Key Results

A. Prevalence of Leakage

Date filters failed to prevent information leakage on both engines.

• Google: 71% of questions had at least one retrieved page with major leakage (score ≥3); 41% had a page directly revealing the answer (score 4).
• DuckDuckGo: 81% of questions had major leakage; 55% had direct answer leakage.
• Overall: ~33–34% of all retrieved URLs contained some post-cutoff information.

B. Impact on Forecasting Accuracy

The presence of leaked data drastically improved model performance, invalidating the evaluation:

• Baseline (No Retrieval): Brier Score = 0.244.
• Leak-Free Documents (Score 0): Brier Score = 0.242 (No improvement over baseline).
• Leaky Documents (Scores 3–4): Brier Score dropped to 0.108 (a massive, artificial improvement).
• Direct Leakage Only (Score 4): Brier Score = 0.129.
• Conclusion: Access to post-cutoff information reduces the Brier score by more than half, making the model appear significantly more accurate than it actually is.

C. Identified Leakage Mechanisms

The paper categorizes four primary ways post-cutoff information enters date-filtered results:

1. Direct Page Updates: Articles are updated over time (e.g., a 2017 missile tracking page updated to include 2023 data), but the search engine returns it based on the original URL or metadata.
2. Related-Content Leakage: Sidebars, "Related Articles," or dynamic modules on a historical page inject current news (e.g., a 2016 article with a 2023 news snippet in the sidebar).
3. Absence-Based Signals: Comprehensive timelines or reports that omit expected events (e.g., a timeline up to 2025 showing no war) allow the model to infer the answer by absence, even if the text itself is old.
4. Unreliable Metadata: Self-reported timestamps in HTML or CMS are often stale or incorrect. A page may claim a 2020 update date but contain text describing 2023 events, bypassing filters that rely on scraped dates.

D. Temporal Variation

Leakage severity is inversely correlated with the recency of the cutoff.

• 2021–2022 Cutoffs: Highest leakage rates (>46% of URLs contained post-cutoff info). Older pages have had more time to accumulate updates.
• 2025 Cutoffs: Lower leakage rates (~27%), as fewer updates have occurred on pages since the cutoff.

5. Significance and Recommendations

• Invalidation of Current RF Pipelines: The study proves that relying on live search-engine date filters for retrospective evaluation is insufficient and leads to misleading conclusions about LLM forecasting capabilities.
• Generalizability: The failure mode applies to any time-sensitive retrieval task, including fact-checking and timeline summarization.
• Recommendations:
  • Move away from live date-filtered search.
  • Adopt frozen, time-stamped web snapshots (e.g., Wayback Machine or custom databases collected prior to the cutoff) for evaluation.
  • Implement stronger retrieval safeguards that verify content dates against the actual text, not just metadata.

The authors have released their dataset, queries, and audit tools to facilitate further research into robust temporal retrieval.