DatedGPT: Preventing Lookahead Bias in Large Language Models with Time-Aware Pretraining

1. Problem Statement

In financial forecasting and temporally grounded prediction tasks, Large Language Models (LLMs) pretrained on internet-scale data face a critical issue known as lookahead bias.

• The Issue: Standard LLMs are trained on massive datasets that often include future information relative to a specific prediction date. Consequently, when asked to predict outcomes (e.g., stock returns based on news from 2020), the model may not be "reasoning" based on the provided context but rather retrieving memorized outcomes encoded in its parameters during training.
• The Consequence: This conflation of memorization and reasoning invalidates the model's predictive validity in backtesting and real-world financial applications. Existing efforts to update models often lack strict temporal boundaries or rely on continual learning, which risks catastrophic forgetting or does not guarantee the exclusion of future data.

2. Methodology

The authors propose DATEDGPT, a family of language models designed with strict temporal constraints to eliminate lookahead bias. The methodology involves three core stages:

A. Time-Aware Dataset Curation

The authors constructed a dataset with explicit annual cutoffs spanning 2013 to 2024.

• Pretraining Data: They utilized FineWeb-Edu (derived from Common Crawl) as the base. Since crawl timestamps do not perfectly match content creation dates, they filtered documents based on the crawl date (rounded to the year). This ensures the model is never exposed to data crawled after its designated cutoff year.
  • Scale: Approximately 100 billion tokens per year for each of the 12 models.
• Instruction-Following Data: To enhance usability, they curated instruction-tuning datasets with the same temporal boundaries:
  • General Domain: Filtered widely used instruction datasets (e.g., Coconot, Tulu-3) using an LLM-based classifier (Llama-3.3-70B-Instruct) to remove time-sensitive information (e.g., queries about specific TV episodes released after the cutoff).
  • Finance-Specific: Curated domain-specific tasks including stock return prediction from news headlines and capital expenditure forecasting from earnings call transcripts. These were constructed by anchoring training instances to the exact date the underlying information became public.

B. Model Architecture and Training

• Architecture: A 1.3B-parameter model architecture, based on hyperparameters from the Llama family and GPT-2 reproductions.
• Training Strategy:
  • Pretraining: Twelve distinct models were trained from scratch (not via continual pretraining) for each year from 2013 to 2024.
  • Budget: Each model was trained for 25,000 iterations (~100B tokens).
  • Instruction Tuning: A second stage mixed general-domain and year-specific instruction data, trained for 3 epochs (~1B tokens).
• Hardware: Training required approximately 2,000 GPU hours per model on NVIDIA A100s.

3. Key Contributions

1. DATEDGPT Family: The introduction of the largest and most complete series of language models (12 models) with explicit annual cutoffs (2013–2024) trained on temporally partitioned data exceeding 100B tokens per year.
2. Rigorous Temporal Boundaries: Unlike previous works that rely on incremental updates, DATEDGPT models are trained independently from scratch, ensuring no data leakage from future years.
3. Finance-Optimized Instruction Tuning: The inclusion of curated, time-sensitive financial datasets (news-to-return, earnings-to-capex) that respect strict information constraints.
4. Interactive Evaluation Tool: A public web interface (www.datedgpt.com) allowing users to query models from different cutoff years to visually demonstrate the absence of future knowledge.

4. Experimental Results

The authors evaluated the models on standard benchmarks and specific temporal probing tasks.

• Language Understanding Performance:

  • On zero-shot benchmarks (HellaSwag, ARC, MMLU, TruthfulQA, IFEval), the DATEDGPT-INSTRUCT series achieved competitive performance against other small-scale models (e.g., SmolLM-1.7B, TinyLlama-1.1B, OPT-1.3B).
  • The models achieved an average score of up to 42.7, with notable gains in instruction following (IFEval), proving that models without access to up-to-date data can still develop strong general reasoning capabilities.
  • Performance remained consistent across different cutoff years (2013–2024), indicating that the temporal partitioning did not degrade general language capabilities.

• Lookahead Bias Verification (Perplexity Probing):

  • Using a perplexity-based probing method on quarterly news headlines (2013–2024), the authors tested DATEDGPT-BASE-2020.
  • Result: The model showed low perplexity for data up to its 2020 cutoff. Crucially, perplexity rose steadily for data published after 2020.
  • Significance: This "perplexity reversal" confirms that the model has not memorized patterns from post-cutoff events, validating the effectiveness of the temporal partitioning strategy in preventing lookahead bias.

5. Significance

• Reliable Financial Backtesting: DATEDGPT provides a foundational tool for the financial community to conduct rigorous, unbiased backtesting of LLM-driven trading strategies, ensuring that performance is due to reasoning rather than data leakage.
• Methodological Benchmark: It establishes a new standard for "time-aware" AI, demonstrating that strict temporal curation is feasible at scale without sacrificing model performance.
• Transparency and Reproducibility: By releasing the full model series, data, and training pipeline, the authors enable the research community to study the evolution of LLM knowledge and the specific impacts of temporal grounding.
• Addressing a Critical Gap: While many models claim to be "up-to-date," DATEDGPT is unique in providing a controlled environment where the absence of future knowledge is guaranteed, which is essential for causal inference and economic forecasting.