Detecting Lookahead Bias in LLM Forecasts

Technical Summary: Detecting Lookahead Bias in LLM Forecasts

Problem Statement
Large language models (LLMs) are increasingly utilized in empirical economics and finance to generate firm-level forecasts from unstructured text, such as news headlines and earnings call transcripts. These models often demonstrate predictive power for realized outcomes (e.g., stock returns, capital expenditures) comparable to or exceeding traditional econometric benchmarks. However, a critical concern is that this predictive ability may not stem from genuine economic reasoning over the input text but rather from the model's "memory" of realized outcomes ingested during pretraining. When a model is evaluated on dates within its training corpus, the realized outcome for a specific firm-date pair may have been absorbed alongside retrospective narratives, creating a "lookahead bias" where the model effectively recalls the answer rather than reasoning through the prompt. Existing mitigation strategies, such as entity masking or restricting evaluation to post-cutoff data, are either partial (leaving subtle memorization channels intact) or statistically limiting (due to short out-of-sample windows). There is a lack of a portable diagnostic tool to determine whether an LLM's forecast relies on memorization or reasoning for a specific task.

Methodology
The authors propose a statistical procedure centered on a metric termed Lookahead Propensity (LAP), denoted as $LAP^{(known)}_{i,t+1}$.

1. Construction of LAP:

   • The method utilizes a "date-only recall query" containing only the firm's name, ticker, and target date (or quarter), with no contemporaneous information (no headline, transcript, or fundamentals).
   • The model is prompted to predict the direction of the outcome (e.g., "up" or "down") or abstain ("unknown").
   • $LAP^{(known)}$ is calculated as the sum of the first-token probabilities assigned to the directional labels: $P_{up} + P_{down}$.
   • A value near 1 indicates the model "knows" the outcome (high memorization), while a value near 0 indicates abstention (no recall).

2. Econometric Framework:

   • The authors model the LLM forecast ($\hat{\mu}_t$) as the sum of a true conditional expectation and a contamination term: $\hat{\mu}_t = \mu(X_t) + \gamma L_t \epsilon_{t+1}$.
   • Here, $L_t$ represents firm-period memorization strength (proxied by $LAP^{(known)}$), and $\gamma$ represents the model's propensity to leak memorized content into text-conditioned forecasts.
   • The detection test involves an accuracy regression of the realized outcome ($Y_{t+1}$) on the LLM forecast ($\hat{\mu}_t$), the LAP measure, and their interaction:
     $$Y_{t+1} = \beta_1 \hat{\mu}_t + \beta_2 LAP_t + \beta_3 (\hat{\mu}_t \times LAP_t) + \epsilon_{t+1}$$
   • Proposition 1: Under mild conditions, a positive and significant interaction coefficient ($\beta_3 > 0$) serves as a one-sided diagnostic for $\gamma > 0$, indicating that the model's predictive power is amplified by memorized content.

3. Validation and Placebo:

   • Validation: The directional component of the recall query ($(U-D) = P_{up} - P_{down}$) is tested against realized outcomes. Predictive power here confirms the existence of memorization ($L_t > 0$).
   • Placebo: The test is applied to a post-training-cutoff sample (dates strictly after the model's knowledge cutoff). In this window, $LAP^{(known)}$ should collapse to zero, and the interaction term should become insignificant, confirming that the in-sample amplification was driven by training-time exposure.

Key Empirical Results
The authors apply this framework to two forecasting tasks using Llama-3.3-70B (training cutoff: December 2023):

1. Stock News Predicting Returns:

   • Using Bloomberg headlines to predict next-day stock returns (in-sample: 2012–2023).
   • The baseline LLM signal predicts returns ($t=12.18$).
   • The interaction term ($LLM \times LAP$) is positive and highly significant ($t=3.64$). A one-standard-deviation increase in $LAP^{(known)}$ raises the marginal effect of the LLM signal by approximately 32%.
   • The recall query alone (without headlines) predicts returns with a $t$-statistic of 3.53, but only for firm-days with high $LAP^{(known)}$.
   • Out-of-Sample (2024): The $LAP^{(known)}$ mean collapses to near zero, and the interaction term becomes statistically insignificant ($t=1.06$), validating the lookahead-bias hypothesis.

2. Earnings Calls Predicting Capital Expenditures:

   • Using transcripts to predict capital expenditures two quarters ahead (in-sample: 2006–2020).
   • The baseline LLM signal predicts future capex ($t=16.69$).
   • The interaction term is positive and significant ($t=2.01$). A one-standard-deviation increase in $LAP^{(known)}$ raises the marginal effect by about 12%.
   • Out-of-Sample (2023Q3–2024Q1): The interaction term loses significance ($t=1.35$), and the amplification effect vanishes as the recall dates fall strictly after the training cutoff.

3. Comparison with Inner Confidence:

   • The authors test the $LAP$ measure against "Inner Confidence" (the model's probability for its chosen answer in the main prompt).
   • Both interaction terms ($LLM \times LAP$ and $LLM \times Inner Confidence$) remain significant when included jointly, indicating they capture distinct components of forecasting performance: $LAP$ captures memorization, while inner confidence captures the model's certainty in its reasoning.

Significance and Claims
The paper claims to provide a portable, cost-efficient, and model-level diagnostic for assessing the validity of LLM-generated forecasts.

• Diagnostic Utility: It allows researchers to quantify how much of an LLM's apparent predictive power is driven by memorization versus reasoning without requiring retraining, access to proprietary training data, or the construction of leak-free models.
• Task-Specific Nature: The authors emphasize that lookahead bias is not a universal failure of LLMs but is task-specific, depending on the input domain, target variable, model, and prompt.
• Practical Application: The test requires only the ability to query a model and read its first-token label probabilities. It enables researchers to "net out" memorization effects to determine the true economic content of a forecast.
• Limitations: The test identifies the sign of the contamination loading ($\gamma > 0$) rather than its precise magnitude. It does not suggest discarding LLM forecasts but rather provides a tool to interpret their reliability in specific contexts.

The paper concludes that a non-trivial share of the predictive content in LLM-based financial forecasts reflects training-time memorization, and the proposed $LAP^{(known)}$ test offers a rigorous method to detect and account for this bias.