💬 NLP

Forecasting With LLMs: Improved Generalization Through Feature Steering

This paper demonstrates that using sparse autoencoders to identify and amplify time-aware features in LLMs can causally reduce look-ahead bias in forecasting tasks while preserving general reasoning performance, thereby improving the models' ability to generalize based on historical patterns.

Original authors: Humzah Merchant, Bradford Levy

Published 2026-06-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Humzah Merchant, Bradford Levy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Spoiler" in the Room

Imagine you are playing a trivia game where you have to guess what will happen in the future, but you are only allowed to use information available up to today.

Now, imagine you are an AI (a Large Language Model) that has read the entire history of the internet, including the answers to tomorrow's quiz. When you ask it, "What will the stock market do next year?" it might accidentally "cheat." Instead of thinking logically based on today's news, it might just remember the actual answer from its training data.

In the paper, the authors call this "look-ahead bias." It's like a student taking a history test who secretly peeked at the answer key from next year. They get the right answer, but they didn't actually learn how to reason; they just memorized the outcome. This is bad for forecasting because if the future is different from the past, the AI will fail.

The Solution: Finding the "Time Travel" Switch

The researchers wanted to know: Is there a specific part inside the AI's brain that knows it's "time-traveling" (using future knowledge), and can we turn that off?

To find this, they used a tool called a Sparse Autoencoder. Think of the AI's brain as a massive, dark warehouse with millions of light switches. Most switches are off. The researchers built a special camera (the autoencoder) that can see exactly which specific switches light up when the AI is thinking about "time" or "dates."

They found two types of switches:

The "Time-Aware" Switch: This lights up when the AI is correctly thinking, "I am in 2018, and I don't know what happens in 2019 yet."
The "Spoiler" Switch: This lights up when the AI is accidentally using its memory of the future to cheat.

The Experiment: Turning Up the Volume

The researchers tried a clever trick. They didn't retrain the AI or delete its memory. Instead, they found the "Time-Aware" switch and turned up its volume (amplified it) while the AI was answering questions.

Think of it like this: If you are trying to listen to a quiet radio station (logical reasoning) but there is loud static (cheating with future knowledge), you don't smash the radio. Instead, you turn up the volume on the station you want to hear until it drowns out the static.

What They Found

Turning up the "Time-Aware" switch worked. When they amplified this feature, the AI stopped cheating. It started making predictions based only on the information available at the time, just like a human analyst would.
- Example: When asked to predict a company merger in 2018, the AI stopped saying, "They merged in 2019, so I'll pick them!" Instead, it said, "Based on 2018 trends, they might pick this other company."
Turning up the "Spoiler" switch didn't work. They tried to amplify the switches they thought were responsible for cheating, but it didn't stop the AI from using future knowledge. This suggests that "cheating" isn't just one simple switch; it's a complex behavior that is harder to fix directly.
The AI didn't get "dumber." Crucially, making the AI more time-aware didn't ruin its ability to answer other questions (like general knowledge or math). It just made it better at not cheating on forecasting tasks.

The Takeaway

The paper proves that we can fix AI forecasting errors not by deleting its memory, but by steering its internal focus. By finding and amplifying the specific "time-aware" parts of the AI's brain, we can force it to reason based on the past and present, rather than peeking at the future.

It's like teaching a student to stop looking at the answer key by gently reminding them, "Remember, you are in the past, and you don't know the future yet," until that reminder becomes their natural way of thinking.

Technical Summary: Forecasting With LLMs: Improved Generalization Through Feature Steering

Problem Statement

The application of Large Language Models (LLMs) to forecasting tasks faces a critical challenge: distinguishing between genuine generalizable reasoning and the mere retrieval of memorized outcomes from training data. When an LLM is asked to forecast an event that occurred within its training window, the model may simply recall the historical outcome rather than reasoning based on the information available at the time the forecast would have been made. This phenomenon, known as look-ahead bias (LAB), results in correct answers that lack generalizability and fail to perform well on out-of-sample observations.

Existing mitigation strategies, such as chronologically restricted models or input-level anonymization, often come at the cost of model capability or are insufficient against modern training paradigms. Furthermore, while temporal awareness can be induced via system prompts or fine-tuning, the internal representations supporting these behaviors—and whether they correspond to distinct, causal features—remain poorly understood.

Methodology

The authors employ Sparse Autoencoders (SAEs) to decompose dense transformer activations into sparse, human-interpretable features. This approach allows for the identification and causal intervention on specific internal representations associated with temporal reasoning.

Feature Discovery:
- The authors utilize prediction-market data (specifically from Kalshi) as a ground truth for historical expectations.
- They construct prompts with specific "knowledge cutoff" dates (e.g., "Today is {Date}...") and ask the model to predict outcomes.
- They identify two contrasting behavioral modes:
  - Time-Aware Reasoning: The model selects the market favorite available at the cutoff date.
  - Look-Ahead Bias (LAB): The model selects the eventual realized outcome, implying access to post-cutoff information.
- By contrasting the activation rates of SAE features between these two modes, they surface candidate features associated with each behavior.
Causal Intervention (Steering):
- Using the formula $\hat{x}_{steered} = W_{dec} (z + \alpha e_j) + b_{dec}$ , the authors amplify specific feature activations ( $z$ ) by adding a steering magnitude ( $\alpha$ ) before decoding.
- This intervention is applied during generation to test if amplifying a feature causally shifts the model's behavior.
Evaluation:
- Discovery Setting: Prediction-market questions (Kalshi).
- Generalization Setting: Two out-of-domain, free-form forecasting tasks where look-ahead bias is identifiable in natural text rather than fixed multiple-choice answers:
  - M&A Forecasting: Predicting acquisition targets one year prior to public announcement.
  - Pharmaceutical Forecasting: Predicting growth drivers for specific companies based on information available up to a specific date.
- Utility Check: The authors monitor performance on MMLU-CoT and MMLU-Pro CoT to ensure that steering does not degrade general reasoning capabilities.

Key Contributions and Results

1. Identification of Interpretable Temporal Features

The study successfully identifies specific SAE features that correlate with temporal reasoning behaviors.

Time-Aware Features: Features such as L40:24283 (associated with "by and before") and L53:9448 (associated with "years and dates") were identified.
Look-Ahead Bias Features: Features such as L31:2450 and L53:9987 were initially identified as candidates for LAB.

2. Causal Reduction of Look-Ahead Bias

The core finding is that amplifying time-awareness features causally reduces look-ahead bias.

When features like L40:24283 (Gemma 3) or L48:28961 (Qwen 3.5) were amplified, the models significantly reduced their reliance on post-cutoff knowledge in both M&A and pharmaceutical forecasting tasks.
The models shifted from citing future events (e.g., "Goldcorp was acquired in 2019") to reasoning based on the strategic landscape of the prompt date (e.g., "Kinross is a logical target based on 2018 data").
Crucially, this reduction in bias occurred without degrading general utility, as measured by stable performance on MMLU benchmarks.

3. Asymmetry of Feature Steering

The paper reports a notable asymmetry in the effectiveness of steering:

Amplifying time-awareness features effectively suppresses LAB.
Amplifying or suppressing candidate look-ahead-bias features did not produce a comparable effect. The authors suggest that LAB may be a default state of the model when not actively constrained by time-aware features, rather than a distinct feature that can be easily "turned off."

4. Cross-Task Generalization

The features identified via prediction-market data generalized effectively to entirely different domains (M&A and Pharma), demonstrating that these temporal representations are not dataset artifacts but fundamental components of the model's reasoning process.

Significance and Claims

The paper claims to advance the understanding of LLM forecasting by moving beyond treating look-ahead bias as a simple consequence of memorization. Instead, it frames LAB as an internal behavioral tendency that can be identified and causally shifted through feature steering.

Interpretability: The work demonstrates that temporal reasoning is supported by specific, interpretable features within the model's latent space.
Causal Control: It provides evidence that these features can be manipulated to induce more historically grounded reasoning, offering a potential mechanism for improving the reliability of LLMs in forecasting tasks.
Limitations and Future Directions: The authors are modest in their claims, stating that feature steering is not a "complete standalone solution." They note that strong interventions can eventually degrade model quality. They propose that the most promising path forward involves combining moderate-strength internal steering with other methods, such as supervised fine-tuning (SFT), reinforcement learning (RL) for time awareness, unlearning approaches, and improved prompt construction.

In summary, the paper establishes that interpretable temporal features exist in LLMs and that amplifying these features can causally reduce look-ahead bias, thereby improving the generalization of forecasting models to future observations.