Leakage-Aware Benchmarking of LLM Forecasting: Real-Time Nowcasts as the Decision-Time Input for Macro Factor Ranking

Technical Summary: Leakage-Aware Benchmarking of LLM Forecasting

Problem Statement

Current benchmarks for Retrieval-Augmented Generation (RAG) in financial forecasting frequently suffer from information leakage. In macro-driven financial contexts, data labeled with a specific timestamp (e.g., the Consumer Price Index for month $t$) is often not observable at the system's decision time (typically month-end), as official releases lag by days or weeks. Standard walk-forward backtests that ingest these future-labeled values inadvertently consume "look-ahead" information, inflating performance metrics. This issue is particularly acute in Large Language Model (LLM) pipelines, where agents ingest macro narratives and event summaries that may not respect historical observability constraints.

Methodology

The authors construct a leakage-controlled, monthly walk-forward benchmark to rank seven U.S. equity style factors (SMB, HML, RMW, CMA, UMD, BAB, QMJ) over a 36-month period (April 2023 – March 2026).

1. Decision-Time Information Constraints:
To ensure strict observability, the system is fed only data available at the decision date $t$:

• Lag-Shifted Macro Variables: FRED variables (CPI, Unemployment) are shifted by one calendar month.
• Real-Time Nowcasts: Unreleased current-month inflation is substituted with the Cleveland Fed's archived daily CPI nowcast queried at date $t$.
• Event Summaries: Recent FOMC and CPI release summaries are included.
• Historical Analog Retrieval: A retrieval module selects historical states from a pool restricted to at least 12 months prior to $t$ (a strict embargo) to prevent data contamination.

2. Model Architecture (Actor-Critic RAG):
The pipeline utilizes a 7B parameter open-source LLM (Qwen2.5-7B-Instruct, quantized to 4-bit) in a two-stage actor-critic configuration:

• Retrieval: A macro-analog module retrieves the top-4 historical months most similar to the current state (based on a 4-dimensional z-scored macro vector: growth pulse, inflation pulse, term spread, VIX pulse).
• Critic: A "critic" LLM reads the retrieved analogs and the current macro state to synthesize one tactical rule (e.g., "Raise HML by 0.15").
• Actor: An "actor" LLM maps the current real-time state, recent event summaries, and a rolling buffer of the last 6 tactical rules into scores for the seven equity factors.

3. Baselines:
The LLM pipeline is compared against non-LLM baselines using the exact same decision-time information set:

• kNN Macro Analog: Retrieves the same 4 analogs but predicts factor returns based on the mean of their realized next-month returns (no LLM).
• Ridge Regression: Linear models using either nowcast-only or full macro/nowcast vectors.

Key Results

1. Performance Metrics:

• The full LLM pipeline achieves a median monthly Spearman rank IC of +0.154 and a mean IC of +0.131.
• The signal is directionally positive across three non-overlapping 12-month sub-windows (+0.111, +0.165, +0.117).
• However, the mean IC is statistically underpowered: the bootstrap 95% confidence interval includes zero $[-0.02, +0.28]$, and the permutation $p$-value is 0.11.

2. Ablation and Attribution:

• The Role of Nowcasts: The most significant jump in median IC (from +0.009 to +0.154) occurred when the Cleveland Fed nowcast was added, indicating that real-time inflation information is a primary driver of the signal.
• LLM vs. kNN: The kNN macro-analog baseline (non-LLM) achieved a comparable median IC (+0.161) to the full LLM pipeline (+0.154). This suggests that the core "median signal" is driven by the retrieval of macro-similar states and the inclusion of real-time nowcasts, rather than the LLM architecture itself.
• LLM's Marginal Contribution: The LLM pipeline outperformed the kNN baseline in mean IC (+0.131 vs. +0.070) and in an economic sanity check (Long-Top-2/Short-Bottom-2 portfolio). The LLM's Sharpe ratio was +0.71 compared to +0.29 for kNN. The authors interpret this as the LLM providing marginal benefits specifically in extreme rankings (top and bottom factors), which drive long-short portfolio formation, rather than in central tendency.

Significance and Claims

The paper does not claim a breakthrough in predictive power or a deployable trading strategy. Instead, its significance lies in diagnostic rigor:

1. Leakage Control: It demonstrates that standard benchmarks often conflate model capability with information leakage. By enforcing strict decision-time constraints (specifically replacing future-labeled CPI with nowcasts), the authors isolate the true information set available to a decision-maker.
2. Disentangling Signal Sources: The study reveals that in this specific macro-conditioned task, real-time inflation nowcasts and macro-similar retrieval account for the majority of the median predictive signal. The LLM's contribution appears to be concentrated in refining extreme rankings rather than discovering new signals de novo.
3. Reproducibility: The authors release the full pipeline, including the "frozen" decision-time input archive (Cleveland Fed nowcasts), prompt templates, and the 36 reproducible critic rules, allowing for rigorous audit of LLM behavior in financial forecasting without data leakage.

The authors conclude that while the 7B retrieval-augmented LLM produces a positive signal, the evidence remains statistically modest ($n=36$), and the marginal value of the LLM architecture is likely confined to handling non-linear interactions in extreme market conditions rather than general factor prediction.