🤖 machine learning

Which Histories Matter for Time Series Forecasting? Learning Predictive Relevance with Future Supervision

This paper proposes a future-supervised learning framework that redefines historical relevance for time series forecasting by training a lightweight reranker to prioritize past examples based on their expected future utility, demonstrating that optimal retrieval strategies are domain-dependent and outperform traditional similarity-based or stationarity-aware methods.

Original authors: Yong-Hoon Choi, Youngjin Cho

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yong-Hoon Choi, Youngjin Cho

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of data, time is the most critical dimension. Scientists and engineers often look at streams of numbers—temperatures, stock prices, or electricity usage—hoping to predict what comes next. A common strategy for making these predictions is to look backward. The logic is simple: if the past looks like the present, the future might look like the past. This idea, known as retrieval, involves finding historical moments that resemble the current situation and assuming their subsequent outcomes will be a good guide. It is a bit like a sailor checking the wind patterns of yesterday to guess the weather for tomorrow. For decades, the standard rule has been to find the most similar past patterns, assuming that similarity is the best proxy for usefulness. If two days look alike, they should be treated as equally valuable for prediction.

However, a new study challenges this long-held assumption. Researchers at Kwangwoon University in Seoul asked a sharper question: just because a past moment looks similar to today, does it actually matter for predicting the future? They discovered that visual similarity is often a poor guide. Two historical periods can look nearly identical at the start but diverge wildly afterward due to changing conditions. To solve this, the team developed a method that learns which historical examples are truly useful by looking at what actually happened next, but only during the learning phase. They created a system that uses the known futures of the past to teach a computer how to rank historical examples, while ensuring that when the system is used in the real world, it relies only on information available up to the present moment.

The researchers tested their approach on six different real-world datasets, including electricity usage, traffic flow, weather patterns, and solar energy generation. They started with a standard method that picks the top one hundred most similar past periods. Then, they introduced a lightweight, intelligent filter to re-rank those candidates. This filter was trained using "privileged information": the actual future outcomes of the historical periods, which are known during training but unavailable during real-time prediction. The system learned to recognize which of the similar-looking pasts would lead to the most accurate forecast for the current situation. The results were striking. In many cases, this learned ranking significantly outperformed the standard similarity-based approach. For instance, in predicting solar energy, the new method was decisively better, reducing prediction errors by a large margin compared to the old rules.

Yet, the story is not one of a single, universal winner. The study revealed that the value of this new learning method depends entirely on the type of data being analyzed. In some domains, like electricity usage and currency exchange rates, the old-fashioned similarity rules remained surprisingly strong. In these areas, the future behavior of a similar past was often consistent enough that a simple geometric match worked well. In fact, for electricity and exchange rates, a straightforward mathematical rule based on the last observed value continued to outperform the complex learned system. This finding is crucial because it proves that there is no single "best" way to find relevant history. Instead, the nature of relevance changes from one domain to another.

The researchers broke down this difference into two distinct types of usefulness. In some cases, a historical example is useful because it is broadly helpful across many different situations; it is a "candidate-global" asset. In other cases, an example is only useful because it fits the specific, unique details of the current moment; this is "query-specific" relevance. The study showed that the new learning method shines when the relevance is query-specific, such as in solar energy prediction where the exact context matters immensely. Conversely, when the relevance is global, like in currency markets, a simple prior or a basic similarity rule is often sufficient. The team also ruled out the idea that their success was simply due to using a more complex computer model. By comparing their system to controls that used the same model architecture but scrambled the future information, they proved that the gains came specifically from learning the correct relationship between the past and its future, not from the model's size or complexity.

Ultimately, this work shifts the focus from simply finding the most similar past to understanding why a past example should matter. It demonstrates that while similarity is a good starting point for finding candidates, it is not the final answer. The most effective way to predict the future depends on the specific structure of the data. In some worlds, a simple look-alike is enough; in others, a deep understanding of how that look-alike behaved in its own future is essential. The study concludes that the future of time-series forecasting lies not in replacing similarity entirely, but in knowing when to trust it and when to look deeper, using the lessons of the past to teach the system how to judge the present more wisely.

Technical Summary: "Which Histories Matter for Time Series Forecasting? Learning Predictive Relevance with Future Supervision"

1. Problem Formulation

Time-series forecasting systems often rely on retrieval-augmented methods, where historical segments are selected based on similarity to a current query window to inform predictions. A fundamental limitation of this approach is that past-pattern similarity (e.g., cosine similarity or Euclidean distance) serves only as a proxy for predictive relevance. Two historical segments may appear nearly identical but diverge significantly in their future trajectories due to nonstationarity or regime shifts.

The paper addresses the question: Which historical examples should be expected to matter for a specific query? The authors argue that while similarity is a practical mechanism for coarse candidate generation, it should not necessarily be the final criterion for ranking candidates. The core challenge is to learn a relevance function that predicts future utility using only information observable at inference time, despite the fact that the true "usefulness" of a history is only known retrospectively.

2. Methodology

The proposed framework treats historical retrieval as a learning problem where future outcomes act as privileged supervision available only during training. The method follows a two-stage pipeline:

A. Coarse Candidate Generation
The system first constructs a candidate set $C_M(q)$ using standard pattern similarity (specifically, cosine similarity on normalized past windows). This step is not replaced by the learning model; rather, the learning model operates strictly within this pre-selected pool. If a relevant candidate is excluded from the initial Top- $M$ set, the reranker cannot recover it.

B. Future-Supervised Reranking
A lightweight residual Multi-Layer Perceptron (MLP) reranks the candidates in $C_M(q)$ .

Input Features: The model uses strictly past-only observable statistics, including current relative level, short-vs-long mean displacement, change metrics, volatility ratios, normalized slope, and lag-1 autocorrelation.
Training Objective (Privileged Supervision): During training, the system has access to the query's future trajectory $y_q$ and the candidate's future $y_i$ . It defines a "utility" $u(q, i)$ based on the negative distance between these futures. The model is trained to predict a ranking that maximizes the expected future utility $E[u(q, i) | o_q, o_i]$ using a listwise loss function (cross-entropy over a soft ranking target derived from future distances).
Inference: At test time, the future branch is absent. The model predicts relevance scores using only the past context and the candidate's past features, effectively learning to approximate the conditional expectation of future utility.

C. Structural Decomposition and Controls
The authors formalize predictive relevance as the sum of two orthogonal components:

Candidate-Level Utility ( $G^*$ ): How broadly useful a candidate is across many queries.
Query-Specific Compatibility ( $H^*$ ): How specifically a candidate matches the current query's future.

To diagnose these regimes, two controls are introduced:

Candidate Prior: Ranks candidates by their average compatibility with historical training queries (query-independent).
Shuffled Future: Uses the same architecture but permutes the query-future correspondence during training. This destroys the query-specific signal while preserving the marginal distribution of futures. Comparing the "Learned" model against "Shuffled" isolates the value of correct query-future correspondence.

3. Key Contributions

Formulation of Predictive Relevance: The paper defines historical retrieval as learning expected predictive utility, introducing a strictly past-only reranker supervised by futures available only during training (privileged information).
Decomposition of Relevance: It decomposes optimal relevance into candidate-level and query-specific components, providing empirical diagnostics (Candidate Prior and Shuffled Future) to identify which regime dominates a given domain.
Domain-Dependent Findings: Through extensive experiments, the paper demonstrates that no single retrieval principle dominates universally. Instead, the efficacy of learned relevance depends on the underlying structure of the domain.

4. Experimental Results

The method was evaluated on six benchmarks (ETTh1, Weather, Electricity, Traffic, Exchange, Solar) across multiple forecasting horizons.

Overall Performance: The future-supervised reranker ("Learned") improved upon the baseline Pattern similarity in all 18 benchmark-horizon settings. In a confirmatory suite of 12 tasks (Electricity, Traffic, Exchange, Solar), it outperformed the baseline in all 12 and surpassed a matched-protocol SARAF retrieval rule in all 12.
Relevance Regimes:
- Query-Specific Domains (e.g., Solar, ETTh1, Traffic): Here, the "Learned" model significantly outperformed the "Shuffled" control, indicating that correct query-future correspondence is the primary driver of relevance. In Solar, the learned model was decisively superior to strong geometric baselines.
- Candidate-Global Domains (e.g., Weather, Exchange): In these domains, the "Candidate Prior" (query-independent utility) significantly improved upon Pattern, while the "Learned" model offered little gain over the "Shuffled" control. Notably, a simple last-value-anchored L2 similarity rule remained superior to the learned model in Exchange and Electricity.
- Mixed Domains (e.g., Electricity): These domains exhibited characteristics of both regimes.
Ablation Studies:
- Gains were not solely due to the MLP architecture or added context features; the "Learned" model consistently outperformed the "Shuffled" control, proving that the training-time query-future correspondence is the critical signal.
- The results were not due to candidate pool coverage; restricting strong geometric baselines (like L2) to the same Top-100 pool used by the learned model did not eliminate the performance gap in query-specific domains, nor did it close the gap in candidate-global domains where L2 remained superior.
Financial Case Study: Applied to S&P 500 stocks, the method showed that cross-stock future-compatible retrieval improved retrieval quality and forecast accuracy over pattern-based baselines, though the final retrieval-based predictor remained slightly behind the strongest direct forecasting baselines.

5. Significance and Claims

The paper makes a modest but structurally significant claim: Predictive relevance is not a universal property but a domain-dependent structure.

Rejection of Universal Superiority: The authors explicitly state that learning does not always defeat similarity. In domains with strong candidate-global utility (like Exchange), simple geometric priors or candidate-level statistics are sufficient and often superior.
Value of Future Supervision: The primary value of the proposed method lies in identifying and exploiting query-specific regimes (like Solar), where the relationship between past patterns and future outcomes is complex and non-stationary. In these cases, privileged future supervision reveals structures that static similarity metrics miss.
Separation of Concerns: The paper emphasizes that retrieval relevance (selecting the right history) is distinct from forecast aggregation (combining retrieved futures). Improvements in relevance do not automatically guarantee state-of-the-art forecasting performance, as aggregation strategies and error cancellation play independent roles.

In conclusion, the work advocates for evaluating historical retrieval not just by how it searches, but by why the retrieved examples are expected to matter, suggesting that the optimal retrieval strategy must be adapted to the specific relevance structure of the target domain.