Are We Winning the Wrong Game? Revisiting Evaluation Practices for Long-Term Time Series Forecasting

This paper critiques the current metric-centric evaluation paradigm in long-term time series forecasting, arguing that an overreliance on aggregate error metrics like MSE and MAE misaligns with real-world objectives, and proposes a multi-dimensional framework prioritizing structural coherence and decision-level relevance to advance meaningful forecasting progress.

The Gist

Imagine you are training a team of athletes to predict the weather for the next year.

Right now, the scientific community has created a giant Trophy Case (a leaderboard) where the only thing that matters is how close the athletes' guesses are to the actual temperature, measured by a single, rigid ruler. If Athlete A guesses 72°F and the real temperature is 72.5°F, they get a slightly better score than Athlete B who guesses 73°F.

The paper "Are We Winning the Wrong Game?" argues that we are so obsessed with winning this specific trophy that we've forgotten what the athletes are actually supposed to be doing.

Here is the breakdown of the problem and the proposed solution, using simple analogies:

1. The Problem: The "Scoreboard Obsession"

Currently, Long-Term Time Series Forecasting (predicting things like energy use, traffic, or stock prices over long periods) has become a game of chasing numbers.

• The Current Game: Researchers build complex AI models just to shave off tiny fractions of error (like reducing a mistake from 0.354 to 0.351). They publish tables showing these tiny improvements to say, "Look, we won!"
• The Trap: The paper calls this a "Metric Monoculture." It's like judging a chef only by how perfectly they can dice a carrot, ignoring whether the soup tastes good, if the ingredients are fresh, or if the meal actually feeds the hungry people.
• The Result: Models are becoming experts at "gaming the system." They are learning the specific quirks of the test datasets (the "practice fields") rather than learning how to understand real-world chaos. They are getting better at the test, but not necessarily better at the job.

2. The Real Issue: "Curve Fitting" vs. "Understanding the Story"

The authors point out a crucial difference between fitting a curve and understanding a story.

• The "Curve Fitter": Imagine a model that tries to draw a line through every single wobble in a stock market chart. It matches the noise perfectly. On the scoreboard, it looks amazing because its error is low. But in the real world, that noise is just random chatter. If you follow that model, you might panic over a temporary blip that means nothing.
• The "Storyteller": Imagine a model that ignores the tiny wobbles and focuses on the big trend: "The market is slowly going up." It might miss a few specific daily numbers (so its "score" is slightly worse), but it tells you the truth about where things are heading.
• The Conflict: The current scoreboard rewards the "Curve Fitter" because it matches the numbers better. But in the real world (like planning a power grid or managing traffic), the "Storyteller" is often more useful because it helps humans make better decisions.

3. The Proposed Solution: A New "Report Card"

The authors suggest we stop looking at a single number and start looking at a 3-Dimensional Report Card. Instead of just asking "How close was the guess?", we should ask three new questions:

A. Statistical Fidelity (The "Accuracy" Check)

• Does the model actually get the numbers right?
• This is the old way, but it's still necessary. We need to know if the model is generally accurate.

B. Structural Coherence (The "Logic" Check)

• Does the model understand the rhythm of the data?
• Analogy: If you are predicting the tides, a good model should know the water goes up and down in a cycle. If the model predicts the water will just go up forever, or go up and down randomly, it has failed the "Logic Check," even if the numbers are close on average.
• We need to check if the model preserves trends (is it going up or down?), seasonality (does it know summer is hot?), and regime shifts (does it know when a sudden storm is coming?).

C. Decision-Level Relevance (The "Usefulness" Check)

• Does this forecast help someone make a decision?
• Analogy: Imagine a doctor predicting a patient's health.
  • Model A predicts the exact temperature every hour but misses the fact that the patient is getting sicker overall.
  • Model B predicts a slight fever but correctly warns, "The patient is trending toward a serious infection."
  • Model B might have a slightly "worse" number score, but it is the winner because it saves the patient's life. The paper argues we need to reward Model B.

4. The Future: No More "Universal Champions"

The paper concludes that there is no such thing as a "Super Model" that wins at everything.

• Old Way: "Model X is the best because it won the leaderboard."
• New Way: "Model X is the best for traffic planning because it handles sudden jams well. Model Y is the best for energy grids because it handles seasonal changes well."

The Bottom Line

We are currently winning the game of getting the highest score on a test, but we might be losing the game of understanding the world.

The authors want us to stop treating forecasting like a video game where you just try to beat the high score. Instead, they want us to treat it like a diagnostic tool—one that helps us understand the underlying story of time, trends, and changes, so we can make better decisions in the real world.

In short: Stop trying to be the perfect calculator; start trying to be a wise predictor.

Technical Summary

Based on the paper "Are We Winning the Wrong Game? Revisiting Evaluation Practices for Long-Term Time Series Forecasting" by Thanapol Phungtua-eng and Yoshitaka Yamamoto, here is a detailed technical summary.

1. Problem Statement

The paper identifies a critical misalignment in the current state of Long-Term Time Series Forecasting (LTSF) research. The field has evolved into a "GAME" driven by benchmark competitions where success is defined almost exclusively by marginal reductions in aggregated pointwise error metrics, specifically Mean Squared Error (MSE) and Mean Absolute Error (MAE).

Key Issues Identified:

• Metric Monoculture: Research is concentrated on a small set of canonical datasets (e.g., ETT, Weather) and fixed forecasting horizons. Models are optimized to win leaderboard tables rather than to solve real-world forecasting problems.
• Structural Misalignment: Optimizing for aggregate pointwise error does not guarantee that a model captures essential temporal structures such as trend stability, seasonal coherence, or robustness to regime shifts.
• Curve Fitting vs. Understanding: The current incentive structure encourages models to fit high-frequency noise in benchmark data (overfitting to specific dataset statistics) rather than learning the underlying temporal dynamics. This risks reducing forecasting to mere curve fitting, ignoring decision-level relevance.
• The "Universal Champion" Fallacy: The assumption that a single model can dominate across all datasets and scenarios is challenged; performance is often context-dependent, yet evaluation treats it as absolute.

2. Methodology and Proposed Framework

The paper does not propose a new forecasting algorithm or a new dataset. Instead, it proposes a paradigm shift in evaluation methodology. The authors argue for moving from a single-number optimization task to a multi-dimensional evaluation perspective.

The proposed framework consists of three complementary dimensions:

A. Statistical Fidelity

While acknowledging the need for standard error metrics, the authors suggest refining how they are used:

• Use scaled errors (e.g., MASE) and probabilistic measures (e.g., CRPS) to improve comparability across different datasets.
• Include computational metrics (inference time, training efficiency) as they are relevant for practical deployment.
• Crucial Shift: Move away from global averages toward window-level analysis. Instead of aggregating errors across the entire test set, evaluate performance within individual sliding windows to detect heterogeneity (e.g., how a model performs during anomalies vs. stable periods).

B. Structural Coherence

This dimension evaluates whether the forecast preserves the underlying temporal structure of the data, regardless of pointwise error:

• Key Metrics: Trend strength, seasonal strength, distributional properties, and randomness tests.
• Goal: Determine if a model's predictions maintain the correct trend direction and seasonal patterns, even if they smooth out noise or anomalies.
• Diagnostic Tools: Use tools like Q-Q plots to visualize error distributions across different windows. This helps distinguish between a model failing to capture dynamics versus a model correctly identifying and smoothing structural irregularities (outliers).

C. Decision-Level Relevance

Evaluation must consider the downstream utility of the forecast:

• Context: In domains like energy planning or finance, stakeholders often prioritize trend direction and risk management over exact replication of short-term fluctuations.
• Metrics: Evaluate based on asymmetric error costs, robustness under distributional changes, and impact on downstream tasks (e.g., anomaly detection, inventory planning).
• Example: A model that smooths out a transient spike (increasing MSE) might be superior for a trading strategy that relies on trend-following, whereas a model that tracks the spike exactly (lower MSE) might trigger false alarms.

3. Key Contributions

1. Critical Re-examination of the "GAME": The paper formally articulates how the current benchmark-driven culture creates a "metric monoculture" that incentivizes specialization to benchmark configurations rather than genuine scientific progress in understanding temporal dynamics.
2. Three-Dimensional Evaluation Framework: It introduces a structured approach to evaluation comprising Statistical Fidelity, Structural Coherence, and Decision-Level Relevance, arguing that all three are necessary for a holistic assessment.
3. Diagnostic Reporting over Leaderboard Ranking: The authors advocate for a shift from reporting a single "best" score to diagnostic reporting. This involves analyzing when and why models fail or succeed (e.g., performance across different regimes, noise levels, and structural conditions).
4. Rejection of the "Universal Champion": The paper challenges the notion of a single state-of-the-art model, suggesting that models should be evaluated relative to specific domain objectives and structural conditions.

4. Results and Evidence

Note: As this is a position paper/review rather than an experimental study proposing a new model, the "results" are derived from the analysis of existing literature and logical deduction.

• Analysis of Current Trends: The authors demonstrate that recent literature (e.g., works following the Informer architecture) focuses heavily on marginal MSE/MAE improvements on fixed datasets (ETTm1, ETTm2, etc.), often ignoring structural validity.
• Hypothetical Scenarios: The paper illustrates that a model could "win" a benchmark by reducing MSE by 0.003 while failing to capture a critical regime shift, thereby being useless in a real-world application.
• Visual Evidence (Figure 1): The paper uses a visual example (DLinear on ETTm2 oil temperature data) to show that aggregated metrics hide significant variability. A "Worst" window (high MSE) might contain a structural anomaly that the model correctly smoothed, while a "Best" window might simply be a period of low variance. Aggregating these masks the model's true structural behavior.
• Literature Synthesis: Citing works by Brigato et al. and Qiu et al., the paper reinforces that no single model dominates all datasets and that time series exhibit high heterogeneity (trend strength, seasonality, regime shifts) that current metrics fail to capture.

5. Significance and Future Impact

The significance of this paper lies in its potential to redirect the research culture of the LTSF community:

• From Optimization to Understanding: It shifts the goal from "winning the benchmark" to "understanding temporal dynamics."
• Practical Utility: By emphasizing decision-level relevance, it bridges the gap between academic research and real-world applications (e.g., energy grids, finance), where structural accuracy often outweighs pointwise precision.
• New Research Directions: It calls for the development of:
  • Evaluation protocols that include regime-specific performance.
  • Metrics that quantify structural preservation (trend/seasonality).
  • Benchmarks that reflect diverse real-world scenarios rather than a fixed set of canonical datasets.
• Cultural Shift: Ultimately, the paper argues that progress in data science should be measured by the generation of practical insight and the ability to act on temporal dynamics, not merely by the reduction of numerical error scores.

In summary, the paper serves as a foundational critique urging the community to stop optimizing for the "wrong game" (leaderboard superiority) and start designing evaluation practices that reflect the true complexity and utility of long-term time series forecasting.