Technical Summary: Fidel-TS: A High-Fidelity Multimodal Benchmark for Time Series Forecasting

Problem Statement

The evaluation of time series forecasting models is currently hindered by a lack of high-quality benchmarks, leading to overestimated assessments of model progress. Existing benchmarks suffer from three primary categories of issues:

1. Unimodal Limitations: Traditional datasets (e.g., ETT, ECL, M4) are often small-scale, outdated, and feature vague variable compositions. Crucially, their long-term public availability has created a significant risk of pre-training contamination for Large Language Models (LLMs), potentially inflating their perceived forecasting capabilities.
2. Multimodal Flaws: Early multimodal benchmarks (e.g., TimeMMD) inherit contamination risks and introduce new methodological failures. They often rely on retrospectively retrieved textual data (e.g., news, reports), causing temporal leakage (accessing future information) and description leakage (text explicitly stating ground truth). Furthermore, they frequently utilize low-frequency data (monthly/weekly), limiting data volume and preventing the evaluation of complex, high-frequency scenarios.
3. Structural Ambiguity: Existing benchmarks often fail to clearly distinguish between "Subjects" (e.g., specific sensors or regions) and "Channels" (variables), making it difficult to rigorously evaluate model generalization to new, unseen entities within a system.

Methodology: The Fidel-TS Framework

To address these shortcomings, the authors formalize three core principles of high-fidelity benchmarking and construct Fidel-TS, a new large-scale benchmark built upon them.

Core Principles

1. Data Sourcing Integrity: The benchmark exclusively utilizes real-time, authentication-protected API data streams. This ensures data recency, sufficiency, and high frequency (sub-hourly), while inherently mitigating pre-training contamination by sourcing data from restricted APIs rather than public repositories.
2. Leak-free Design: Textual modalities are restricted to verifiably scheduled exogenous information, such as weather forecasts and control events (e.g., maintenance schedules). This ensures that textual data available at the forecasting timestamp is strictly verifiable, preventing temporal and description leakage common in open-ended news retrieval.
3. Structural Clarity: The benchmark implements a strict demarcation between Subjects (distinct entities like sensors or rooms) and Channels (variables like speed or temperature). This architecture enables rigorous evaluation of generalization by training on observed subjects and testing on held-out, unseen subjects.

Data Curation Pipeline

The construction of Fidel-TS involves a pipeline that translates raw API streams into structured datasets:

• Imperfection Handling: Real-world sensor noise and missing values are preserved. Short gaps are interpolated, while long gaps are treated as "Sensor Downtime" events, marked with zero values and corresponding textual descriptions (e.g., "sensor downtime"), transforming data quality issues into contextual features.
• Textual Integration: Static metadata (dataset/subject/channel descriptions) and dynamic, time-aligned events (weather, control logs) are consolidated into a Dynamic Event Database, ensuring strict temporal alignment with numerical timestamps.
• Evaluation Scenarios: The benchmark offers multiple dataset variations:
  • -h: Downsampled to hourly frequency for uniformity.
  • -mini: Smaller subsets created via importance sampling (stratified by multimodal dependency) to make LLM evaluation computationally feasible without introducing bias.
  • -obs / -hid: Observed (training) and Hidden (testing) subsets to evaluate subject-level generalization (zero-shot transfer to new sensors/rooms).

Key Contributions

1. Formalization of Principles: The paper establishes a formal framework for high-fidelity time series benchmarking, emphasizing data integrity, leak-free design, and structural clarity.
2. Fidel-TS Release: The introduction of a new, large-scale benchmark containing fresh, high-frequency, leak-free data across diverse domains (Photovoltaics, Renewable Energy, Atmospheric Physics, Traffic, Power Grid, HVAC).
3. Unified Evaluation Framework: A comprehensive framework compatible with diverse modalities (unimodal, multimodal, LLM-based) and architectures, enabling fair comparison across the spectrum of forecasting models.

Experimental Results

The authors evaluated a wide range of models, including domain-specific unimodal models (DLinear, PatchTST, iTransformer), foundation models (Chronos, TimeMoE, Sundial), multimodal models (GPT4MTS, FIATS), and various LLMs (Qwen series, DeepSeek-R1, Chattime).

Key Findings

1. Unimodal Performance: No single unimodal model consistently dominates across all datasets. Furthermore, Time Series Foundation Models (TSFMs) fail to exhibit the claimed robust zero-shot forecasting ability; their performance degrades significantly as the forecast horizon extends, often underperforming specialized models trained on specific datasets.
2. Multimodal Gains: The effectiveness of multimodal forecasting is highly dependent on model architecture and the relevance of textual inputs. Models like FIATS, which leverage text within the prediction horizon as explicit state guidance, achieve state-of-the-art performance. Conversely, models that treat text merely as historical covariates (e.g., GPT4MTS) often fall behind unimodal baselines. Ablation studies confirm that models actively utilize semantic context rather than treating text as a generic regularizer.
3. LLM Limitations: While LLM performance correlates with general reasoning abilities, even domain-finetuned variants remain unstable. Under realistic, context-rich, and leakage-free long-term settings:
   • LLMs employing direct prompting suffer from severe precision and reliability issues.
   • Pass rates drop significantly with longer prediction horizons, multivariate inputs, and dense textual contexts.
   • Most LLMs lag behind dedicated forecasting models, suggesting previous benchmarks may have offered overly simplified tasks or suffered from data leakage.

Significance

The paper argues that Fidel-TS reveals a performance landscape distinct from previous benchmarks, exposing evaluation biases inherent in prior works. By establishing a rigorous, leak-free standard, the authors aim to:

• Correct the overestimation of model capabilities, particularly regarding LLMs and zero-shot foundation models.
• Provide a more realistic assessment of generalization capabilities through subject-level splitting.
• Encourage the research community to develop more reliable, truly generalizable benchmarking methods that align with real-world deployment scenarios.

The work concludes that while progress in time series forecasting is evident, current evaluation standards have not kept pace with model advancements, necessitating the high-fidelity approach presented in Fidel-TS.