Harmonic Dataset Distillation for Time Series Forecasting

The paper proposes Harmonic Dataset Distillation (HDT), a novel framework that leverages frequency-domain decomposition and harmonic matching to synthesize compact time series datasets, thereby overcoming the scalability and overfitting limitations of conventional methods while achieving strong cross-architecture generalization for large-scale forecasting.

The Gist

Imagine you are trying to teach a student how to predict the weather.

The Problem: The Library vs. The Textbook
In the past, you had a massive library (the original dataset) with millions of pages of historical weather data. To train your student (the AI model), you'd have them read every single page. This takes forever, costs a fortune in electricity, and requires a giant bookshelf to store it all.

The Old Solution: The "Highlighter" Method
Scientists tried to solve this by creating a "Dataset Distillation" method. Think of this as trying to shrink the library down to a single, tiny cheat sheet.

• The Flaw: Previous methods were like a student randomly highlighting sentences from different chapters. They might highlight a sentence about "rain in July" and another about "snow in December," but they missed the connection between them.
• The Result: The cheat sheet worked great if the student was studying exactly how you taught them. But if you gave that same cheat sheet to a different student (a different AI model) or asked them to predict a longer timeframe, they failed miserably. The cheat sheet was too specific to the first student's style and didn't capture the essence of the weather patterns.

The New Solution: HDT (Harmonic Dataset Distillation)
The authors of this paper propose a new way to make the cheat sheet, called HDT. Instead of looking at the weather data as a list of daily temperatures, they look at it as a song.

Here is how HDT works, using a musical analogy:

1. Turning Data into Music (The FFT)

Imagine the weather data isn't a list of numbers, but a complex piece of music played by an orchestra.

• The "loud" drums are the daily temperature swings.
• The "steady" bass line is the seasonal change (summer to winter).
• The "high-pitched" violins are the random, noisy gusts of wind.

The old methods tried to memorize specific notes (data points) in a specific order. HDT uses a tool called FFT (Fast Fourier Transform) to listen to the song and break it down into its individual instruments (frequencies).

2. Finding the Core Melody (Harmonic Matching)

HDT realizes that to understand the song, you don't need to memorize every single note played by the drummer. You just need to capture the core melody (the dominant harmonics).

• It identifies the most important "instruments" (the main frequencies that drive the weather patterns).
• It creates a tiny, perfect "MIDI file" (the distilled dataset) that contains only these core instruments.

3. Why This is a Game-Changer

Because HDT focuses on the melody (the global structure) rather than the specific notes (local windows), it has two superpowers:

• It Scales Up: If you want to make the cheat sheet bigger, you don't just add more random notes. You add more instruments to the orchestra, making the melody richer and more detailed. The old methods just added more noise; HDT adds more meaning.
• It Works for Everyone (Cross-Architecture Generalization): This is the magic part. Whether your student is a "Piano Player" (one type of AI model) or a "Guitar Player" (a completely different AI model), they can all read the same MIDI file. They all understand the melody. The old cheat sheets were written in a language only one specific student could read; HDT's cheat sheet is written in the universal language of music.

The Real-World Impact

The paper shows that this method:

1. Saves Time: You can train a massive AI model on a dataset that is 1,000 times smaller, cutting training time from hours to seconds.
2. Saves Money: You don't need to store terabytes of data; you just need the tiny "MIDI file."
3. Works Better: Even on huge, messy real-world data (like traffic sensors across an entire state), this method predicts the future more accurately than previous methods, especially when switching between different types of AI models.

In a Nutshell:
Previous methods tried to summarize a book by copying random sentences. HDT summarizes the book by writing down the plot outline and the main themes. Because it captures the story rather than just the words, anyone can read it and understand the story, no matter how they like to read.

Technical Summary

1. Problem Statement

Time Series Forecasting (TSF) faces critical challenges in the modern era due to the massive scale of real-world data (terabytes daily from sensors, healthcare, etc.) and the increasing computational cost of training large foundation models (e.g., TimesFM, Moirai).

Dataset Distillation (DD) aims to synthesize a small, compact dataset ($S$) that trains a model to perform comparably to the original large dataset ($X$). However, applying conventional DD methods to time series fails due to two main limitations:

1. Limited Scalability: Existing methods treat time series as a collection of independent local windows ("Window-based DD"). Increasing the synthetic dataset size merely elongates local patterns rather than capturing the broader global structure, leading to diminishing returns.
2. Architectural Overfitting: By optimizing only within local windows, these methods ignore global temporal dependencies (e.g., long-range periodicity). Consequently, the distilled data overfits to the specific inductive biases of the backbone model used during distillation, failing to generalize to different architectures (cross-architecture generalization).

2. Methodology: Harmonic Dataset Distillation (HDT)

The authors propose HDT, a novel approach that shifts the distillation optimization from the time domain to the frequency domain using the Fast Fourier Transform (FFT).

Core Concept

Instead of updating individual data points within local time windows, HDT decomposes the time series into sinusoidal basis functions. It identifies and updates the dominant harmonics (the top-$k$ frequency components with the largest amplitudes) that represent the core periodic structure of the data.

Key Components

1. Harmonic Matching (Global Structure Preservation):

   • Process: Both the original data ($X$) and synthetic data ($S$) are transformed into the frequency domain via FFT.
   • Selection: The top-$k$ frequency components (harmonics) are selected based on amplitude to filter out noise.
   • Alignment: A harmonic loss ($L_{harm}$) is computed to minimize the distance between the amplitude distributions of the selected harmonics in $X$ and $S$.
   • Mechanism: Since each harmonic is a global sinusoidal basis function, updating it modifies the entire synthetic sequence simultaneously. This ensures that updates preserve global temporal dependencies without disrupting local structures.
   • Theoretical Guarantee: The authors provide a proof (Theorem 1) showing that minimizing the difference in harmonic magnitudes bounds the difference in the Autocorrelation Function (ACF) between the original and synthetic data, theoretically guaranteeing the preservation of temporal dependencies.

2. Gradient Matching (Model Adaptation):

   • To ensure the distilled data trains models effectively, HDT employs a standard gradient matching objective ($L_{grad}$).
   • The selected harmonics are reconstructed into the time domain via Inverse FFT (iFFT) to create signals $X_H$ and $S_H$.
   • The model parameters are trained on these signals, and the distance between the training trajectories (gradients) of the original and synthetic data is minimized.

3. Optimization Objective:
   The final objective combines both losses:
   $$\min_{F_S} L_{grad} + \lambda L_{harm}$$
   where $\lambda$ balances the preservation of global structure (Harmonic Matching) and the ability to train specific models (Gradient Matching).

3. Key Contributions

• Novel Paradigm: Introduction of HDT, the first dataset distillation method specifically designed for time series that operates in the frequency domain to capture global structures.
• Theoretical Justification: A formal proof demonstrating that aligning harmonic distributions preserves the essential global temporal structure (autocorrelation) of the original time series.
• Solving Scalability & Overfitting: HDT addresses the "local-to-local" limitation of previous methods. By updating global harmonics, it scales effectively with dataset size and achieves strong cross-architecture generalization.
• Empirical Validation: Comprehensive experiments across diverse datasets and backbone architectures (Linear, Transformer, CNN) demonstrating state-of-the-art performance.

4. Experimental Results

The authors evaluated HDT on standard benchmarks (ETT, Electricity, Traffic) and a massive-scale dataset (CA, 200k+ points, 8.6k features).

• Cross-Architecture Generalization:
  • Previous methods (DC, MTT, CondTSF) often suffer severe performance degradation when the backbone used for distillation differs from the evaluation model (e.g., distilling with DLinear but evaluating with iTransformer).
  • HDT maintains robust performance across all architectures, showing minimal MSE increase in cross-architecture settings compared to fixed-architecture settings.
• Scalability:
  • As the synthetic dataset size ($M$) increases, HDT's performance consistently improves. In contrast, window-based methods saturate quickly, confirming HDT's ability to capture long-range context.
• Large-Scale Performance:
  • On the massive CA dataset, HDT significantly outperformed other distillation methods, approaching the performance of full-data training.
• Foundation Model Fine-Tuning:
  • HDT was used to fine-tune the large Moirai-Large (311M parameters) model.
  • Efficiency: It achieved an 80x speed-up compared to full fine-tuning while incurring only a 2.5% degradation in MSE.
• Computational Cost:
  • The overhead of HDT (including FFT/iFFT) is marginal compared to the gradient computations of the backbone models, especially as model complexity increases.

5. Significance

This work bridges the gap between the need for efficient data handling in large-scale time series applications and the limitations of current distillation techniques. By leveraging the frequency domain, HDT provides a model-agnostic representation of time series data that preserves global periodicity and dependencies.

Its ability to distill massive datasets into compact forms that work across diverse architectures (from simple linear models to complex Transformers) makes it highly practical for:

• Resource-constrained environments (e.g., edge devices, IoT).
• Online learning scenarios where data storage is limited.
• Efficient fine-tuning of large foundation models, drastically reducing training time and cost without sacrificing accuracy.

In summary, HDT represents a fundamental shift in how time series data is compressed for learning, moving from local window matching to global harmonic alignment.