🤖 AI

Filter then Attend: Improving attention-based Time Series Forecasting with Spectral Filtering

This paper proposes enhancing Transformer-based models for long time-series forecasting by integrating learnable spectral filters at the input stage, which improves performance, reduces model size, and enables better utilization of the full frequency spectrum while adding minimal parameters.

Original authors: Elisha Dayag, Nhat Thanh Van Tran, Jack Xin

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Elisha Dayag, Nhat Thanh Van Tran, Jack Xin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the future of a complex system, like the weather, traffic flow, or electricity usage, by looking at a long history of data. This is called Time Series Forecasting.

For a long time, the smartest computers (AI models) used a specific tool called a Transformer to do this. Think of a Transformer like a very attentive librarian who reads a huge stack of books (data) to find patterns. However, this librarian has a quirk: they are great at understanding the slow, steady storylines (low-frequency patterns) but they often miss the quick, sharp details or sudden spikes (high-frequency patterns) because they tend to "smooth things out" too much.

The paper "Filter Then Attend" proposes a simple but powerful fix: Give the librarian a pair of specialized glasses before they start reading.

Here is the breakdown of the paper's idea, using everyday analogies:

1. The Problem: The "Blurry" Librarian

The authors noticed that standard Transformer models have a "bias." They are like a camera with a slow shutter speed; they capture the general motion of a car well, but the details of the spinning wheels get blurry. In data terms, the models focus too much on slow trends and ignore the fast, high-frequency changes that are often crucial for accurate predictions.

2. The Solution: The "Spectral Filter" Glasses

The authors added a new step at the very beginning of the process. Before the data enters the Transformer (the librarian), it passes through a Learnable Spectral Filter.

The Analogy: Imagine you are listening to a song that has a lot of background static and a few important, high-pitched notes. If you put on a pair of noise-canceling headphones that are tuned to boost those high-pitched notes and cut out the muddy low frequencies, the song becomes much clearer.
How it works: This filter looks at the data in the "frequency domain" (like looking at a sound wave's spectrum). It learns which parts of the signal are important and amplifies them, while dampening the noise. Crucially, these glasses are learnable, meaning the computer figures out exactly what kind of "tuning" is needed for each specific dataset.

3. The Result: "Filter Then Attend"

The paper calls this new approach FilterFormer (and variations like iFilterFormer). The workflow is simple:

Filter: The data goes through the special glasses first.
Attend: The cleaned-up, sharpened data is then handed to the Transformer.

Because the data is already "prepped" and the high-frequency details are preserved, the Transformer can do its job much better.

4. Key Findings (The "Magic" of the Method)

The authors tested this on nine different real-world datasets (like electricity usage, traffic, and weather) and found some surprising benefits:

Better Accuracy: In many cases, adding these filters improved prediction accuracy by 5% to 10%. In the world of AI, a single-digit improvement is often a huge deal, usually requiring massive, complex overhauls. Here, it was just a small addition.
Smaller Models: Because the filter does so much of the heavy lifting, the main Transformer model doesn't need to be as "muscular." The authors found they could shrink the model's size (embedding dimension) and still get better results. It's like giving a runner better shoes so they don't need to be as naturally strong to win the race.
Cheap and Fast: The filter adds almost no extra weight to the computer's memory or processing power (only about 1,000 extra parameters). It's a "free lunch" in the AI world: you get better performance without paying a heavy cost in speed or memory.
Fixing the "Oversmoothing": The authors proved that the filter specifically helps the model see the "fast" parts of the data that the Transformer usually misses. Without the filter, the Transformer blurs out the sharp edges; with the filter, those edges stay sharp.

5. A Caveat: Garbage In, Garbage Out

The paper also notes that the filter isn't a magic wand for bad data. In one specific case (a traffic dataset with a broken sensor), the filter initially made things worse because the data itself was inconsistent between training and testing. However, once that bad data was removed, the filter worked perfectly. This proves the filter enhances the model's ability to learn, but it can't fix broken inputs.

Summary

The paper argues that Transformers are great, but they are a bit "low-pass" (they like slow things). By adding a simple, learnable frequency filter at the start, we can sharpen the data, allowing the Transformer to see the whole picture clearly. This results in models that are more accurate, smaller, and faster, making them better at predicting the future of complex systems like energy grids and traffic flows.

Technical Summary: Filter Then Attend

Problem Statement

Long Time-Series Forecasting (LTSF) has seen significant advancements through Transformer-based models, which often achieve state-of-the-art results. However, these models suffer from two primary limitations: a bias toward low-frequency components in the data and high computational and memory requirements. While recent works have attempted to integrate frequency learning into deep forecasting models, many rely on Multilayer Perceptrons (MLPs) to process filtered signals, failing to resolve the inherent inefficiencies of Transformer architectures. Furthermore, existing literature suggests that Transformers may act as low-pass filters, potentially "oversmoothing" signals and failing to capture high-frequency patterns essential for accurate long-term prediction.

Methodology

The paper proposes a simple yet effective architectural modification: placing a learnable spectral filter at the beginning of Transformer-based models. This approach, termed "Filter Then Attend," integrates frequency domain processing directly into the Transformer pipeline.

Core Architecture: FilterFormer

The authors introduce FilterFormer, a model that passes time series data through a learnable filter before applying standard Transformer attention mechanisms. The architecture consists of the following components:

Input Embedding and Patching: Following the PatchTST framework, the input time series is treated as independent univariate sequences. These are normalized and divided into overlapping or non-overlapping patches. Each patch is embedded into a latent space and augmented with positional embeddings.
Spectral Block: Before the attention mechanism, the embedded patches undergo spectral processing.
- The signal is transformed into the frequency domain using the Fast Fourier Transform (FFT).
- A learnable filter is applied via pointwise multiplication in the frequency domain. This filter is initialized as a random set of real parameters and transformed via rFFT.
- The filtered signal is converted back to the time domain using the Inverse Discrete Fourier Transform (IDFT).
- Unlike previous works (e.g., FilterNet) that process the filtered signal solely through an MLP, this method passes the filtered signal directly into the Transformer's attention block.
- Note: In the specific implementation of FilterFormer, a small MLP is included within the spectral block, but the authors note that for other variants (iFilterFormer, FilterLeddam), the MLP can be omitted to reduce parameters.
Attention Block: The filtered patches are processed by standard multi-head self-attention mechanisms. The model alternates between spectral blocks and attention blocks, with the ratio controlled by a hyperparameter $\alpha$ .
Output: The final representation is flattened and mapped to the prediction horizon via a linear head.

The authors also created variants of this approach applied to other base models: iFilterFormer (based on iTransformer) and FilterLeddam (based on Leddam).

Key Contributions

Integration of Learnable Filters: The paper demonstrates that adding a learnable frequency filter to the input of Transformer-based models significantly enhances their performance in LTSF tasks.
Efficiency and Parameter Reduction: The addition of the filter introduces approximately 1,000 parameters. Crucially, this addition allows for a reduction in the embedding dimension of the underlying Transformer. Consequently, the resulting models are often smaller and faster than their non-filtering counterparts while achieving superior or comparable accuracy.
Spectral Utilization Analysis: Through synthetic experiments, the authors provide evidence that the filters enable Transformers to better capture high-frequency patterns. They hypothesize that standard Transformers act as low-pass filters (causing oversmoothing), and the learnable filters counteract this by amplifying mid-to-high frequency components.
Empirical Validation: The method was validated across nine diverse datasets (including ETT, Electricity, Weather, Traffic, and Solar-Energy) and multiple prediction horizons.

Experimental Results

The authors evaluated their models against strong baselines, including iTransformer, PatchTST, Leddam, FilterNet, Fredformer, and DLinear.

Performance Gains: Across multiple datasets, the FilterFormer variants achieved relative improvements of 5–10% in forecasting performance (measured by MSE and MAE). For instance, on the Electricity (ECL) dataset, FilterFormer showed an 11.7% improvement over PatchTST.
Robustness: The improvements were consistent across different domains, though the authors noted that on the Traffic dataset, the model initially underperformed due to a specific outlier channel causing spectral discrepancies between training and testing sets. Removing this outlier restored the model's superior performance.
Efficiency: Table 6 in the paper highlights that FilterFormer models require significantly fewer parameters (e.g., 0.6M vs. 3.7M for PatchTST) and fewer FLOPs while maintaining competitive accuracy. This is attributed to the ability to use smaller embedding dimensions when filters are present.
Synthetic Experiments: On a synthetic signal containing low, mid, and high frequencies, the standard iTransformer struggled (MSE = 1.0e-1) due to frequency bias, whereas iFilterFormer (with the filter) achieved a significantly lower MSE (2.6e-4), effectively modeling the high-frequency components.

Significance and Claims

The paper positions "Filter Then Attend" as a "free lunch" in the context of deep time-series forecasting. The authors claim that:

The addition of a learnable filter is a minor architectural change that rarely degrades performance but has the potential to drastically improve it.
Filters act as a necessary building block to enable Transformers to utilize the full spectrum of data, specifically countering the low-pass nature of self-attention.
The method is not a panacea; its effectiveness is ultimately bounded by the capacity of the backbone Transformer. However, it allows these backbones to operate more efficiently.
The authors advocate for the learned filter as a basic configuration for future Transformer-based forecasting models, suggesting that further work could focus on developing more advanced, steerable, and interpretable filters.

The paper concludes that combining frequency filtering with attention mechanisms results in a model that is "stronger than the sum of its parts," offering a lightweight, effective solution for long-term time-series forecasting.