🤖 machine learning

Super-Linear: A Lightweight Pretrained Mixture of Linear Experts for Time Series Forecasting

Super-Linear is a lightweight, scalable mixture-of-experts model that replaces complex deep architectures with frequency-specialized linear experts and a spectral gating mechanism to achieve efficient, robust, and interpretable time series forecasting with strong zero-shot performance.

Original authors: Liran Nochumsohn, Raz Marshanski, Hedi Zisling, Omri Azencot

Published 2026-05-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Liran Nochumsohn, Raz Marshanski, Hedi Zisling, Omri Azencot

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Swiss Army Knife" vs. The "Specialized Team"

Imagine you need to predict the weather, stock prices, or energy usage for the future. Most modern AI models trying to do this are like giant, heavy supercomputers. They are incredibly powerful but require massive amounts of electricity, take a long time to run, and are hard to understand. They try to learn everything about time and patterns all at once using complex, deep neural networks (like Transformers).

The authors of this paper asked: "Do we really need a supercomputer to do this?"

They built Super-Linear, a model that is the opposite of those giants. It is tiny (only 2.5 million parameters, compared to hundreds of millions in other models), incredibly fast, and surprisingly accurate. Instead of trying to be a "deep" brain, it acts like a highly organized team of specialists.

The Core Problem: "Frequency Confusion"

To understand Super-Linear, you first need to understand the problem it solves.

Time series data (like electricity usage) is full of rhythms.

Some rhythms happen every hour (like people turning lights on and off).
Some happen every day (like traffic patterns).
Some happen every week (like weekend sales).

If you try to teach a single, simple math equation (a "linear model") to predict all of these at once, it gets confused. It's like trying to teach one person to be a master drummer, a master violinist, and a master singer simultaneously. They might get the beats mixed up, leading to bad predictions. The paper calls this "frequency confusion."

The Solution: A "Mixture of Experts"

Super-Linear solves this by using a Mixture of Experts (MoE) approach. Think of it not as one giant brain, but as a manager who runs a team of specialized workers.

The Specialists (The Experts):
Instead of one model trying to do everything, Super-Linear has many small, simple models.
- Expert A is trained only on hourly patterns.
- Expert B is trained only on daily patterns.
- Expert C is trained only on weekly patterns.
- There are also "Complementary Experts" who handle the messy stuff that doesn't fit a perfect rhythm, and even a "Naive Expert" who just guesses the last value (a safe fallback).
The Manager (The Gating Mechanism):
When you give the model a new set of data to predict, a lightweight "manager" looks at the data's rhythm (its frequency).
- If the data looks like it has a strong daily rhythm, the manager says, "Hey, Expert B, you handle this!"
- If the data is messy, the manager might say, "Let's ask Expert C and the Naive Expert to help out."

The manager doesn't force everyone to work; it only picks the top few experts needed for that specific job. This makes the system incredibly efficient.

The Secret Sauce: "Resampling" (The Time-Travel Trick)

The paper highlights a very clever trick used during training called Resampling.

Imagine you have a video of a bird flying.

If you watch it at normal speed, you see the flapping of wings.
If you watch it in slow motion, you see the detailed muscle movement.
If you watch it in fast forward, you see the general path.

Most AI models are trained on data at just one speed (usually the original speed). Super-Linear, however, takes its training data and artificially speeds it up or slows it down (resampling) to create thousands of different "versions" of the same pattern.

This teaches the model: "Hey, a daily pattern looks like a 1-hour pattern if you slow time down, and a 10-minute pattern if you speed time up."

By doing this, the model learns to recognize the shape of the rhythm, regardless of how fast or slow the data is coming in. This is why Super-Linear is so good at handling data it has never seen before (Zero-Shot), even if that data comes from a different country or a different sampling rate.

Why It Matters (The Results)

The paper compares Super-Linear to the current "giants" (like Chronos, TimesFM, and Timer-XL) and finds:

Speed: It is hundreds of times faster to run. While a giant model might take seconds to process a batch of data, Super-Linear does it in milliseconds.
Size: It is tiny. It uses a fraction of the memory and computing power.
Accuracy: Despite being small and simple, it beats or matches the massive models on many standard benchmarks.
Interpretability: Because it uses simple linear math and you can see which expert was chosen, you can actually understand why it made a prediction. You can look at the "Manager" and say, "Ah, it chose the 'Daily' expert because the data has a daily cycle."

Summary Analogy

Imagine you are trying to predict the tide.

The Old Way (Transformers): You hire a massive team of 100 PhD oceanographers with supercomputers to analyze every wave, wind, and moon phase simultaneously. It's expensive and slow.
The Super-Linear Way: You hire a small team of 37 specialists. One knows only about 12-hour tides, one knows only about 24-hour tides, and one knows about storm surges. You have a smart foreman who looks at the current water and instantly calls the one specialist who knows that specific rhythm.

The result? You get the same (or better) prediction, but you do it with a fraction of the cost, in a fraction of the time, and you can actually explain how the foreman made the decision.

Technical Summary: Super-Linear

Problem Statement

Time series forecasting (TSF) is critical across domains such as energy, finance, and healthcare. While recent large pre-trained models (e.g., Chronos, Time-MoE, TimesFM) have demonstrated strong zero-shot (ZS) generalization capabilities, they suffer from significant computational costs, large parameter counts, and limited interpretability. Conversely, while linear models are efficient and fast, a single linear model often lacks the expressivity to generalize across diverse datasets with varying periodic patterns. This limitation leads to "frequency confusion," where a single set of weights struggles to represent multiple overlapping temporal frequencies, degrading accuracy in zero-shot settings.

Methodology

The authors introduce Super-Linear, a lightweight, sparse Mixture-of-Experts (MoE) model designed to bridge the gap between the efficiency of linear models and the adaptability of deep architectures.

Core Architecture

Super-Linear replaces deep, non-linear backbones with a modular architecture consisting of:

Frequency-Specialized Linear Experts: The model comprises $N$ linear experts. Most are "frequency experts," each trained to specialize in a specific dominant frequency regime ( $\hat{\omega}_i$ ). These are implemented using RLinear (a linear layer combined with instance normalization via RevIN) to handle distribution shifts.
Complementary Experts: A set of non-frequency-specific linear experts trained jointly to capture residual temporal signals not handled by the frequency experts.
Heuristic Experts: Non-learnable experts, specifically a "naïve" (last value) and "mean" (input average) forecaster, are included to provide robustness in low-frequency or short-sequence regimes.
Spectral Gating Mechanism: A lightweight, interpretable router dynamically selects the relevant experts for a given input. It computes gating weights based on the input's spectral signature (normalized periodogram derived from the Fast Fourier Transform). The mechanism employs sparse gating, activating only the top- $k$ scoring experts per input.

Training Framework

Super-Linear utilizes a two-stage training process:

Stage 1 (Frequency Expert Pre-training): Each frequency expert is trained independently on data aligned with its designated fundamental frequency. Crucially, the authors employ extensive resampling augmentations (linear interpolation) during this stage. This is not merely for data augmentation but to expose each expert to a diverse abundance of frequency instances, ensuring the model learns robust frequency representations rather than dataset-specific patterns.
Stage 2 (Router and Complementary Training): The weights of the pre-trained frequency experts are frozen. The MoE router and the complementary expert layers are trained jointly. The router learns to dynamically select the most suitable expert based on the input signal's spectral characteristics.

Adaptability

The model handles varying input lengths (lookbacks) and forecast horizons through:

Resampling Heuristics: Inputs shorter than the training length (512 steps) are upsampled via linear interpolation; longer inputs are adaptively downsampled to preserve signal energy and align frequencies with the expert span.
Autoregressive Roll-out: Long forecast horizons are handled via autoregressive generation.

Key Contributions

Efficient Zero-Shot Forecasting: Super-Linear achieves competitive zero-shot performance with only 2.5 million parameters, significantly smaller than Transformer-based baselines (e.g., Timer-XL has 84M parameters).
Frequency-Aware Design: The paper posits that frequency confusion is a primary bottleneck for linear models in zero-shot settings. By decomposing the forecasting task into frequency-specialized experts, the model mitigates this interference.
Data Construction over Architecture: The authors demonstrate that resampling and input adaptation are more critical to performance than architectural complexity. Training on diverse frequency regimes via resampling allows the model to generalize effectively without massive parameter counts.
Interpretability: Unlike complex deep MoE models, Super-Linear's single-layer expert structure and spectral gating allow for direct analysis of expert selection, revealing clear links between sampling rates, seasonal patterns, and activated experts.

Experimental Results

The model was evaluated on standard benchmarks (LTSF: ETT, Electricity, Traffic, Weather) and the comprehensive GIFT-Eval benchmark.

Zero-Shot Performance: Super-Linear reduces average Mean Squared Error (MSE) by up to 26.2% compared to prominent models like Chronos and TimesFM. It outperforms Timer-XL in MSE and MAE on more than twice as many benchmarks while being only 3% of its size.
GIFT-Eval: On the GIFT-Eval benchmark, Super-Linear achieves a competitive MASE score (0.857), surpassing or matching larger foundation models like VisionTS, Moirai, and TTM.
Efficiency: Super-Linear offers substantial speedups. It is 8.3x faster than Chronos-Bolt-Small and 383.3x faster than Time-MoE for a batch size of 64, while maintaining competitive accuracy.
Robustness: The model demonstrates superior robustness to varying sampling rates (e.g., 10, 20, 30, 40, 50 minutes) compared to models that rely heavily on hourly-sampled training data.
Ablation Studies: Removing the resampling strategy during training caused the most significant performance degradation (MASE increased from 0.857 to 1.603), confirming that frequency diversity in data construction is the primary driver of success.

Significance and Claims

The paper claims that Super-Linear demonstrates that linear models, when combined with a sparse mixture-of-experts architecture and frequency-aware data construction, can rival or surpass large, complex pre-trained foundation models.

The authors emphasize that the key to this success is not architectural complexity but rather data construction—specifically, the use of resampling to create frequency diversity and the use of spectral gating to dynamically assemble predictions. This approach offers a path toward forecasting models that are:

Scalable: Running efficiently on a single GPU with minimal parameters.
Interpretable: Allowing users to understand which frequency patterns drive specific predictions.
Generalizable: Capable of zero-shot transfer across diverse domains and sampling rates without fine-tuning.

The paper concludes that while linear experts limit performance on highly nonlinear or chaotic series, the proposed framework provides a robust, efficient, and transparent alternative to the current trend of scaling up Transformer-based time series models.