📊 statistics

Scalable estimation of VARMA models

This paper introduces a scalable estimation framework for high-dimensional VARMA models that achieves near-linear computational cost independent of series length by leveraging partial-autocorrelation reparametrization and Fourier-based sufficient statistics, thereby enabling likelihood-based estimation to outperform traditional VAR approaches in dimensions where classical methods fail.

Original authors: Daniel Paulin, Victor Elvira

Published 2026-08-07

📖 9 min read🧠 Deep dive

CC BY 4.0

Original authors: Daniel Paulin, Victor Elvira

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the future, but instead of looking at a crystal ball, you are looking at a tangled web of moving parts. In the world of data science, this is called time series analysis. It's the art of looking at how things change over time—like the temperature outside, the price of your favorite sneakers, or the number of people visiting a park—to guess what will happen next. Usually, these things don't move in isolation; they dance together. When the temperature rises, ice cream sales go up, and when it rains, umbrella sales spike.

For a long time, scientists had a powerful tool to model these dances called VARMA (Vector Autoregressive Moving Average). Think of VARMA as a super-precise dance instructor that understands not just how a dancer moves today based on yesterday (the "autoregressive" part), but also how a sudden push or a slip from the past still affects the dancer's balance right now (the "moving average" part). However, there was a catch: this instructor was incredibly slow and clumsy. If you tried to teach it a dance with too many dancers (too many variables) or a dance that went on for too long (too much data), the instructor would get confused, crash, or give up entirely. It was like trying to solve a massive jigsaw puzzle while someone kept shaking the table. Because of this, most people settled for a simpler, less accurate instructor called VAR, which only looked at the past steps and ignored the slips and pushes, just to get the job done without breaking a sweat.

Now, two researchers, Daniel Paulin and Víctor Elvira, have built a new, super-fast version of that dance instructor. They figured out a way to teach the VARMA model to be just as smart as the old one but as quick as the simple one. They did this by changing how the model "thinks" about the dance steps. Instead of trying to memorize every single step of the entire history of the dance every time it makes a guess, they taught it to summarize the whole history into a few key "reference sheets" (called sufficient statistics) before the game even starts. Once those reference sheets are ready, the model can make predictions at lightning speed, regardless of how long the dance has been going on. They also invented a special "safety net" (a mathematical reparameterization) that guarantees the model never gets confused or loses its balance, even when the dance gets very complex.

In their experiments, they tested this new method on real-world data, from predicting how many people would buy cereal based on its price, to forecasting hourly weather patterns in Singapore and air quality in Beijing. The results were impressive: their new method was able to handle huge amounts of data and many variables (up to 40 different things at once) without crashing. It was often more accurate than the simpler methods people usually use, and it managed to capture those tricky "slips and pushes" (the moving average parts) that the simpler models miss. In fact, on some difficult data, the old methods would completely fail and give nonsensical answers, while this new approach kept dancing perfectly. They showed that you don't have to choose between being smart and being fast anymore; you can have both.

The Big Idea: Making the Impossible, Easy

The core problem the authors tackled is that VARMA models are the "gold standard" for predicting complex, interconnected systems, but they are notoriously difficult to use. Imagine you are trying to predict the weather in a city where the wind, rain, and temperature all influence each other. A simple model might say, "If it rained yesterday, it will rain today." But a VARMA model knows that "If it rained yesterday and the wind blew from the north and the temperature dropped, the rain might stop today, but only if the humidity was high." This extra layer of understanding makes VARMA much more accurate, but it also makes the math incredibly heavy.

For decades, the only way to train these models was to look at the entire history of data every single time the model made a small adjustment. It's like trying to learn a song by reading the entire sheet music from start to finish every time you hit a wrong note. If the song is 10 minutes long, that's fine. But if the song is 10 hours long, or if you are trying to learn 50 songs at once, you would spend all your time reading and no time singing. This made VARMA models impractical for anything but the smallest, simplest problems.

The Solution: The "Reference Sheet" Revolution

Paulin and Elvira's breakthrough is a new framework that changes the rules of the game. Instead of reading the whole sheet music every time, they realized you can write down a reference sheet (what they call "sufficient statistics") just once at the beginning. This reference sheet summarizes all the important patterns in the data. Once the reference sheet is written, the model can practice and improve without ever looking at the original data again.

Here is how they made it work:

The Safety Net (Partial-Autocorrelation Reparametrization):
One of the biggest headaches with VARMA models is that they can easily get "unstable." Imagine a dancer who starts spinning faster and faster until they fly off the stage. In math terms, this is called a lack of "stationarity" or "invertibility." To fix this, the authors used a clever mathematical trick called partial-autocorrelation reparametrization. Think of this as putting the dancer in a harness. No matter how much they try to spin out of control, the harness (the math) gently pulls them back to a safe, stable rhythm. This means the computer doesn't have to waste time checking if the model is going to crash; it's guaranteed to be safe by design.
The Reference Sheets (Sufficient Statistics):
They showed that to train the model, you don't need the raw data (the thousands of daily temperature readings). You only need a few pre-calculated numbers that summarize the relationships between the variables. It's like summarizing a 500-page novel into a one-page plot summary. Once you have that summary, you can figure out the story's ending without re-reading the book. This allows the model to train on massive datasets (like years of hourly weather data) in seconds, rather than hours or days.
The Fourier Magic (Parseval Evaluation):
To make the reference sheets even faster to use, they used a technique from signal processing called the Fourier transform (or Parseval's theorem). Imagine trying to count every single grain of sand on a beach. It would take forever. But if you could turn the sand into a wave and measure the wave's height, you could figure out the total amount of sand instantly. This mathematical trick allows the model to calculate complex patterns much faster, making it possible to use very long "memory" (looking far back in time) without slowing down.

What They Found: Speed Meets Accuracy

The authors tested their new method on three very different real-world challenges to see if it actually worked.

The Retail Test (Dominick's Data):
They tried to predict how many units of food (like cereal and soup) people would buy, based on the price. This is a VARMAX problem because it involves an outside factor (price) influencing the sales. The results showed that including the "moving average" part of the model was crucial. The simpler models that ignored the "slips and pushes" of the past were much less accurate. The new method predicted sales almost perfectly, while the old methods struggled or failed completely when the data got too complex.
The Weather Test (Singapore):
They looked at 11 different weather variables (temperature, humidity, wind, etc.) recorded every hour. Weather has a strong daily cycle (it's hot during the day, cool at night). The new method, which included a special "seasonal" feature, captured this daily rhythm beautifully. It used far fewer parameters (simpler math) to get the same or better results than the complex, non-seasonal models. It was like finding a shortcut that led to the same destination but took less energy.
The Air Quality Test (Beijing):
They analyzed ozone levels at 12 different monitoring stations. This data is messy and interconnected. Here, they tested a common belief: that "sparse" models (which try to ignore most connections and only keep the big ones) are better for complex data. Surprisingly, they found that dense models (which keep all the connections, even the tiny ones) were actually better. The air quality signal was spread out across many small connections, and ignoring them made the predictions worse. Their new method, which handles all these connections efficiently, beat the sparse models by a significant margin.

The Verdict

The paper demonstrates that VARMA models are not dead; they just needed a better engine. By combining a safety harness, a reference sheet, and a speed-boosting magic trick, the authors have made it possible to use these powerful models on large, complex datasets that were previously out of reach.

They proved that you can have a model that is both statistically efficient (very accurate) and computationally scalable (very fast). In their tests, the new method stayed close to the "oracle" (the perfect, theoretical best prediction) even when the data was huge and messy. It outperformed the standard tools that practitioners have been relying on for years, especially in situations where the data has complex patterns like daily cycles or external drivers like price.

The authors are careful to note that while their method is a huge step forward, it is not a magic wand that solves every problem. They showed that for certain types of data, the "dense" approach works best, while for others, the "sparse" approach might still be useful. But for the complex, interconnected, and seasonal data that makes up much of our modern world, this new framework offers a way to finally unlock the full power of VARMA models. It turns a tool that was once too heavy to lift into one that is light enough to carry anywhere, ready to help us understand and predict the complex dances of our world.

Technical Summary: Scalable Estimation of VARMA Models

Problem Statement
Vector Autoregressive Moving-Average (VARMA) models are theoretically superior to pure Vector Autoregressive (VAR) models for multivariate time series because they capture cross-variable dependencies and short-term dynamics with fewer parameters. However, VARMA estimation has historically been deemed impractical for high-dimensional series ( $d \gtrsim 10$ ) due to three primary barriers:

Non-convexity: The likelihood surface is non-convex, leading to convergence failures in standard optimization.
Identifiability: Parameters are only identified up to equivalence classes, requiring complex canonical forms (e.g., Echelon form) that involve combinatorial searches.
Computational Cost: Standard Maximum Likelihood Estimation (MLE) via recursive innovations or Kalman filters requires $O(T)$ operations per gradient evaluation, where $T$ is the series length. This linear scaling makes fitting dense VARMA models intractable for long series or high dimensions.

Consequently, practitioners often default to VAR models, even when a moving-average (MA) term would be more parsimonious.

Methodology
The authors propose a scalable estimation framework for dense VARMA $(p, q)$ models that decouples the per-iteration optimization cost from the series length $T$ . The framework relies on three core technical innovations:

Partial-Autocorrelation (PAR) Reparametrization:
Instead of optimizing over the raw coefficients $(\Phi_{1:p}, \Theta_{1:q})$ , the method maps unconstrained Euclidean parameters $(M^\Phi_{1:p}, M^\Theta_{1:q})$ to the stable set using a smooth bijection based on the multivariate Levinson recursion (Ansley and Kohn, 1986; Heaps, 2023). This transformation guarantees that stationarity and invertibility hold by construction, eliminating the need for projection steps or augmented Lagrangian penalties during optimization.
Sufficient Statistics and Truncation:
The method utilizes a truncated VARMA $(p, q, m)$ model, approximating the infinite-VAR representation of the process at lag $m$ . Crucially, the loss functions depend on the data only through fixed-size sufficient statistics (sample autocovariances and cross-covariances) computed in a single pre-pass.
- Regularized Least Squares (RLS): Minimizes a prediction error loss with Gaussian shrinkage priors on the PAR parameters.
- $\Sigma$ -Marginalized MAP: Extends the framework to a full Bayesian setting by placing an inverse-Wishart prior on the noise covariance $\Sigma$ and marginalizing it out analytically. This yields a loss function that adapts to the noise structure without explicitly optimizing $\Sigma$ .
Fourier (Parseval) Evaluation:
To avoid the $O(m^2 d^2)$ storage and $O(m^2 d^3)$ computation required to materialize the second-moment block matrix of the truncated coefficients, the authors employ a Parseval (Fourier) evaluation. By exploiting the block-Toeplitz structure of the autocovariances, the quadratic form is evaluated in the frequency domain. This reduces the per-gradient evaluation cost to near-linear in the truncation length $m$ ( $O(m \log m \cdot d^2 + m d^3)$ ), making large $m$ regimes tractable.

Key Contributions

Scalable Framework: The first dense, likelihood-motivated VARMA estimator where the per-iteration cost is independent of the series length $T$ . It replaces the $O(T)$ gradient cost of MLE with a one-time $O(T \log m)$ pre-pass.
Statistical Guarantees: The authors prove that both estimators recover the infinite-VAR representation of the true process at a near-parametric rate in fixed dimension. This demonstrates that the truncation does not introduce asymptotic bias; the estimators converge to the true dynamics as $T$ and $m$ grow.
Practical Extensions: The sufficient-statistics machinery extends naturally to:
- Seasonal VARMA: Multiplicative seasonal models for periodic data.
- VARMAX: Models with exogenous regressors (e.g., price in demand forecasting).
- Rolling Windows: Incremental updates for refitting models as new data arrives.

Results
The paper evaluates the estimators (RLS and $\Sigma$ -marginalized MAP) on synthetic and real-world datasets:

Synthetic Data: In dimensions $d \in \{10, 20, 40\}$ , the $\Sigma$ -marginalized MAP estimator consistently outperforms isotropic-noise RLS and sparse-VARMA baselines (e.g., bigtime). Crucially, standard MLE implementations (e.g., MTS in R) frequently fail to converge or return non-invertible fits with diverging forecasts in these dimensions, whereas the proposed method remains stable.
Retail Demand (VARMAX): On Dominick's scanner data, conditioning demand on price (exogenous regressor) significantly reduces forecast error. The moving-average term proves decisive, with the full VARMAX model outperforming VARX and sparse baselines.
Meteorological Data (Seasonal): On Singapore hourly weather data, the seasonal VARMA captures diurnal cycles more parsimoniously than dense VARMA or VAR models. The seasonal moving-average factor provides a significant accuracy boost over non-seasonal alternatives.
Air Quality (Sparsity vs. Structure): On Beijing ozone data, the dense seasonal VARMA outperforms sparse estimators. The authors argue that for this data, the signal is diffuse (many small cross-station couplings) rather than sparse; thus, $\ell_1$ -based sparsity penalties discard usable signal, whereas the dense $\ell_2$ shrinkage of the proposed method preserves it.

Significance and Claims
The paper claims to bring likelihood-based VARMA estimation to the problem sizes (high dimension, long series) where practitioners have historically relied on VAR models. By removing the computational barrier of $O(T)$ scaling and ensuring stability through PAR reparametrization, the method allows for the use of more parsimonious and statistically efficient VARMA models. The authors modestly note that while they provide statistical guarantees for the estimators, the optimization is performed numerically via L-BFGS on a non-convex objective, and convergence to the global optimum is not certified, though the recovery guarantees apply to any near-minimizer. The work suggests that for many real-world, densely coupled time series, a structured dense seasonal VARMA is more accurate and parameter-economical than sparse alternatives.

The Big Idea: Making the Impossible, Easy

The Solution: The "Reference Sheet" Revolution

What They Found: Speed Meets Accuracy

The Verdict

Technical Summary: Scalable Estimation of VARMA Models

More like this