📊 statistics

Prediction Inference of Time Series with Standard ReLU Deep Neural Networks

This paper proposes a methodology using standard ReLU Deep Neural Networks to generate prediction intervals for time series that quantify both future variability and estimation uncertainty, supported by theoretical proofs of consistency and beta-mixing properties under dependent data conditions.

Original authors: Kejin Wu

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kejin Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Predicting the future is one of humanity's oldest and most persistent challenges. Whether a farmer checking the sky for rain, an economist watching stock markets, or a doctor monitoring a patient's vital signs, the goal is always the same: to look at what has happened and guess what comes next. In the world of data science, this task is known as time series forecasting. For decades, scientists have relied on mathematical models to find patterns in these sequences of numbers. The most traditional tools are built on the idea that the future is a simple, straight-line extension of the past. However, the real world is rarely so simple. Weather systems, financial crashes, and biological rhythms often twist and turn in complex, non-linear ways that straight lines cannot capture. When these traditional tools fail, researchers have turned to a more powerful, flexible approach: deep neural networks. These are computer systems inspired by the human brain, capable of learning intricate patterns from vast amounts of data. Yet, while these networks have become famous for their ability to predict, a critical piece of the puzzle has been missing. Knowing a prediction is one thing; knowing how much to trust it is another. Without a clear measure of uncertainty, a prediction is just a guess, no matter how sophisticated the machine that made it.

This is the specific problem addressed by a new study from the Department of Mathematics and Statistics at Loyola University Chicago. The researchers set out to build a method that not only uses deep neural networks to forecast time series but also rigorously quantifies the uncertainty of those forecasts. In the past, when scientists used these powerful networks, they often treated them as "black boxes," accepting the output without understanding the reliability of the result. The study introduces a new way to construct what are called prediction intervals. Think of a prediction interval not as a single number, but as a range of likely outcomes. If a weather model predicts a high of 75 degrees, a simple prediction might just say "75." A prediction interval would say, "We are 95 percent confident the temperature will be between 70 and 80." The challenge the researchers tackled was ensuring this range was wide enough to be accurate, especially when the amount of data available was small.

The team focused on a specific type of deep neural network that uses a standard activation function known as ReLU. This function helps the network learn by deciding which signals to pass through and which to ignore, much like a neuron firing or staying silent. The researchers first proved that these networks could consistently learn the underlying patterns in data that changes over time, even when that data is dependent on its own past. They then developed a sophisticated simulation technique, known as a forward bootstrap, to test their predictions. Imagine taking a single timeline of data, like a record of daily temperatures, and creating thousands of slightly different versions of that timeline by shuffling the random errors within it. By running their neural network on each of these thousands of variations, the researchers could see how much the predictions bounced around. This bouncing, or variability, revealed two things: the natural randomness of the future itself, and the uncertainty caused by the fact that the model was trained on a limited amount of data.

By combining these two sources of uncertainty, the researchers created what they call a "pertinent prediction interval." This method is designed to be more reliable than older techniques, particularly when the dataset is small. In their tests, they compared their new approach against standard methods used in statistics. They simulated three different types of complex, non-linear data patterns, ranging from simple curves to highly intricate, high-dimensional shapes. In every scenario, the neural network approach proved to be more accurate at finding the true future values than the traditional methods. More importantly, when it came to the prediction intervals, the new method consistently captured the true outcome more often than the older techniques. For example, in one simulation involving complex, high-dimensional data with a small sample size, the traditional method achieved only an 89.8% coverage rate, while the new neural network method achieved 95.5%. The study also applied this method to real-world data by analyzing a subset of the M4 competition dataset, specifically selecting seven yearly time series with lengths between 300 and 350 observations. The results held up: the neural network-based intervals were more stable and reliable, especially when the researchers had to guess the complexity of the data without knowing the exact rules beforehand.

The significance of this work lies in its ability to bring statistical rigor to the world of machine learning. For a long time, deep learning has been celebrated for its raw predictive power but criticized for its lack of transparency regarding error. This study bridges that gap. It shows that by carefully analyzing how a neural network behaves when trained on different versions of the same data, we can build a safety net around its predictions. This is crucial for fields where being wrong has serious consequences, such as managing water resources, modeling climate change, or making economic decisions. The researchers did not claim to have solved every problem in forecasting, nor did they suggest that their method works perfectly in every single situation. Instead, they provided a robust, mathematically sound framework that works well in practice, particularly when data is scarce or the patterns are complex. They demonstrated that with the right tools, we can not only predict the future with deep learning but also understand the limits of that prediction with a high degree of confidence.

Technical Summary: Prediction Inference of Time Series with Standard ReLU Deep Neural Networks

Problem Statement

The paper addresses the challenge of performing statistical prediction inference for non-linear autoregressive (NLAR) time series using Deep Neural Networks (DNN). While DNNs have demonstrated superior flexibility and universal approximation capabilities compared to traditional linear (ARMA) or non-parametric kernel methods, their application in time series forecasting often treats them as "black boxes." A critical gap exists in the rigorous statistical quantification of prediction uncertainty, specifically regarding the estimation variability arising from the model fitting process.

Classical prediction intervals (PIs) often fail to capture this estimation variability, leading to undercoverage in finite sample cases. Furthermore, existing methods for multi-step ahead forecasting in non-linear models often suffer from bias in optimal $L_2$ point predictions or fail to provide optimal $L_1$ predictions when data lacks finite second moments. The author aims to bridge this gap by developing a framework that utilizes standard ReLU DNNs to provide consistent point predictions and valid prediction intervals that account for both future variability and estimation uncertainty.

Methodology

1. Model Framework

The study focuses on a general autoregressive model:
$Y_t = f_0(Y_{t-1}) + \epsilon_t$
where $f_0$ is the unknown mean function and $\epsilon_t$ are i.i.d. innovation terms. The author employs a standard fully connected ReLU Deep Neural Network (DNN), specifically a Multi-layer Perceptron (MLP), as the estimator $\hat{f}_n$ for $f_0$ . The DNN class is defined by depth $L_n$ , width $H_n$ , and a bound on the output norm, allowing these parameters to grow with the sample size $n$ .

2. Theoretical Foundations

The paper establishes several theoretical properties under assumptions of strict stationarity, geometric $\beta$ -mixing, and smoothness of the target function (Hölder space):

Consistency: The DNN estimator is shown to be consistent with the true function $f_0$ in both $L_2$ and $L_\infty$ norms. The paper derives non-asymptotic error bounds for the DNN estimator, demonstrating that it can handle expanding domains and ranges of the target function.
Bootstrap Validity: The author proposes a forward bootstrap procedure. They prove that the bootstrap time series generated by the DNN estimator retains the $\beta$ -mixing property and converges to the same stationary distribution as the original series. This is a crucial condition for the validity of subsequent inference.

3. Prediction Intervals

Two types of prediction intervals are constructed:

Standard Quantile Prediction Interval (QPI): Based on the empirical distribution of the bootstrap forecast errors. The paper proves its asymptotic validity.
Pertinent Prediction Interval (PPI): This is the core methodological contribution. The PPI is constructed based on the distribution of predictive roots ( $R_{n+j} = Y_{n+j} - \hat{Y}_{n+j}$ $R_{n + j} = Y_{n + j} - \hat{Y}_{n + j}$ ). Unlike the QPI, which estimates the conditional distribution of the future value directly, the PPI estimates the distribution of the error term (the root).
- The PPI captures both the innovation variance and the estimation error variance.
- The method relies on the "pertinence" property: the distribution of the bootstrap predictive roots approximates the distribution of the real-world predictive roots.
- The construction involves re-estimating the DNN model on bootstrap samples to capture the variability of the estimator itself.

4. Algorithms

Algorithm 1: Generates multi-step ahead point predictions (optimal $L_1$ and $L_2$ ) and the standard QPI using the forward bootstrap.
Algorithm 2: Generates the Pertinent Prediction Interval (PPI). It involves a "burn-in" period to ensure stationarity in the bootstrap series, re-estimation of the DNN on bootstrap data, and the calculation of predictive roots to determine the interval bounds.

Key Results

Simulation Studies

The author conducted simulations on three non-linear models (including high-dimensional scenarios with $d=10$ ) with varying sample sizes ( $n=50, 200, 500$ ).

Point Prediction: The DNN-based point predictions consistently outperformed kernel estimators in terms of Mean Squared Error (MSE), particularly in high-dimensional settings where the "curse of dimensionality" affected kernel methods more severely.
Prediction Intervals:
- Coverage: The PPI consistently achieved coverage rates closer to the nominal level (e.g., 95%) compared to the QPI, especially in small sample sizes and complex non-linear models.
- Robustness: In high-dimensional cases (Model 2 and 3 with $d=10$ ), kernel-based PIs struggled significantly (e.g., coverage dropped to ~15% for kernel QPI in Model 3, $n=50$ ), whereas DNN-based PIs maintained robust performance (e.g., ~92% coverage).
- Length: While PPIs were generally wider than QPIs (due to capturing estimation uncertainty), they provided more reliable coverage.

Real Data Analysis

Using the M4 competition dataset (yearly time series), the author employed a moving window approach to evaluate performance across different window sizes ( $w$ ) and input dimensions ( $d$ ).

Stability: The DNN-based PIs, particularly the PPI, demonstrated greater stability across varying window sizes and model orders compared to kernel-based methods.
Performance: When the ratio of window size to dimension ( $w/d$ ) was small, kernel estimators showed degraded performance (lower coverage, higher variance in interval length). The DNN approach remained stable, suggesting it is a superior option when the true model order is unknown or when data is scarce relative to dimensionality.

Significance and Claims

The paper claims to provide a rigorous statistical framework for using standard ReLU DNNs in time series forecasting, moving beyond their use as mere "black box" predictors.

Uncertainty Quantification: The primary contribution is the demonstration that Pertinent Prediction Intervals (PPI) can be successfully constructed using DNN estimators. This allows for the quantification of estimation variability, addressing a known weakness of standard bootstrap methods in non-parametric settings.
Robustness to Dimensionality: The study provides empirical and theoretical evidence that DNNs are more robust to the curse of dimensionality than standard kernel methods in time series contexts, making them suitable for high-dimensional non-linear forecasting.
Minimal Assumptions: The validity of the PPI is established under minimal assumptions, specifically relying on the convergence of the predictive roots and the preservation of mixing properties in the bootstrap world.
Practical Utility: The author suggests that in practice, where sample sizes may be limited and model orders unknown, the DNN-based PPI offers a more reliable and stable approach for constructing prediction intervals compared to traditional kernel-based methods.

The paper concludes that while the universal approximation property of DNNs is well-known, this work extends the understanding to statistical inference, proving that DNNs can yield consistent point predictions and valid, uncertainty-aware prediction intervals for dependent data.