📊 statistics

Interpretable models for forecasting high-dimensional functional time series

This paper proposes an interpretable forecasting framework for high-dimensional functional time series that combines a functional analysis of variance (FANOVA) decomposition with a functional factor model to separate deterministic structural effects from stochastic trends, thereby improving forecast accuracy and providing transparent insights for policy decisions, as demonstrated on Japanese subnational mortality data.

Original authors: Han Lin Shang, Cristian F. Jiménez-Varón

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Han Lin Shang, Cristian F. Jiménez-Varón

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the future weather patterns for 47 different cities, but instead of just "sunny" or "rainy," you have a complex, wavy line representing the temperature for every single hour of the day, for every single day of the year, for both men and women in those cities. That is a lot of data! In statistics, this is called High-Dimensional Functional Time Series. It's a mouthful, but think of it as a massive, tangled ball of yarn where every thread represents a different place, gender, and time.

The paper by Shang and Jiménez-Varón is about how to untangle that ball of yarn, understand what makes the threads move, and predict where they will go next, all while keeping the explanation simple enough for a human to understand.

Here is the breakdown of their method using everyday analogies:

1. The Problem: The "Black Box" vs. The "Recipe"

Most modern forecasting tools are like black boxes. You feed them a mountain of data, and they spit out a prediction. They might be accurate, but they don't tell you why. It's like a chef giving you a delicious soup but refusing to tell you the recipe. You know it tastes good, but you don't know if it's the salt, the heat, or the secret spice that makes it work.

The authors wanted a transparent recipe. They wanted to break the data down into understandable parts so policymakers could see exactly what is driving changes in mortality rates (death rates) across Japan.

2. The Solution: The "Layer Cake" Approach

Instead of guessing the whole cake at once, they slice it into three distinct, easy-to-digest layers.

Layer 1: The "Grand Mean" (The Base of the Cake)

First, they look at the big picture. What is the average trend for everyone in Japan?

The Analogy: Imagine the "average Japanese person." This layer captures the general shape of the curve: "People die more as they get older." It's the foundation.
The Math: They call this the Functional Grand Effect.

Layer 2: The "Specific Flavors" (The Toppings)

Next, they look at how specific groups differ from that average.

The Analogy: How is the "average person in Okinawa" different from the "average person in Tokyo"? How is the "average woman" different from the "average man"?
The Math: They use a Two-Way Functional ANOVA (Analysis of Variance). Think of this as separating the data into "Region" (Prefecture) and "Gender" (Sex) buckets.
- Region Effect: Maybe Okinawa has lower death rates overall because of the climate.
- Gender Effect: Women generally live longer than men.
- The Twist: They also look at the Interaction. Does being a woman in Okinawa change things differently than being a woman in Tokyo? It's like realizing that while coffee is generally bitter, your specific cup of coffee might taste different because of the local water.

Layer 3: The "Random Noise" (The Sprinkles)

After removing the average, the regional differences, and the gender differences, there is still some data left over. This is the "residual."

The Analogy: Imagine you've accounted for the weather, the season, and the time of day. But sometimes, a sudden gust of wind blows a leaf onto your path. That's the "noise." In mortality, this could be a flu outbreak in one specific city or a sudden change in healthcare access.
The Math: They use a Functional Factor Model to catch these remaining random trends. It's like using a net to catch the last few fish that swam away from the main school.

3. The Prediction: Putting It Back Together

Once they have these three layers, they forecast the future for each layer separately and then stack them back up.

The Recipe: Future Mortality = (Future Average) + (Future Regional Difference) + (Future Gender Difference) + (Future Random Noise).

Because they forecast the "Reasons" (like region and gender) separately, the final prediction is interpretable. If the prediction says "Mortality will rise in 2030," the model can tell you: "It's rising because the regional trend in Hokkaido is shifting, not because the whole country is changing."

4. The Safety Net: "Conformal Prediction"

Predicting the future is risky. What if the model is wrong?

The Analogy: Imagine you are throwing a dart at a target. A standard prediction is just a single dot where you think the dart will land. But what if you draw a circle around that dot?
The Method: The authors use a technique called Conformal Prediction. Instead of just guessing a single number, they draw a "confidence ring" around their prediction.
- They tested two ways to draw this ring:
  1. Split: They used old data to practice drawing the ring, then used the rest to test it. (Like a practice exam).
  2. Sequential: They updated the ring size every time a new piece of data arrived, learning on the fly. (Like a GPS that recalculates your route as you drive).
- The Result: The "Sequential" method was better at keeping the actual future data inside the safety ring, even as time went on.

Why Does This Matter?

In the real world, governments need to know why things are happening to make good decisions.

If a model just says "Death rates will go up," a government might panic.
If this model says "Death rates will go up specifically for men in rural areas due to a specific trend," the government can send doctors and resources to those specific rural areas.

In summary: The authors took a messy, complex ball of data, sorted it into a "Base," "Regional," "Gender," and "Random" pile, forecasted each pile individually, and then reassembled them. The result is a forecast that is not only accurate but also tells a clear, understandable story about why the future looks the way it does.

1. Problem Statement

The paper addresses the challenge of modeling and forecasting High-Dimensional Functional Time Series (HDFTS). HDFTS are characterized by:

Temporal Dependence: Observations evolve over time.
Cross-Sectional Dependence: Multiple functional curves (e.g., mortality rates across different regions) are correlated.
High Dimensionality: The number of cross-sectional units ( $N$ , e.g., prefectures) often exceeds the number of time points ( $T$ ).

Key Limitation of Existing Methods:
Current state-of-the-art approaches, such as functional factor models using Principal Component Analysis (PCA), are computationally efficient for dimensionality reduction but often lack interpretability. The resulting latent factors and loading matrices are mathematical constructs ("black boxes") that do not directly correspond to observable, real-world drivers (e.g., specific geographic regions or demographic cohorts like sex). This limits their utility for evidence-based policy making where understanding the source of variation is crucial.

2. Methodology

The authors propose a structured, interpretable decomposition framework that separates deterministic mean structures from stochastic residual trends. The methodology consists of three sequential stages:

Stage I: Two-Way Functional Analysis of Variance (TWA-FANOVA)

The HDFTS ( $Y_{it}^g(u)$ ) is decomposed into a deterministic mean structure and a residual process:
$Y_{it}^g(u) = \mu(u) + \alpha_i(u) + \delta_g(u) + \gamma_{ig}(u) + X_{it}^g(u)$
Where:

$\mu(u)$ : Functional grand mean.
$\alpha_i(u)$ : Functional row effect (Prefecture/Region).
$\delta_g(u)$ : Functional column effect (Sex).
$\gamma_{ig}(u)$ : Interaction term between region and sex.
$X_{it}^g(u)$ : Time-varying residual component.

Unlike previous works, this model explicitly estimates the interaction term $\gamma_{ig}(u)$ to capture cross-group heterogeneity.

Stage II: One-Way Functional Analysis of Variance (OWA-FANOVA)

To further refine the interaction term and the residuals from Stage I, the authors apply a one-way FANOVA to the combined term $X_{it}^{g,\dagger}(u) = \gamma_{ig}(u) + X_{it}^g(u)$ :
$X_{it}^{g,\dagger}(u) = \theta^g(u) + \eta_i^g(u) + R_{it}^g(u)$

$\theta^g(u)$ : Grand effect for gender $g$ .
$\eta_i^g(u)$ : Prefecture-specific mean effect for gender $g$ .
$R_{it}^g(u)$ : The remaining stochastic residual process.

This step isolates the deterministic regional patterns specific to each gender, leaving a purely stochastic residual.

Stage III: Functional Factor Model (FFM)

The remaining stochastic residuals $R_{it}^g(u)$ are modeled using a unified functional factor model (unifying scalar factors with functional loadings and vice versa):
$R_{it}^g(u) = \sum_{k=1}^{k^*} \int B_{ik}^g(u, v) F_{tk}^g(v) dv + \varepsilon_{it}^g(u)$

$F_{tk}^g(v)$ : Time-varying latent factors.
$B_{ik}^g(u, v)$ : Functional factor loadings.
The number of factors ( $q$ ) is selected using an information criterion (Leng et al., 2026).

Forecasting and Uncertainty Quantification

Point Forecasts: The final forecast is the sum of the estimated deterministic components (from TWA and OWA) and the forecasted stochastic component (from FFM).
Interval Forecasts: To quantify uncertainty without relying on strict distributional assumptions, the authors employ Conformal Prediction:
- Split Conformal: Uses a validation set to calibrate tuning parameters based on residual summary statistics (standard deviation or quantiles).
- Sequential Conformal: A model-agnostic approach that updates predictive quantiles autoregressively as new data arrives, avoiding the need for a fixed validation set.

3. Key Contributions

Interpretability: The framework replaces "black-box" latent factors with structurally defined components (Grand Mean, Region, Sex, Interaction), allowing policymakers to directly attribute variations to specific demographic or geographic drivers.
Hierarchical Decomposition: The novel integration of Two-Way FANOVA, One-Way FANOVA, and Functional Factor Models allows for a granular separation of systematic variation (deterministic) from stochastic dynamics.
Robust Uncertainty Quantification: The application of conformal prediction (specifically sequential conformal prediction) provides reliable prediction intervals that are robust to model misspecification and finite-sample limitations.
Empirical Validation: The study demonstrates that explicitly modeling the interaction term (Region $\times$ Sex) improves both point and interval forecast accuracy compared to models that ignore this interaction.

4. Results

The methods were evaluated using Japanese subnational age- and sex-specific mortality rates (1975–2023) across 47 prefectures.

Point Forecast Accuracy:
- The proposed framework (TWA + OWA + FFM) consistently outperformed the benchmark method (TWA + Multivariate Functional Time Series) and the simplified version (TWA + FFM).
- Metrics: The proposed method achieved the lowest Root Mean Square Forecast Error (RMSFE) and Mean Absolute Forecast Error (MAFE) across 1 to 10-step-ahead horizons for both males and females.
- Key Finding: Including the One-Way FANOVA step (to capture the interaction term) yielded a slight but consistent improvement in accuracy, confirming that regional heterogeneity varies significantly by sex.
Interval Forecast Accuracy:
- Sequential Conformal Prediction generally provided better coverage probabilities (closer to the nominal 95%) compared to Split Conformal, which tended to under-cover at longer horizons.
- The proposed framework produced narrower prediction intervals (better sharpness) with comparable or better coverage than benchmark methods, resulting in superior Interval Scores (IS).

5. Significance

Policy Relevance: By decomposing mortality trends into interpretable components (e.g., "how much is due to regional differences vs. sex differences"), the model provides transparent insights for local and national policymakers regarding resource allocation for pension and healthcare systems.
Methodological Advancement: The paper bridges the gap between high-dimensional statistical efficiency and the need for interpretability in functional data analysis. It proves that structured decomposition does not sacrifice forecasting performance; rather, it enhances it by better isolating the signal from the noise.
Generalizability: While applied to mortality data, the framework is applicable to other HDFTS domains, such as temperature curves across weather stations or financial return curves across hundreds of stocks.

In conclusion, the paper presents a robust, interpretable, and accurate framework for forecasting complex functional data, demonstrating that explicitly modeling structural interactions and using conformal prediction for uncertainty yields superior results for evidence-based decision-making.