📊 statistics

Integrating Complex Covariate Transformations in Generalized Additive Models

This paper introduces a novel framework for embedding interpretable, differentiable covariate transformations directly into multi-parameter Generalized Additive Models (GAMs), enabling joint estimation of transformation and regression parameters via maximum a posteriori methods and empirical Bayes smoothing selection, as demonstrated through applications to electricity demand forecasting and London house price modeling.

Original authors: Claudia Collarin, Matteo Fasiolo, Yannig Goude, Simon N. Wood

Published 2026-03-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Claudia Collarin, Matteo Fasiolo, Yannig Goude, Simon N. Wood

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to create the perfect soup. Traditionally, statisticians (the chefs) would first spend hours in the "prep kitchen" chopping vegetables, peeling potatoes, and marinating meat (this is called feature engineering or covariate transformation) before they even think about cooking the soup (fitting the model). They do this because they think, "If I prep the ingredients just right, the soup will taste better."

The problem with this old way is that the chef has to guess the perfect prep method beforehand. If they guess wrong, the soup is ruined, and they have to start over from scratch.

This paper introduces a new way of cooking: a "Smart Pot" that does the prep work while it cooks.

Here is a breakdown of what the authors did, using simple analogies:

1. The Core Idea: Cooking and Prepping at the Same Time

Instead of chopping vegetables before turning on the stove, this new method lets the pot automatically figure out the best way to chop and marinate the ingredients while it is simmering.

The Old Way: You take a temperature forecast, guess how much "thermal inertia" (how long it takes a building to heat up or cool down) exists, apply a formula, and then feed that number into your model.
The New Way: You feed the raw temperature data into the model, and the model says, "Hmm, I think the building reacts to temperature changes with a delay of 4 hours," or "Actually, it reacts with a delay of 12 hours." It learns the best "chopping method" (transformation) automatically as it learns the recipe.

2. The "Smart Pot" (The Model)

The authors built a sophisticated statistical tool called a Generalized Additive Model (GAM). Think of a GAM as a very flexible recipe that can handle many different ingredients at once.

Usually, a recipe says: "Add 1 cup of flour, 2 eggs, and 1 cup of sugar."
This new "Smart Pot" says: "Add a variable amount of flour that depends on how humid it is, and mix the eggs in a way that changes based on the altitude."

The paper adds a special feature: Nested Transformations.

Analogy: Imagine a Russian nesting doll.
- The Outer Doll: The smooth curve that predicts the result (e.g., how much electricity people will use).
- The Inner Doll: The transformation of the data (e.g., smoothing out the temperature data to account for how buildings retain heat).
- The Magic: In the past, you had to build the inner doll by hand before putting it inside the outer one. Now, the model builds the inner doll inside the outer one, adjusting the shape of the inner doll until the whole thing fits perfectly.

3. Real-World Examples from the Paper

Example A: The British Power Grid (Electricity Demand)

The Problem: When it gets cold, people turn on heaters. But buildings don't get cold instantly; they have "thermal inertia" (like a slow-cooking pot). If the temperature drops at 6 PM, the heating demand might not peak until 8 PM. Also, the wind might be blowing, which generates free electricity, lowering the demand from the grid.

The Old Way: Experts would guess, "Let's smooth the temperature data over 3 hours," and then run the model. If they guessed wrong, the prediction was off.

The New Way: The model looked at the data and realized, "Actually, for cooling, the building reacts very fast (low inertia), but for heating, it reacts slowly (high inertia)." It automatically created two different "temperature filters" to handle these two different behaviors. It also figured out exactly how much weight to give to wind speed in different regions.

Result: A much more accurate prediction of how much electricity is needed, saving money and preventing blackouts.

Example B: London House Prices

The Problem: House prices are contagious. If a house sells for a high price, the houses next to it often sell for more. This is called "spatial autocorrelation." Traditionally, statisticians would create a "neighborhood index" first, then run the model.

The Old Way: They would pick a fixed radius (e.g., "look at the 10 closest houses") and calculate an average price. If they picked the wrong radius, the model missed the trend.

The New Way: The model learned the "radius of influence" automatically. It figured out, "For this specific street, the price of a neighbor affects my house if they are within 200 meters, but for that other street, it's 500 meters." It combined the raw location data with the neighbor's prices inside the model.

Result: The model separated the "general vibe of the neighborhood" from the "specific influence of the immediate neighbors," leading to a more accurate price estimate.

4. Why This Matters (The "Secret Sauce")

The authors didn't just invent a new pot; they invented a new way to calculate the heat.

The Challenge: When you change how you chop the vegetables (the transformation parameters) while cooking, the math gets incredibly messy and slow. It's like trying to solve a Rubik's cube while juggling.
The Solution: They used a clever mathematical trick called implicit differentiation. Think of this as a "shortcut" that allows the computer to figure out how to adjust the chopping without having to restart the whole calculation every time. This makes the "Smart Pot" fast enough to be useful in the real world.

Summary

This paper is about automating the "prep work" of data science.
Instead of a human expert guessing how to clean and slice the data before analysis, the model itself learns the best way to process the data while it learns the patterns.

Old Way: Pre-process data $\rightarrow$ Build Model $\rightarrow$ If wrong, go back to step 1.
New Way: Feed raw data $\rightarrow$ Model learns the best processing AND the best prediction simultaneously.

It's like giving your GPS the ability to not just find the fastest route, but also to automatically fix the road signs and traffic lights along the way to make the journey even smoother.

1. Problem Statement

In applied statistics, covariate transformations (e.g., log, Box-Cox) and feature engineering (creating new variables, dimension reduction) are typically performed as a separate pre-processing step before model fitting. This separation creates several issues:

Suboptimal Estimation: Transformation parameters are often fixed based on expert knowledge or simple heuristics rather than being optimized jointly with the model parameters.
Loss of Uncertainty Quantification: Treating transformations as fixed ignores the uncertainty associated with their parameters, leading to overly confident inference.
Inflexibility: Standard Generalized Additive Models (GAMs) struggle to handle complex covariates (like time series with thermal inertia or spatially misaligned data) without ad-hoc preprocessing.
Computational Inefficiency: Iterative approaches where the model is re-fitted for different transformation parameters are computationally expensive.

The authors aim to fully integrate feature engineering into the modeling stage, allowing for the estimation of transformation parameters jointly with regression coefficients within a multi-parameter GAM framework.

2. Methodology

The paper proposes a novel framework for Nested Smooth Effects within multi-parameter Generalized Additive Models (GAMs), including Generalized Additive Models for Location, Scale, and Shape (GAMLSS).

2.1 Model Structure

The response $y_i$ follows a distribution $D(y_i; \theta_i)$ where the parameters $\theta_i$ are linked to linear predictors $\eta_{ji}$ via link functions $g_j$ :
$g_j(\theta_{ji}) = \eta_{ji} = Z_{ji}^T \gamma_{j0} + \sum f_{jk}(x_i^{S_{jk}}) + \sum s_{ju}[\tilde{s}_{ju}(x_i^{\tilde{S}_{ju}})]$

$\tilde{s}(\cdot)$ : A scalar-valued, parameterized covariate transformation (e.g., exponential smoothing, kernel smoothing, linear combination).
$s[\cdot]$ : A smooth function (spline-based) applied to the transformed covariate.
Joint Estimation: The parameters of the transformation ( $\mathbf{a}$ ) and the regression coefficients ( $\gamma, \beta$ ) are estimated simultaneously.

2.2 Specific Transformations

The framework supports any sufficiently differentiable transformation. The paper details three specific instances:

Adaptive Exponential Smoothing: Captures thermal inertia in time-series data (e.g., temperature affecting energy demand). The smoothing factor $\omega$ can be constant or modeled as a function of covariates.
Multivariate Kernel Smoothing: Handles spatially misaligned data (e.g., wind speeds at weather stations vs. wind farms). It creates a local index by weighting neighboring observations, with the bandwidth matrix estimated during model fitting.
Linear Combinations (Single Index): Reduces dimensionality by creating a smooth function of a linear combination of covariates ( $\mathbf{x}^T \mathbf{a}$ ).

2.3 Technical Challenges & Solutions

The Scaling Problem: Since the argument of the outer smooth $s(\cdot)$ $s (\cdot)$ depends on parameters $\mathbf{a}$ $a$ , the range of the input changes during optimization. This can cause spline basis functions to become inefficient or undefined.
- Solution: The authors impose constraints on the sample mean and variance of the transformed covariate ( $\hat{\mu}=0, \hat{\sigma}^2=1$ ). They derive deterministic and probabilistic bounds (using Chebyshev's and Samuelson's inequalities) to set the knot placement for the outer spline, ensuring the basis remains valid regardless of $\mathbf{a}$ .
Identifiability: To ensure the outer smooth is identifiable, constraints are imposed: $s(0)=0$ (no intercept) and linear extrapolation beyond the knot range with higher-order derivative constraints ( $s''=s'''=s''''=0$ ) at boundaries.

2.4 Fitting and Inference

Estimation: Parameters are estimated via Maximum A Posteriori (MAP) methods using a Laplace approximation to the marginal likelihood (LAML).
Optimization: Smoothing parameters are selected via empirical Bayes using quasi-Newton optimization (BFGS).
Efficient Derivatives: A key contribution is the extension of implicit differentiation techniques (Wood et al., 2016). The authors derive efficient $O(np^2)$ algorithms to compute the gradient of the marginal likelihood with respect to smoothing parameters, even when nested transformations are present. This avoids the $O(np^3)$ cost of naive differentiation.
Uncertainty: Joint uncertainty is quantified using approximate Bayesian techniques, treating smoothing parameters as fixed at their LAML maximizers.

3. Key Contributions

Unified Framework: First framework to fully integrate general, interpretable covariate transformations directly into the GAM fitting process, eliminating the need for separate preprocessing.
Computational Efficiency: Development of efficient implicit differentiation methods that allow for the joint estimation of complex transformation parameters and smoothing parameters without prohibitive computational costs.
Theoretical Rigor: A principled solution to the "scaling problem" in nested smooth effects, ensuring numerical stability and differentiability of the likelihood.
Software Implementation: The methods are implemented in the R package gamFactory.

4. Results and Applications

The methodology was validated through two real-world applications:

A. Electricity Net-Demand Forecasting (Great Britain)

Context: Forecasting hourly net-demand using weather forecasts and lagged demand.
Application: Used adaptive exponential smoothing to model thermal inertia of buildings. The model learned two distinct smoothing regimes: a low-inertia regime (heating/cooling) and a high-inertia regime (heating only).
Performance: The nested model outperformed standard GAMs and models with fixed transformation parameters in terms of Log-score, CRPS, RMSE, and MAE.
Efficiency: While the nested model took longer to fit (10 mins vs 0.5 mins for standard GAMs), it was significantly faster than iterative re-fitting approaches (which took ~1160 mins).

B. London House Price Modelling

Context: Modeling house prices while accounting for spatial autocorrelation and local socio-economic factors.
Application: Used multivariate kernel smoothing to create a local price index from neighboring sales, allowing the spatial decay rate (bandwidth) to be estimated from the data.
Performance: The nested model achieved predictive performance comparable to a standard spatial model with a massive number of basis functions ( $k=2000$ ), but with only $k=400$ basis functions.
Efficiency: The nested model was ~20 times faster to fit than the high-dimensional standard spatial model.
Insight: The model successfully separated the "local price spillover" effect from the broader spatial trend, revealing that spatial autocorrelation is stronger in affluent areas.

5. Significance

Bridging the Gap: This work bridges the gap between traditional statistical modeling (GAMs) and modern feature engineering, allowing for "automatic" feature creation within the model.
Interpretability: Unlike deep learning "black boxes," this approach maintains the interpretability of additive models while handling complex, non-linear dependencies.
Scalability: By solving the computational bottleneck of joint estimation, the method makes complex, data-driven transformations feasible for routine use in high-stakes applications (e.g., energy planning, real estate valuation).
Future Directions: The framework is modular and extensible, potentially supporting nested composite transformations (deep feature engineering) and constrained single-index models.