📊 statistics

Warped multifidelity Gaussian processes for data fusion of skewed environmental data

This paper introduces the warped multifidelity Gaussian process (WMFGP), a novel data fusion method designed to handle skewed environmental data and varying data resolutions, which is demonstrated through simulations and a case study on ARPA Lombardia's wind speed network to effectively fill data gaps and improve air quality management.

Original authors: Pietro Colombo, Claire Miller, Xiaochen Yang, Ruth O'Donnell, Paolo Maranzano

Published 2026-04-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pietro Colombo, Claire Miller, Xiaochen Yang, Ruth O'Donnell, Paolo Maranzano

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to paint a perfect picture of the wind blowing across a region. You have two types of helpers to give you information:

The Expert (High-Fidelity): This person stands at a few specific spots, gives you very accurate, high-quality wind readings, but they are expensive to hire and can only be there for short times. Sometimes, they even go on vacation, leaving big gaps in your data.
The Novice (Low-Fidelity): This person is everywhere, giving you wind readings constantly. However, their measurements are a bit "noisy" or unreliable. They might be slightly off, but they are always there to fill in the blanks.

The Problem:
Usually, wind data isn't a smooth, predictable curve like a bell. It's "skewed." Think of it like a crowd of people: most days the wind is gentle, but occasionally, a massive storm hits. If you try to use standard math tools (which assume everything is a perfect bell curve) to combine the Expert's and the Novice's data, the math gets confused by those sudden storms. It tries to force the data into a shape that doesn't fit, leading to bad predictions.

The Solution: The "Warped" Gaussian Process
The authors of this paper invented a new tool called the Warped Multifidelity Gaussian Process (WMFGP). Here is how it works, using a simple analogy:

The Warping: Imagine the wind data is a piece of rubber that is stretched and twisted out of shape because of those sudden storms. The "warping" part of their method is like a magical pair of hands that gently stretches and reshapes that rubber until it becomes smooth and round (normal).
The Fusion: Once the data is smoothed out, the tool combines the Expert's accurate but sparse data with the Novice's abundant but noisy data. Because the data is now "smooth," the math can easily see how the two sources relate to each other.
The Un-warping: After the math does its job and predicts the missing wind speeds, the tool reverses the magic. It stretches the smooth prediction back into its original, "skewed" shape so it matches reality again.

Why is this special?
Most previous methods tried to use a single "stretching rule" for both the Expert and the Novice. But since they are different people, they need different rules. If you stretch them both the same way, the relationship between them gets broken.

This new method is smart enough to learn a unique stretching rule for each data source while ensuring that the order of the data stays the same. (If the wind was stronger at Station A than Station B before stretching, it stays stronger after stretching). This allows them to fuse the data perfectly without breaking the connection between the two sources.

The Real-World Test
The authors tested this on real data from ARPA Lombardia, an environmental agency in Italy. Their network of weather stations has many holes in the data (gaps) because of equipment failures or bad weather.

The Challenge: They needed to fill in missing wind speed data, especially during long gaps (up to a week), to help predict air pollution.
The Result: Their new "Warped" method was better at filling in these gaps than older methods. It was particularly good because wind speed data is naturally "skewed" (lots of calm days, few stormy days), and their method handled that shape perfectly.

In Summary
This paper presents a clever way to mix high-quality but scarce data with low-quality but abundant data. By first "warping" the data into a smooth shape, doing the math, and then "un-warping" it back, they can accurately fill in missing weather records, even when the data is messy and full of extreme outliers. This helps environmental agencies get a clearer picture of the wind, which is crucial for managing air quality.

1. Problem Statement

Environmental monitoring networks, such as those managed by regional agencies (e.g., ARPA Lombardia in Italy), frequently suffer from data gaps (missing sequences) and skewed data distributions.

The Challenge: Standard data fusion and imputation methods, particularly those based on Gaussian Processes (GP), assume that data and errors follow a normal (Gaussian) distribution. However, environmental variables like wind speed, temperature, and precipitation often exhibit significant skewness.
The Limitation of Existing Methods:
- Standard Multifidelity Gaussian Processes (MFGP) assume linear relationships and normality, leading to poor predictive performance when data is skewed.
- Parametric transformations (e.g., Box-Cox) are often used to normalize data, but they require data-dependent parameters. In a multifidelity context involving multiple data sources (High-Fidelity and Low-Fidelity), applying different transformations to each source can distort the relationship between them, violating the principle of quantile invariance.
- Existing methods struggle to recover long missing sequences (structural missingness) effectively, especially when the gap length exceeds the correlation length scale of standard GPs.

2. Methodology: Warped Multifidelity Gaussian Process (WMFGP)

The authors propose the Warped Multifidelity Gaussian Process (WMFGP), an extension of the autoregressive MFGP framework that incorporates a non-parametric warping function to handle skewness.

Core Concept

The method operates in a latent space where non-normal data is transformed into a normal distribution, the multifidelity model is trained, and results are back-transformed to the original space.

Key Steps:

Non-Parametric Warping (Transformation):
- Instead of parametric transformations, the method uses a data-driven, non-parametric warping function ( $g(\cdot)$ ).
- It estimates the Cumulative Distribution Function (CDF) of the observed data (both High-Fidelity $y_H$ and Low-Fidelity $y_L$ ) using kernel density estimation.
- The data is mapped to a latent space using the inverse of the standard normal CDF ( $\Phi^{-1}$ ):
  $N_y = \Phi^{-1}(F_y(y))$
- This ensures quantile invariance: the relative ordering of data points is preserved, and the relationship between HF and LF data remains consistent in the latent space.
Multifidelity Modeling in Latent Space:
- The autoregressive MFGP model is applied to the normalized latent data ( $N_{y_H}, N_{y_L}$ ).
- The model assumes a recursive relationship: $u_H = \rho u_L + \delta + \epsilon$ , where $\rho$ is the scaling parameter and $\delta$ is a discrepancy process.
- Because the data in the latent space is normalized, the standard GP assumptions (normality of errors) are satisfied.
Back-Transformation:
- Predictions (mean and variance) made in the latent space are transformed back to the original physical space using an inverse lookup table derived from the estimated CDFs.
- This step reconstructs the skewed distribution characteristics of the original data.
Application Strategy (Clustering):
- To handle spatial heterogeneity, the authors employ constrained k-means clustering based on latitude, longitude, and altitude.
- Monitoring stations are grouped into clusters. Within a cluster, a station with missing data (HF) is paired with a nearby, highly correlated station (LF) to perform data fusion.

3. Key Contributions

Novel Algorithm: Introduction of WMFGP, which successfully integrates non-parametric warping with autoregressive multifidelity Gaussian processes.
Handling Skewness: The method explicitly addresses the non-normality of environmental data without violating the correlation structure between different fidelity levels.
Quantile Invariance: By using CDF-based warping, the method ensures that the statistical relationship between High-Fidelity and Low-Fidelity data is preserved during transformation.
Automation: Unlike parametric approaches that require ad-hoc tuning for each time series, WMFGP can be automated for large networks of diverse time series.

4. Experimental Results

The study validates the method through two simulation experiments and a real-world case study.

Experiment 1: Randomized Missingness

Setup: Simulated wind speed data with varying degrees of skewness (using Closed Skew-Normal and Weibull distributions) and random missing data patterns.
Comparison: WMFGP vs. Standard GP, Warped GP (WGP), Standard MFGP, and Box-Cox MFGP (BCMF).
Findings:
- WMFGP consistently outperformed all other models in terms of Mean Absolute Error (MAE), regardless of the skewness level.
- Standard MFGP and BCMF struggled with high skewness. BCMF failed particularly when errors were generated from Weibull distributions, highlighting the sensitivity of parametric transformations to data distribution.
- WMFGP showed stability even when the ratio of HF to LF data was low (up to 30-35% HF data).

Experiment 2: Structural Missingness (Long Gaps)

Setup: Simulated long missing sequences (24 to 192 hours) in real ARPA Lombardia wind speed data.
Comparison: WMFGP vs. GP, MFGP, Surrogate (LF only), and Simple Imputation (SI).
Findings:
- WMFGP achieved the lowest MAE across all gap lengths.
- It reduced the discrepancy between the prediction and the true High-Fidelity data by approximately 50% compared to the raw difference between LF and HF data.
- Unlike standard interpolation, WMFGP performance did not degrade significantly as gap length increased, suggesting robustness for long-term missing data.
- Uncertainty: The 95% prediction intervals had a coverage probability of ~90-91%, indicating reliable uncertainty quantification.

Case Study: ARPA Lombardia

Application: Filling wind speed gaps in the Lombardy regional network (120+ stations).
Result: The method successfully recovered missing wind speed data, capturing seasonal and cyclic patterns that simple interpolation missed. This is critical for air quality management, as wind speed drives pollutant dispersion.

5. Significance and Impact

Environmental Management: Provides a robust tool for environmental agencies to fill data gaps in monitoring networks, leading to more accurate air quality assessments and better-informed policy decisions.
Scalability: The method is computationally efficient and automatable, making it suitable for large-scale environmental datasets with thousands of time series.
Theoretical Advancement: Bridges the gap between multifidelity modeling and non-normal data handling, offering a solution where traditional GP assumptions fail.
Future Directions: The authors note that while the current framework focuses on temporal correlation, future work could incorporate spatial dimensions (e.g., Lidar integration for wind farms), though this would increase computational complexity.

In summary, the paper presents a significant advancement in environmental data fusion by demonstrating that non-parametric warping allows multifidelity Gaussian processes to effectively handle skewed data, outperforming existing methods in both accuracy and robustness for filling critical data gaps.