📊 statistics

Convolution Smoothed Quantile Regression for XGBoost

This paper introduces QXGB, a quantile-based gradient boosting framework that utilizes a convolution smoothed loss function to enable XGBoost to accurately estimate conditional quantiles, construct dense cumulative distribution functions, and characterize extreme outcomes while maintaining computational efficiency and minimizing quantile crossing.

Original authors: Mandy Yao (University of Toronto), Meredith Franklin (University of Toronto)

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mandy Yao (University of Toronto), Meredith Franklin (University of Toronto)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of weather and environmental science, knowing the average temperature or the typical level of air pollution is often not enough. Scientists and public health officials are frequently driven by the rare, dangerous extremes: the days when wildfire smoke turns the sky orange, or when a heat wave pushes temperatures past a critical safety limit. For decades, the most common tools for prediction have focused on finding a single "best guess" number for what will happen next. While useful for planning a picnic, a single number tells us very little about the likelihood of a disaster. It cannot easily answer questions like, "How probable is it that pollution will exceed the safety standard?" or "How thick will the tail of the distribution be when things go wrong?" To answer these questions, researchers need to map out the entire shape of the possible outcomes, not just the center. This requires estimating a full range of possibilities, from the most likely scenarios to the rare, extreme events, and understanding how likely each one is to occur.

A team of researchers at the University of Toronto has developed a new way to build these detailed maps of risk using a powerful machine learning technique called XGBoost. They created a system they call QXGB, which is designed to predict not just a single number, but a dense grid of conditional quantiles. In plain terms, instead of predicting one average value, the system predicts many different thresholds simultaneously, effectively reconstructing the full probability distribution of the outcome. This allows them to calculate exactly how likely it is that a value will cross a dangerous threshold, such as the 35.0 micrograms per cubic meter limit set for fine particulate matter, or PM2.5. The challenge they faced was that the mathematical tools usually used to find these rare extremes are often too jagged or unstable for the fast, efficient algorithms that power modern machine learning. Standard methods for finding these extremes can get stuck or produce contradictory results, such as predicting that a rare event is more likely than a common one.

To solve this, the researchers introduced a smoothing technique based on convolution, which gently rounds off the sharp edges of the mathematical problem without losing the crucial information needed for the computer to make precise decisions. They tested this new approach against existing methods using simulated data that mimicked the volatile, spike-filled nature of wildfire seasons, as well as real-world data from 53 monitoring sites in Northern California spanning from 2012 to 2018. The simulations were designed to include the kind of sudden, massive spikes in pollution that occur during fire seasons, where levels can jump from normal to dangerous in a single day. The researchers found that their new method, particularly when combined with a strategy that allows the model to learn all the different thresholds at the same time rather than one by one, produced highly accurate predictions. It successfully estimated the full range of outcomes, from the most common days to the most extreme spikes, with almost no contradictions in its predictions.

The results showed that this smoothed approach was far superior at handling the chaotic nature of extreme events compared to older methods. When the researchers applied their model to the real PM2.5 data from Northern California, it accurately captured the massive pollution spikes caused by wildfires in 2018. The model was able to provide reliable estimates of the probability that pollution would exceed the safety standard, a critical piece of information for public health officials. Unlike some previous methods that either produced predictions that were too wide to be useful or failed to cover the actual data often enough, this new approach struck a balance. It provided intervals that were narrow enough to be informative but wide enough to capture the true value most of the time. The study demonstrated that by smoothing the mathematical landscape, the computer could navigate the treacherous terrain of extreme weather and pollution data much more effectively, offering a clearer picture of risk when it matters most.

The researchers also discovered that the way the model was structured made a significant difference. When the system tried to learn each risk level independently, it often produced confusing results where the predicted order of events was scrambled. However, by forcing the model to learn all the levels together in a single structure, these contradictions vanished. This joint learning approach allowed the model to share information across different levels of risk, stabilizing the predictions even in the most volatile conditions. The team found that this method worked best when the data was first adjusted to a standard scale, which prevented the mathematical calculations from becoming unstable when faced with the massive numbers associated with extreme pollution events. This simple adjustment, combined with the new smoothing technique, allowed the model to maintain its accuracy even when the data was highly erratic.

In the end, the work offers a practical tool for understanding and predicting the unpredictable. By refining how machine learning handles the tails of a distribution—the rare but consequential extremes—the researchers have provided a way to generate more trustworthy forecasts for events like wildfire smoke. The method does not just tell us what the average day will look like; it quantifies the risk of the worst days with a precision that was previously difficult to achieve with such fast and efficient algorithms. For communities living in fire-prone areas, this means better, more calibrated warnings about when the air quality might become hazardous, allowing for more informed decisions about health and safety. The study confirms that with the right mathematical adjustments, machine learning can be tuned to see the full picture of risk, not just the middle of the road.

Technical Summary: Convolution Smoothed Quantile Regression for XGBoost

Problem Statement

While machine learning (ML) algorithms like Extreme Gradient Boosting (XGBoost) are widely adopted for prediction, they primarily focus on point estimation (e.g., conditional means). This focus limits the ability to characterize predictive uncertainty, conditional distributions, and rare or extreme outcomes. Scientific applications, such as predicting wildfire smoke episodes or heat waves, often require understanding tail behavior and exceedance probabilities rather than just average behavior.

Quantile regression offers a solution by estimating conditional quantiles directly without parametric assumptions. However, integrating quantile regression into XGBoost presents significant optimization challenges:

Non-differentiability: The standard pinball loss function is not differentiable at zero residual ( $y = \hat{y}$ ), hindering convergence.
Lack of Curvature: The second derivative of the pinball loss is zero almost everywhere. Since XGBoost relies on second-order gradient (Hessian) information for efficient tree splitting and Newton updates, the standard quantile objective effectively reduces to a first-order method, forfeiting the speed and efficiency advantages of XGBoost.
Quantile Crossing: When estimating multiple quantiles independently, the resulting cumulative distribution functions (CDFs) often violate monotonicity (quantile crossing), rendering them invalid, especially for out-of-distribution data.

Existing smoothed alternatives, such as Huber pinball or arctan pinball losses, attempt to address differentiability but may still suffer from limited curvature or instability in volatile data regimes.

Methodology

The authors propose QXGB, a framework integrating convolution-smoothed quantile regression into XGBoost. The core methodology involves:

1. Convolution-Smoothed Loss Function

Instead of using the non-smooth pinball loss directly, the authors convolve the pinball loss with a kernel function $K(\cdot)$ and a bandwidth parameter $h$ .

Formulation: The empirical smoothed loss is defined as $\hat{Q}_h(\beta) = \frac{1}{n} \sum_{i=1}^n \ell_h(y_i - \langle x_i, \beta \rangle)$ , where $\ell_h = \rho_\tau * K_h$ .
Properties: This approach yields a loss function that is globally convex and twice continuously differentiable. Crucially, it provides non-zero Hessians everywhere, allowing XGBoost to utilize second-order information for tree splitting.
Kernels: The paper evaluates two specific kernels:
- Gaussian Kernel: Provides a smooth baseline.
- Laplace Kernel: Heavier-tailed, better suited for data with significant outliers or extreme spikes (e.g., wildfire smoke).
Derivatives: The authors derive explicit formulas for the gradients and Hessians for both kernels, enabling seamless integration into the XGBoost optimization loop.

2. Multi-Output Trees vs. Independent Trees

The framework is implemented in two configurations:

Independent Trees: A separate boosted ensemble is trained for each quantile level $\tau$ . This often leads to quantile crossing.
Multi-Output Trees: A single tree structure is shared across all quantile levels, with each leaf storing a vector of predicted quantiles. Splits are chosen to minimize a loss aggregated across all quantile levels. This coupling shares information across quantiles, significantly reducing quantile crossing.

3. Target Standardization (Z-Scaling)

Convolution smoothing relies on exponential functions (e.g., $e^{-z}$ ) for gradients and Hessians. Large prediction errors can cause floating-point overflow and gradient instability. To mitigate this, the authors introduce a target standardization step prior to tree fitting:

Targets are scaled as $y_{scaled} = (y - \mu_{train}) / \sigma_{train}$ .
Predictions are inverse-transformed post-training.
This ensures numerical stability, particularly for extreme values in spatially volatile data.

4. Dense CDF and Exceedance Probability Estimation

The method estimates a dense grid of quantiles ( $\tau \in \{0.05, \dots, 0.95\}$ ) to reconstruct the conditional CDF. Exceedance probabilities for specific thresholds (e.g., $P(Y > y_{thresh}|X)$ ) are derived via linear interpolation of the sorted predicted quantiles.

Key Contributions

Algorithmic Integration: The derivation and integration of convolution-smoothed quantile loss into XGBoost, restoring Hessian information and enabling efficient second-order optimization for distributional prediction.
Stability Mechanisms: The introduction of target standardization to prevent numerical overflow in convolution kernels and the use of multi-output trees to enforce monotonicity and eliminate quantile crossing.
Comprehensive Benchmarking: A rigorous comparison against the native XGBoost quantile objective, Huber pinball loss, and arctan pinball loss across simulation studies and real-world data.
Application to Extreme Events: Demonstration of the method's utility in predicting fine particulate matter (PM2.5) during wildfire seasons, where data is highly skewed and volatile.

Results

Simulation Study

Quantile Crossing: Independent tree estimation resulted in high crossing rates (up to 0.69 for Huber). Multi-output trees reduced crossing rates to near-zero (0.00–0.03) across all smoothed methods.
Accuracy (CRPS): Convolution methods (Gaussian and Laplace) achieved low Continuous Ranked Probability Scores (CRPS) comparable to or better than other smoothed methods, particularly when tuning the smoothing parameter.
Exceedance Probabilities: Convolution smoothing methods yielded the highest Brier Skill Scores (BSS) and most accurate exceedance probability estimates for the 35.0 $\mu g/m^3$ threshold.
Interval Estimation: Convolution methods produced well-calibrated 90% prediction intervals (PICP $\ge$ 0.9) with reasonable widths. The Huber method struggled with undercoverage (PICP = 0.73), while the arctan method produced excessively wide intervals in some settings.

Real-World Application (Northern California PM2.5)

Data: Daily PM2.5 measurements from 53 sites (2012–2018), including the extreme 2018 wildfire season.
Performance:
- Crossing Rates: Multi-output trees reduced crossing rates to 0.00 for convolution and arctan methods at moderate-to-high smoothing levels, whereas independent trees showed rates up to 0.84.
- CDF Estimation: Convolution methods (Laplace and Gaussian) achieved lower CRPS (1.36–1.41) than Huber and Arctan methods at optimal smoothing, providing sharper and better-calibrated distributions.
- Exceedance: The Gaussian kernel convolution method provided the best calibration and highest BSS (0.70) for predicting exceedance of the 35.0 $\mu g/m^3$ threshold.
- Intervals: Convolution methods achieved near-perfect coverage (PICP 0.97–0.98) for 90% intervals. While the arctan method also achieved perfect coverage, its intervals were excessively wide (PINAW = 1.02), rendering them less useful.

Significance and Claims

The paper claims that convolution-smoothed QXGB, particularly when paired with multi-output trees and target standardization, offers a robust framework for distributional prediction. Its primary significance lies in:

Restoring Efficiency: It preserves the computational efficiency of XGBoost by providing the Hessian information necessary for second-order updates, which standard quantile objectives lack.
Handling Extremes: It effectively characterizes tail behavior and exceedance probabilities for rare events (like wildfire smoke) without the instability often found in raw quantile boosting.
Practical Reliability: It delivers accurate predictions with near-zero quantile crossing and well-calibrated CDFs, making it suitable for risk assessment and decision-making in scientific domains where extreme outcomes are critical.

The authors note limitations, specifically the difficulty in estimating probabilities for extremely rare events (far-right tail) due to data sparsity, and suggest future work exploring asymmetric or skew-adaptive kernels to better fit extreme distributions.