📊 statistics

Hierarchical Probabilistic Conformal Prediction for Distributed Energy Resources Adoption

This paper proposes a novel hierarchical probabilistic conformal prediction framework that combines a multivariate Hawkes process with a tailored split conformal algorithm to provide statistically valid, aggregated uncertainty quantification for distributed energy resource adoption across different grid levels, outperforming existing baselines in both accuracy and calibration.

Original authors: Wenbin Zhou, Shixiang Zhu

Published 2026-06-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenbin Zhou, Shixiang Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the electrical grid as a massive, complex city. In this city, there are substations (the main power hubs) and circuits (the smaller neighborhoods or streets branching off from those hubs).

For a long time, the city planners (utility companies) knew exactly how much power was flowing into the city. But now, something new is happening: people are installing their own power sources, like rooftop solar panels and home batteries. These are called Distributed Energy Resources (DERs).

This is great for the environment, but it's a headache for the planners. If too many neighbors on one street install solar panels, it can overload the local wires or cause voltage spikes. If they don't plan ahead, the grid could break.

The problem is: Nobody knows exactly who will install a panel, when, or where. It's like trying to guess which houses in a city will buy a new car next year. The growth is fast, uneven, and full of surprises.

The Old Way vs. The New Way

The Old Way (Point Prediction):
Imagine a weather forecaster saying, "It will rain exactly 0.5 inches tomorrow." If it rains 0.6 inches, they were "wrong." If it rains 0.4 inches, they were "wrong." In the real world, being slightly off can be dangerous. Planners need a range, like "It will rain between 0.3 and 0.7 inches."

The Problem with Old Ranges:
Even if the forecaster gets the range right for each individual street, the math often fails when you add them all up.

Street A: Predicted 1–3 panels.
Street B: Predicted 1–3 panels.
The Hub (Substation): If you just add the ranges, you might think the total is 2–6 panels. But because the streets are connected, if Street A gets a lot, Street B might get a lot too. The "real" total might actually be 0–10. The old math creates a false sense of security, leading to either wasted money (building too much) or blackouts (building too little).

The Solution: HPCP (The "Smart Neighborhood Watch")

The authors propose a new method called Hierarchical Probabilistic Conformal Prediction (HPCP). Think of it as a "Smart Neighborhood Watch" for the power grid.

Here is how it works, using simple analogies:

1. Learning the "Social Contagion" (The Hawkes Process)
The paper uses a model called a Multivariate Hawkes Process. Imagine a rumor spreading in a town. If one person starts a rumor, their neighbors are more likely to start one too.

In the Grid: If a homeowner on Circuit A installs a solar panel, it makes it more likely that their neighbors on Circuit A (and even neighbors on a nearby Circuit B connected to the same substation) will do the same.
The model learns these "social" patterns. It doesn't just look at history; it understands that adoption is contagious.

2. The "Simulation Game" (Probabilistic Sampling)
Instead of guessing one number, the model runs thousands of "what-if" games.

It simulates: "What if 5 people on Street A install panels? What if 10? What if 2?"
It does this for every street, creating a massive library of possible futures.

3. The "Sibling Check" (The Secret Sauce)
This is the most important part. In standard prediction, you check if the prediction is right for Street A, then check Street B separately.

HPCP's Trick: It checks the prediction for Street A, but it also looks at the "siblings" (other streets connected to the same substation).
If the model predicts Street A will have 5 panels, but the "sibling" streets are all predicting 0, the model knows something is off. It adjusts the prediction to make sure that when you add Street A + Siblings together, the total still makes sense.
The Metaphor: Imagine a teacher grading a class. Instead of just checking if Student A passed, the teacher checks if Student A's grade makes sense along with the grades of Student A's study group. If the group's total score is impossible, the teacher adjusts the individual grades to ensure the group total is valid.

Why This Matters

The paper tested this on real data from Indianapolis, Indiana, looking at solar panel installations.

It's More Accurate: The method correctly predicted the number of new panels more often than other methods.
It's Tighter: It didn't just give a huge, safe range like "0 to 100 panels." It gave a useful range like "15 to 25 panels," which is much better for planning.
It's Safe at the Top Level: Most importantly, it guarantees that if you add up all the predictions for the neighborhoods, the total prediction for the main power hub is also correct.

The Bottom Line

The authors built a tool that helps power companies plan for the future without guessing. By understanding how solar panel adoption spreads like a rumor through neighborhoods and by checking that the math adds up from the street level to the city level, they can build a grid that is ready for the green energy revolution without wasting money or risking blackouts.

Limitations mentioned in the paper:

It assumes the grid structure (who is connected to whom) doesn't change rapidly.
It is best for short-term planning (next few months) rather than predicting 10 years into the future, because it relies heavily on recent history.
It currently handles a two-level system (Street -> Hub), not a complex 4-level system (Street -> Neighborhood -> Hub -> Region).

Technical Summary: Hierarchical Probabilistic Conformal Prediction for Distributed Energy Resources Adoption

Problem Statement

The rapid adoption of Distributed Energy Resources (DERs), such as rooftop solar panels, introduces significant operational challenges for electric grid management. Accurate forecasting of DER growth is essential for proactive infrastructure planning, yet traditional point-prediction approaches fail to capture the inherent uncertainty and spatial disparity of adoption patterns.

A critical, often overlooked challenge is the hierarchical structure of distribution grids. Grids consist of multiple circuits connected to substations. Operational decisions require reliable predictions at both the individual circuit level and the aggregated substation level. Standard conformal prediction methods ensure validity at the individual level (circuit) but do not guarantee that aggregated predictions (substation) maintain statistical coverage. Naive aggregation of circuit-level intervals can lead to logical inconsistencies, resulting in substation-level intervals that either understate or overstate true uncertainty, thereby compromising risk management and planning decisions.

Methodology

The paper proposes Hierarchical Probabilistic Conformal Prediction (HPCP), a distribution-free framework designed to construct prediction intervals that satisfy statistical validity guarantees simultaneously at both the circuit and substation levels. The methodology integrates three core components:

Probabilistic Base Model (Multivariate Hawkes Process):
The framework utilizes a Multivariate Hawkes Process to model the spatio-temporal dynamics of DER adoption. Unlike traditional time-series models, this approach captures:
- Self-excitation: The influence of past installations on future adoption within the same circuit.
- Mutual influence: The "contagion" effect where installations in one circuit influence neighboring circuits.
- Exogenous factors: Socio-economic and policy covariates.
  The model generates probabilistic forecasts via thinning sampling, producing a distribution of future installation counts rather than a single point estimate. This preserves variance and tail behavior crucial for uncertainty quantification.
Hierarchical Non-Conformity Score:
The core innovation lies in the design of the non-conformity score used in the split conformal prediction algorithm.
- Standard methods often treat dimensions independently or pool residuals globally.
- HPCP defines a score based on "sibling circuits" (circuits connected to the same substation). For a given circuit $k$ , the score incorporates the residuals of all circuits associated with the same substation.
- Specifically, the score $e_{j,k}$ is the minimum $\ell_\infty$ norm of the difference between the observed vector and simulated vectors, restricted to the set of sibling circuits. This leverages the grid topology to align the bounds of aggregated predictions.
Split Conformal Calibration:
The data is split into training and calibration sets.
- The training set fits the Hawkes process parameters.
- The calibration set simulates future outcomes to compute the non-conformity scores.
- Prediction intervals are constructed by estimating the appropriate quantiles of these scores.
- To address potential distributional shifts, the method employs weighted quantile adjustments based on the temporal proximity of calibration windows to the prediction time.

Key Contributions

Topology-Aware Aggregation: The paper introduces a novel non-conformity score that explicitly incorporates the grid's affiliation matrix (circuit-to-substation connectivity). This ensures that prediction intervals remain valid not just for individual circuits but also for their linear aggregation at the substation level.
Theoretical Guarantees: The authors establish finite-sample validity guarantees under mild regularity conditions (specifically, strong mixing of residuals). They prove that the method satisfies coverage constraints at both levels, addressing the non-trivial challenge of dependent data and hierarchical aggregation.
Efficiency Improvements: Theoretical analysis (Proposition 1) demonstrates that the proposed score yields tighter (more efficient) prediction intervals compared to standard marginal conformal prediction, joint conformal prediction, and $\ell_p$ -norm based alternatives. This is achieved by exploiting the sparsity of the affiliation matrix and utilizing the $\ell_\infty$ norm properties.
Empirical Validation: The framework is evaluated on both synthetic data and real-world customer-level solar installation data from Indianapolis, Indiana.

Results

Synthetic Data: Across various simulation scenarios (varying sparsity, intensity, and correlations), HPCP consistently achieved empirical coverage rates above the nominal reference level while producing significantly smaller interval sizes than joint and Bonferroni-corrected baselines.
Real-World Data (Indianapolis):
- Predictive Accuracy: The Multivariate Hawkes process base model outperformed standard baselines (RNN, LSTM, VAR, Gaussian Processes) in terms of Mean Absolute Error (MAE) across multiple time resolutions (monthly to annual).
- Uncertainty Quantification: HPCP achieved the desired 90% coverage rate at both circuit and substation levels. In contrast, other methods either failed to meet coverage targets or produced excessively wide intervals.
- Sensitivity: The method remained robust to variations in simulation size ( $M$ ) and nominal miscoverage rates, with optimal efficiency observed at moderate simulation sizes.
- Spatial Insights: The model successfully identified regional disparities in adoption growth and associated uncertainty, highlighting areas with high potential growth but compounded planning risks.

Significance and Claims

The paper claims that HPCP provides a theoretically sound and practically reliable tool for topology-aware DER operation planning. Its significance is threefold:

Reliability: It addresses the critical gap in existing literature where circuit-level predictions do not translate to valid substation-level guarantees, a necessity for risk-averse grid management.
Efficiency: By leveraging network topology, it avoids the excessive conservativeness typical of naive aggregation methods, providing sharper intervals that are more useful for decision-making.
Applicability: The framework is particularly valuable in settings with limited data and high stochasticity, such as emerging DER markets, where traditional point forecasts are prone to high error.

The authors acknowledge limitations, noting that the current algorithm assumes a fixed two-level hierarchy and is best suited for short-term forecasting. They suggest that future work could extend the framework to deeper hierarchies and non-static network structures.