🤖 machine learning

Ensemble Score Filtering for Real-Data Energy Consumption Forecast Correction

This paper proposes using the Ensemble Score Filter (EnSF) to correct forecasts from a pretrained black-box spatio-temporal model for real-world energy consumption data, demonstrating that EnSF effectively handles partial and noisy observations and outperforms the Ensemble Kalman Filter in nonlinear settings.

Original authors: Ruoyu Hu, Dahai Yu, Feng Bao, Guang Wang, Guannan Zhang

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ruoyu Hu, Dahai Yu, Feng Bao, Guang Wang, Guannan Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Predicting Power Usage

Imagine you are trying to guess how much electricity a whole city will use every hour. This is crucial for power companies so they don't run out of power or waste money generating too much.

The problem is that real-world data is messy. Sometimes measurements are missing, sometimes they are noisy (like a radio with static), and sometimes they arrive late. Because of this, just guessing based on past patterns isn't enough.

The Two-Step Solution

The authors of this paper propose a two-step team to solve this:

The "Super-Guesser" (The Pretrained Model):
First, they use a very smart computer program (a "spatio-temporal large language model") that has already studied years of electricity data. Think of this model as a super-experienced weather forecaster. If you ask it, "What will the weather be like tomorrow based on today?" it gives a great answer.
- However, if you ask it to predict the weather for the next 30 days without checking the actual sky, it starts to drift. It might predict a sunny day when it's actually raining because it's just guessing based on its memory, not reality. In the paper, they call this "open-loop propagation," and they show it gets unreliable quickly.
The "Reality Check" (Data Assimilation):
To fix the Super-Guesser's drifting predictions, they add a second step: Data Assimilation. This is like having a spotter on the ground who calls in with real-time updates.
- The Super-Guesser makes a prediction.
- The Spotter says, "Hey, I can only see 25% of the city, and it's actually cloudy here."
- The system combines the Guess with the Spotter's report to correct the forecast.

The New Tool: Ensemble Score Filtering (EnSF)

The paper introduces a specific new method for doing this "Reality Check" called the Ensemble Score Filter (EnSF).

To understand why this is special, imagine you are trying to find a lost hiker in a dense forest (the "true" electricity usage).

The Old Way (EnKF): The traditional method (Ensemble Kalman Filter) assumes the hiker is likely in a simple, round circle around the last known spot. It's like drawing a perfect circle on a map and saying, "They are somewhere in here." This works well if the forest is flat and open, but if the terrain is jagged and complex (non-linear), a simple circle is a bad guess.
The New Way (EnSF): The EnSF is smarter. Instead of drawing a simple circle, it uses a "score" to map the entire shape of the forest. It understands that the hiker might be in a valley, on a ridge, or hiding in a cave. It doesn't assume the answer is a simple shape; it learns the complex shape of where the hiker actually is based on the clues.

Key features of this new tool:

No Retraining: Usually, when you get new information, you have to retrain your AI model from scratch. This is slow and expensive. The EnSF is like a GPS that recalculates your route instantly without needing to download a new map. It uses math tricks to adjust the forecast on the fly without relearning anything.
Handles Missing Data: It works even if the "Spotter" only sees a few houses in the city (partial observations).

What They Found (The Results)

The researchers tested this on real electricity data from 5,000 users in Tallahassee, Florida.

The "Super-Guesser" alone fails over time: If they let the computer model predict the future without checking real data, the predictions got wildly wrong very quickly. It was like a GPS that keeps telling you to turn left when you've already passed the turn.
The "Reality Check" fixes it: When they used the EnSF to mix the model's guess with real (but partial) data, the predictions became accurate again. It successfully tracked the true electricity usage even when the model was initially wrong or when data was missing.
New Tool vs. Old Tool: They compared their new EnSF method against the standard "Old Way" (EnKF). In situations where the relationship between the data and the reality was complex (non-linear), the EnSF was much better. It corrected the forecast more accurately than the traditional method, which struggled with the complexity.

Summary

Think of this paper as a recipe for a better electricity forecast.

Ingredient A: A smart AI that knows the general patterns of power usage.
Ingredient B: A new math method (EnSF) that acts like a real-time correction system.
The Result: Even if the AI starts to drift or if we only have partial data, the combination keeps the forecast accurate. The new method (EnSF) is better at handling messy, complex real-world data than the older methods currently in use.

Technical Summary: Ensemble Score Filtering for Real-Data Energy Consumption Forecast Correction

Problem Statement
Accurate energy consumption forecasting is critical for power system operation, planning, and demand-side management. However, real-world energy data often suffer from partial, noisy, delayed, or incomplete measurements. While modern machine learning models, particularly deep spatio-temporal forecasting models, can capture complex temporal and spatial dependencies, they face significant limitations when used for long-horizon prediction in an "open-loop" setting. Without corrective observational data, errors in learned models accumulate over time, leading to unreliable state estimates. The core problem addressed is how to effectively correct the forecast trajectory of a pretrained black-box forecasting model using sequential, partial, and noisy observations, particularly in high-dimensional, nonlinear settings where traditional filtering methods may struggle.

Methodology
The paper proposes a framework combining a pretrained spatio-temporal large language model (STLLM) with the Ensemble Score Filter (EnSF) for data assimilation.

Forward Propagator (STLLM):
- The study utilizes a pretrained STLLM designed for energy consumption. The model employs a factored attention structure to separate temporal and spatial dependency modeling, utilizing components such as RMSNorm, Rotary Positional Embeddings (RoPE), and SwiGLU feed-forward networks.
- In the assimilation framework, this model acts as a fixed, black-box state propagator ( $f_\theta$ ). It is not retrained during the filtering process; instead, it generates the prior forecast trajectory based on historical inputs.
Data Assimilation via Ensemble Score Filter (EnSF):
- The EnSF is employed to correct the forecast trajectory by assimilating observations. Unlike traditional filters that rely on Gaussian assumptions (like the Ensemble Kalman Filter) or require training neural networks to approximate score functions (like standard diffusion-based filters), EnSF utilizes a training-free approach.
- Score Approximation: The method approximates the score function (the gradient of the log-density) of the filtering distribution using a closed-form expression and Monte Carlo estimation. It leverages an ensemble of state samples to compute empirical weights, avoiding the computational cost of retraining score networks at every assimilation step.
- Diffusion Process: The filter operates within a reverse-time diffusion process. It starts from a standard Gaussian distribution at pseudo-time $\tau=1$ and evolves toward the target posterior distribution at $\tau=0$ .
- Update Mechanism: The prior score (derived from the ensemble propagated through the STLLM) is updated by incorporating the likelihood gradient of the new observation. A damping function $g(\tau) = 1-\tau$ controls the integration of observational information, ensuring full incorporation at the target distribution ( $\tau=0$ ) while vanishing at the Gaussian reference ( $\tau=1$ ).
- Handling Nonlinearity: The framework explicitly handles nonlinear observation operators (e.g., arctangent maps) by computing the Jacobian of the observation operator within the likelihood gradient term, allowing for corrections in non-Gaussian settings.

Key Contributions

Application of EnSF to Real Energy Data: The paper demonstrates the application of the Ensemble Score Filter to a high-dimensional (5,000 users) real-world energy consumption dataset, treating a complex pretrained STLLM as the forward dynamics model.
Training-Free Score Approximation: The work highlights the efficacy of using a closed-form, Monte Carlo-based score approximation within EnSF, which eliminates the need for iterative retraining of score networks during sequential data assimilation.
Robustness to Model Deficiencies: The study shows that the framework improves state estimation even when the underlying forward model is "insufficiently trained" (trained on only 25% of data) or when the model is fully trained but suffers from open-loop error accumulation.
Comparison with EnKF: The paper provides a direct comparison between EnSF and the Ensemble Kalman Filter (EnKF) under mixed observation settings (combining direct and nonlinear arctangent observations).

Experimental Results
Numerical experiments were conducted on hourly energy consumption data from 5,000 utility users in Tallahassee, Florida.

Open-Loop Failure: Experiments confirmed that while the pretrained STLLM achieves reasonable accuracy in one-step or short-horizon direct predictions, its open-loop forecasts degrade significantly over time (e.g., RMSE increases drastically for the 12-to-1 model over 800 steps).
EnSF Correction: Applying EnSF with partial observations (25%, 50%, 100%) substantially reduced the Root Mean Squared Error (RMSE) compared to open-loop propagation. The correction was effective for both insufficiently trained and fully trained 4-to-1 models, recovering the main oscillatory structures of the true consumption trajectories.
EnSF vs. EnKF: In a challenging setting with nonlinear (arctangent) observation operators, EnSF outperformed EnKF. While EnKF improved upon open-loop forecasts, it produced overly smooth or displaced trajectories and maintained higher RMSE. EnSF provided stronger correction and better trajectory recovery, attributed to its ability to handle non-Gaussian filtering distributions without relying on Gaussian approximations.

Significance and Claims
The paper claims that the proposed framework offers a practical and effective solution for correcting learned energy-consumption forecasts in high-dimensional, nonlinear environments. By treating the forecasting model as a fixed black-box and using EnSF for sequential correction, the method bridges the gap between powerful but imperfect learned dynamics and real-world observational data.

The authors emphasize that the significance lies in demonstrating that:

Learned forecasting models alone are insufficient for reliable long-horizon state tracking without sequential correction.
EnSF provides a computationally feasible, training-free mechanism for high-dimensional data assimilation that outperforms standard Gaussian-based filters (EnKF) in nonlinear observation settings.
The approach is robust, improving state estimation regardless of whether the underlying forecasting model was trained on full or partial datasets.

The paper concludes that this framework represents a viable direction for improving power system planning and demand-side management, with future work suggested to incorporate exogenous variables (like temperature) and validate the approach on additional real-world datasets.