⚡ electrical engineering

SmartMeterFM: Unifying Smart Meter Data Generative Tasks Using Flow Matching Models

This paper proposes SmartMeterFM, a unified flow matching model that efficiently addresses diverse smart meter data challenges—including synthetic generation, imputation, and super-resolution—through a single conditional training framework, thereby eliminating the need for task-specific models while outperforming existing baselines in data realism and consistency.

Original authors: Nan Lin, Yanbo Wang, Jacco Heres, Peter Palensky, Pedro P. Vergara

Published 2026-02-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nan Lin, Yanbo Wang, Jacco Heres, Peter Palensky, Pedro P. Vergara

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how a city uses electricity. You have a giant notebook filled with data from smart meters, showing exactly how much power every building uses every 15 minutes. This data is the "lifeblood" for planning the grid, fixing problems, and predicting the future.

But there's a big problem: The notebook is often incomplete, messy, or missing pages.

Privacy: Sometimes, the data is hidden because of privacy laws.
Glitches: Sometimes, the meters break or the internet cuts out, leaving gaps in the data.
Blurry: Sometimes, the data is too "fuzzy" (like a low-resolution photo) and needs to be sharpened to see the details.

Traditionally, engineers had to hire a different "specialist" for each problem. One team would fix the missing pages, another would sharpen the blurry photos, and a third would invent fake but realistic pages to fill in the blanks. It was slow, expensive, and inefficient.

Enter "SmartMeterFM": The Universal Swiss Army Knife.

This paper introduces a new AI model called SmartMeterFM. Think of it not as a specialist, but as a master chef who can cook any dish using the same set of ingredients and the same kitchen.

How It Works: The "Flow" Analogy

The secret sauce behind this model is something called Flow Matching.

Imagine you have a cup of clear water (this is random noise) and a cup of chocolate milk (this is real electricity data).

Old AI models (like GANs or VAEs) try to guess the recipe to turn water into chocolate milk. They often struggle, making the milk taste weird or getting stuck trying to learn the recipe for every single type of chocolate.
SmartMeterFM learns the "flow" or the current that naturally moves the water into the chocolate milk. It learns the path. Once it knows the path, it can start anywhere on that path and flow toward the destination.

The Magic Trick: One Model, Many Jobs

The real breakthrough is that SmartMeterFM doesn't need to be retrained for every new job. It learns the "shape" of electricity data once, and then we just give it different instructions at the end (called Inference-Time Guidance).

Here is how it handles three different jobs with the same brain:

Filling in the Blanks (Imputation):
- The Problem: You have a graph with a big chunk missing in the middle because the meter broke.
- The Old Way: You'd use a simple ruler to draw a straight line between the dots. It looks okay, but it misses the spikes and dips (the "peaks" of electricity use).
- SmartMeterFM: It looks at the surrounding data and the "flow" it learned. It fills in the gap with a realistic, wiggly line that captures the true energy spikes, just like a human would guess, but with math precision.
Sharpening the Image (Super-Resolution):
- The Problem: You only have data recorded every hour, but you need to see what happened every 15 minutes.
- The Old Way: You just stretch the hourly data out. It looks smooth and boring, missing all the quick bursts of power.
- SmartMeterFM: It takes the blurry, slow data and "hallucinates" the missing fast details. It adds the tiny wiggles and spikes that make the data look real, based on the patterns it learned.
Following the Rules (Conditional Generation):
- The Problem: You need to create fake data for a factory, but you know exactly how much total power they used that month and what their peak hour was.
- The Old Way: You'd have to train a whole new model just for that specific factory's constraints.
- SmartMeterFM: You tell it, "Make me a month of data, but make sure the total is X and the peak is Y." It instantly adjusts its flow to fit those rules without needing a new lesson.

Why This Matters (According to the Paper)

The researchers tested this model on real data from thousands of Dutch customers. They compared it to the "specialist" models (like GANs and VAEs) and simple math tricks.

Better Accuracy: SmartMeterFM filled in missing data and sharpened blurry data much more accurately than the competition. It didn't just smooth things out; it kept the important "spikes" and "dips" that matter for grid safety.
Realism: When they checked if the fake data looked like real data, SmartMeterFM passed the test. The other models often produced data that looked "off" or too perfect.
Efficiency: Instead of training five different models for five different problems, they trained one model and used it for everything.

In short: SmartMeterFM is a single, smart AI that learned the "rhythm" of electricity. Whether you need to fix a broken record, sharpen a blurry photo, or create a new song that follows specific rules, this AI can do it all without needing to go back to school.

Technical Summary: SmartMeterFM

Problem Statement

Smart meter data is fundamental for the planning and operation of electrical distribution networks. However, the utility of this data is frequently compromised by three primary challenges:

Data Unavailability: Privacy regulations and commercial sensitivities often prevent the sharing of raw data, while limited sensor deployment leads to partial observability.
Data Corruption: Sensor failures, transmission errors, and communication breakdowns result in missing values.
Insufficient Resolution: Many downstream applications (e.g., demand response, load aggregation) require high-frequency temporal data that existing metering infrastructure cannot provide.

Current solutions typically address these issues through separate, task-specific models. For instance, distinct models are trained for synthetic data generation, missing data imputation, and super-resolution. This fragmentation leads to redundancy, inefficiency, and a lack of adaptability; a model trained for one specific missing pattern or resolution scale often cannot be applied to another without re-training. Furthermore, existing deep generative models (such as VAEs and GANs) often struggle with high-dimensional time series data, suffer from mode collapse, or fail to capture complex dynamics and extreme values (peaks) accurately.

Methodology

The paper proposes SmartMeterFM, a unified framework based on Flow Matching (FM) models to address diverse smart meter data generative tasks with a single trained model.

Core Framework: Flow Matching with Inference-Time Guidance

Unlike diffusion models that learn a reverse stochastic process, Flow Matching learns a probability flow Ordinary Differential Equation (ODE) to transport a simple prior distribution (Gaussian noise) to the complex data distribution.

Training: The model is trained exclusively on conditional generation. It learns a velocity field $u_\theta(x_t, t, y)$ that maps noise to data conditioned on variables $y$ (e.g., customer category, month, or peak constraints).
Inference-Time Guidance: The key innovation is the ability to adapt the single trained model to new tasks (imputation, super-resolution) without re-training. This is achieved by treating these tasks as constraints on the generation process.
- The generation process is guided by a term $g_t$ that projects the estimated destination of the ODE trajectory onto a feasible set defined by the specific task constraints (e.g., matching observed values for imputation or matching downsampled averages for super-resolution).
- This allows the model to satisfy hard constraints (like specific peak values or missing data masks) dynamically during the sampling phase.

Model Architecture

The velocity network $u_\theta$ utilizes a Transformer-based architecture tailored for time-series data:

Aligned Positional Embeddings: To handle monthly profiles of varying lengths and different start days of the week, the model employs a shifted sinusoidal positional embedding. This ensures the model retains awareness of daily and weekly periodicity regardless of the month's start day.
Valid Step Masking: To handle variable-length months (28–31 days) within fixed-length batches (31 days), a binary mask is applied to the attention mechanism. This prevents the model from attending to padded zero-values, ensuring it only learns from valid data steps.
Conditioning Mechanisms: The model integrates conditions via:
1. Adaptive Normalization: Modulating LayerNorm parameters based on categorical inputs (e.g., customer type).
2. Multi-Modal Attention (MMAttention): Concatenating condition vectors with the time-series sequence to facilitate information exchange.

Task Adaptation

The paper demonstrates how the same model handles four distinct scenarios by defining the functional relationship $y = f(x)$ for the guidance term:

Categorical Generation: Generating data conditioned on customer type, month, etc.
Numerical Constraint Generation: Generating data that satisfies specific peak, minimum, and total consumption values.
Imputation: Reconstructing missing blocks of data by projecting the generation trajectory onto the subspace of observed values.
Super-Resolution: Upsampling coarse-resolution data (e.g., 4-hour) to fine resolution (15-min) by ensuring the downsampled generated data matches the coarse input.

Key Contributions

Unified Model: The proposal of SmartMeterFM, which unifies synthetic data generation, imputation, and super-resolution into a single conditional flow matching framework, eliminating the need for task-specific re-training.
High-Fidelity Generation: The ability to generate high-dimensional monthly profiles (15-min resolution, ~3000 time steps) with high fidelity, outperforming specialized supervised baselines.
Inference-Time Adaptability: A novel guidance mechanism that allows a model trained solely on conditional generation to solve complex inverse problems (like imputation and super-resolution) by injecting task-specific constraints at inference time.
Architectural Innovations: The design of a Transformer backbone with aligned positional embeddings and valid step masking, specifically optimized for the variable-length, periodic nature of monthly smart meter data.

Results

The method was evaluated on a dataset of ~350,000 monthly smart meter segments from a Dutch Distribution System Operator (DSO), covering four customer categories (three consumption-only, one with solar generation).

Task 1 (Categorical Generation): SmartMeterFM achieved Maximum Mean Discrepancy (MMD) permutation test p-values consistently above 0.05, indicating the generated distribution is statistically indistinguishable from real data. In contrast, VAE and GAN baselines yielded p-values < 0.0001, rejecting the null hypothesis of distributional similarity.
Task 2 (Numerical Constraints): When conditioned on peak and total consumption, SmartMeterFM achieved the lowest Continuous Ranked Probability Score (CRPS) and the lowest Root Mean Square Error (RMSE) for peak and average values compared to VAE, GAN, and algebraic scaling methods. It successfully adhered to constraints while maintaining data fidelity.
Task 3 (Imputation): In missing data scenarios (Missing Not at Random), SmartMeterFM significantly outperformed specialized baselines like LoadPIN and Masked Autoencoders, achieving approximately half the average CRPS. Qualitative results showed that while baselines produced over-smoothed reconstructions that underestimated peaks, SmartMeterFM preserved high-frequency dynamics and extreme values.
Task 4 (Super-Resolution): For upscaling 4-hour data to 15-minute resolution, SmartMeterFM achieved the lowest average and worst-case CRPS. It also demonstrated superior Peak Load Error (PLE) performance compared to ProfileSR (a GAN-based baseline), effectively reconstructing short-duration peak events often lost in low-resolution data.

Significance

The paper claims that SmartMeterFM successfully addresses the fragmentation in current smart meter data processing by providing a single, versatile model capable of handling diverse generative tasks. By leveraging the mathematical properties of Flow Matching and inference-time guidance, the approach eliminates the redundancy of training separate models for imputation, super-resolution, and generation.

The significance lies in the model's ability to generate realistic, high-dimensional time series that respect both intrinsic consumption patterns and external constraints (such as privacy-preserving summaries or specific operational limits). This capability is crucial for distribution system operators (DSOs) who require reliable synthetic data for planning, risk assessment, and the development of advanced control strategies in the absence of complete, high-resolution, or shareable real-world data. The paper emphasizes that this unification is achieved without compromising the quality of the generated data, often surpassing specialized state-of-the-art methods in both quantitative metrics and qualitative realism.