📊 statistics

CROCS: A Two-Stage Clustering Framework for Behaviour-Centric Consumer Segmentation with Smart Meter Data

The paper proposes CROCS, a novel two-stage clustering framework that leverages smart meter data to create robust, behavior-centric consumer segments by first summarizing individual daily load profiles into representative sets and then clustering consumers based on the weighted similarity of these sets, thereby effectively capturing behavioral diversity and handling data anomalies for improved Demand-Side Management.

Original authors: Luke W. Yerbury, Ricardo J. G. B. Campello, G. C. Livingston Jr, Mark Goldsworthy, Lachlan O'Neil

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Luke W. Yerbury, Ricardo J. G. B. Campello, G. C. Livingston Jr, Mark Goldsworthy, Lachlan O'Neil

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a city planner trying to understand how thousands of different families use electricity. You have a massive pile of data: every single day for years, you know exactly when they turned on the lights, the oven, or the air conditioner.

The problem is that people are messy. One family might cook dinner at 6 PM on Tuesdays but at 8 PM on Fridays. Another might have a "lazy Sunday" routine that looks nothing like their "busy Monday" routine. Traditional methods of grouping these people often fail because they try to force everyone into a single, rigid "average" schedule. It's like trying to describe a person's personality by only looking at their photo taken at noon on a Tuesday; you miss the whole picture.

This paper introduces a new tool called CROCS (Clustered Representations Optimising Consumer Segmentation). Think of CROCS as a smart, two-step sorting machine designed to find groups of people who behave similarly, even if they don't do it at the exact same time.

Here is how it works, broken down into simple steps:

Step 1: The "Daily Diary" Sort (Stage One)

Instead of squashing a whole year of data into one boring average, CROCS looks at each person's daily habits individually.

The Analogy: Imagine asking every family to sort their own daily photos into piles.
- Family A might have a pile for "Early Riser" days, a pile for "Late Night Gamer" days, and a pile for "Vacation Mode" days.
- Family B might have a pile for "Work Day" and a pile for "Weekend Party."
What CROCS does: It creates a Representative Load Set (RLS) for each family. This isn't just one photo; it's a small album of their most common "personas." It captures the fact that a family isn't just one thing; they have a few different ways of living.
The Benefit: This step is very flexible. It doesn't care if a family has data for 10 days or 1,000 days. It doesn't care if they have "bad data" (like a day they went on vacation and didn't use the fridge). It just groups their own days together first.

Step 2: The "Matching Game" (Stage Two)

Now that every family has their own little album of daily habits, CROCS needs to group the families together.

The Analogy: Imagine you are trying to match families based on their albums.
- Old Method: "Do you both eat dinner at 6:00 PM exactly?" If Family A eats at 6:00 and Family B eats at 6:15, the old method says, "You are different!"
- CROCS Method: "Do you both have a 'Dinner Time' habit, even if one eats at 6:00 and the other at 6:15?"
The Secret Sauce (WSMD): CROCS uses a special math trick called Weighted Sum of Minimum Distances (WSMD). Think of this as a scoring system that says:
1. Look at Family A's "Dinner" habit. Find the closest match in Family B's album.
2. Crucial Step: Count how often that habit happens. If Family A eats at 6:00 PM 90% of the time, that habit is very important. If they only do it 1% of the time, it's a fluke.
3. The system gives more weight to the common habits and ignores the weird, one-off days.
The Result: It finds that Family A and Family B are actually very similar, even if their clocks are slightly out of sync. It groups them together because they share the same types of days, just on different schedules.

Why This Matters (The "Aha!" Moments)

The paper highlights three big things this new method does better than the old ones:

It Handles "Asynchronous" Twins:
Imagine two families who are identical in every way, but one works a 9-to-5 job and the other works a 10-to-6 job. Their daily electricity use looks exactly the same, just shifted by one hour. Old methods would put them in different groups because their clocks don't match. CROCS sees them as twins because it looks at the shape of the day, not the exact time on the calendar.
It Ignores the "Bad Days":
If a family has a power outage or a weird holiday where they use no electricity, old methods get confused. They might think the whole family is weird. CROCS says, "Oh, that's just one weird day in their album. Let's look at the other 364 days." It focuses on what happens most of the time.
It Scales Up:
Smart meters are being installed in millions of homes. Old methods get slow and crash when you try to process that much data at once. CROCS is like a team of workers where everyone sorts their own pile first (Step 1) before they come together to compare notes (Step 2). This means it can handle huge datasets without breaking a sweat.

The Final Output: "Refined" Groups

Finally, CROCS doesn't just stop at grouping the families. It creates a Refined Representative Load Set (RRLS).

The Analogy: Instead of just saying "Group A eats dinner," it says, "Group A has three main ways of eating: 60% of the time they eat early, 30% of the time they eat late, and 10% of the time they skip dinner."
This gives energy companies a much clearer, more honest picture of how people actually live, rather than a blurry, averaged-out guess.

Summary

In short, CROCS is a smarter way to sort electricity users. It stops trying to force everyone into a single "average" schedule. Instead, it acknowledges that people have multiple "modes" of living, it ignores the weird outliers, and it groups people based on what they do, not when they do it. This helps energy companies design better programs to save money and balance the grid, because they finally understand the real, messy, beautiful complexity of how people use power.

Technical Summary: CROCS – A Two-Stage Clustering Framework for Behaviour-Centric Consumer Segmentation

1. Problem Statement

The proliferation of smart meter infrastructure has generated unprecedented volumes of consumption data, offering the potential for behaviour-centric consumer segmentation to optimize Demand-Side Management (DSM) and Demand Response (DR) programs. However, existing clustering methodologies suffer from eight fundamental limitations that hinder their effectiveness in real-world deployments:

Inadequate representation of intra-consumer diversity: Methods often aggregate daily load profiles (DLPs) into a single Representative Load Profile (RLP), obscuring the fact that individual consumers exhibit multiple distinct behavioural modes.
Oversimplified cluster characterisation: Clusters are typically summarised by a single prototypical profile, failing to capture shared multi-modal behaviours within a group.
Reliance on temporally anchored comparisons: Many methods assume behavioural similarity requires alignment on the same calendar days, failing to identify consumers with similar patterns occurring on different schedules (asynchronous similarity).
Sensitivity to anomalies: Outlier days (e.g., holidays, travel) are often weighted equally with routine days, distorting consumer representations.
Requirement for time-synchronised data: Many approaches require input series to start and end at identical timestamps, excluding consumers with staggered installation dates or varying data lengths.
Sensitivity to regular discontinuities: Methods struggle when data is segmented (e.g., weekdays vs. weekends), introducing spurious temporal dependencies.
Poor handling of missing data: Missing intervals often necessitate the exclusion of entire consumer records rather than just affected days.
Limited scalability: Global representation methods (e.g., pooling all data for Principal Component Analysis) scale poorly with population size and require recomputation when new consumers are added.

No existing approach simultaneously addresses all these limitations, creating a significant gap in the literature for robust, scalable, and interpretable consumer segmentation.

2. Methodology: The CROCS Framework

The authors propose CROCS (Clustered Representations Optimising Consumer Segmentation), a novel two-stage framework designed to overcome the aforementioned limitations.

Stage One: Independent Consumer Clustering (RLS Generation)

Instead of aggregating a consumer's entire history into a single average profile, Stage One independently clusters the Daily Load Profiles (DLPs) of each consumer.

Process: For each consumer $i$ , their set of $p_i$ DLPs is clustered into $k_i$ groups.
Output: A Representative Load Set (RLS), denoted $\hat{S}_i = \{\pi^1_i, \pi^2_i, \dots, \pi^{k_i}_i\}$ , where each $\pi$ is a prototype (e.g., a medoid) representing a distinct behavioural mode.
Key Feature: This stage captures intra-consumer variability. The number of clusters $k_i$ can vary per consumer or be fixed to a uniform value $k$ across all consumers. The framework allows for the overestimation of $k$ to ensure all behavioural modes are captured without degrading performance.
Robustness: By clustering DLPs independently, the method naturally handles missing data (affected days are simply omitted), irregularities, and non-synchronised time series.

Stage Two: Consumer Clustering via WSMD

Stage Two clusters the consumers themselves based on their RLSs.

Distance Measure: The authors introduce the Weighted Sum of Minimum Distances (WSMD). Unlike standard set-to-set distances (e.g., Hausdorff), WSMD accounts for the prevalence of each behavioural mode.
- Formula: $\Delta_{WSMD}(\hat{S}_i, \hat{S}_j) = \frac{1}{2} \left[ \frac{1}{p_i} \sum_{a=1}^{k_i} n^a_i \min_b d(\pi^a_i, \pi^b_j) + \frac{1}{p_j} \sum_{b=1}^{k_j} n^b_j \min_a d(\pi^b_j, \pi^a_i) \right]$
- Here, $n^a_i$ is the number of DLPs represented by prototype $\pi^a_i$ , and $d(\cdot, \cdot)$ is a time-series dissimilarity measure (e.g., DTW).
Mechanism: WSMD finds the closest prototype in the other consumer's set for every prototype in the current set, weighting the distance by the frequency of that behaviour. This ensures that common behaviours dominate the similarity metric, while rare anomalies have minimal influence.
Asynchronous Detection: Because the RLS is a set of patterns rather than a time-ordered sequence, WSMD can identify similarity between consumers even if their patterns occur on different days (asynchronous similarity).

Optional Step: Refined Representative Load Sets (RRLS)

To enhance interpretability, the framework includes a graph-based community detection step.

Process: A graph is constructed where nodes are the prototypes from all consumers in a cluster, and edges represent the WSMD-induced similarities.
Output: Community detection algorithms (e.g., Leiden) identify densely connected subgraphs, which are summarised as Refined Representative Load Sets (RRLS). These provide multi-profile summaries of the shared behavioural modes defining a cluster, rather than a single average.

3. Key Contributions

Comprehensive Review and Taxonomy: The paper provides a structured review of existing consumer clustering methods, categorising them by their limitations and introducing a taxonomy of two-stage methods.
Novel Framework (CROCS): The first framework to simultaneously address all eight identified limitations of existing methods.
Representative Load Set (RLS): A flexible, set-based representation learned independently for each consumer, capturing the diversity of recurring behaviours without temporal anchoring.
Weighted Sum of Minimum Distances (WSMD): A novel set-to-set dissimilarity measure that weights behavioural patterns by their prevalence, enabling robust comparison of RLSs.
Graph-Based RRLS: A procedure to extract multi-profile cluster summaries that improve interpretability by explicitly revealing shared behavioural modes.
Empirical Validation: Extensive experiments on synthetic and real-world Australian smart meter datasets (Ausgrid and SGSC) validating the framework's performance, scalability, and robustness.

4. Experimental Results

The authors evaluated CROCS against a wide range of single-stage and two-stage baselines (including RLP, DCP, GPF, and ensemble methods) using Adjusted Rand Index (ARI), Adjusted Mutual Information (AMI), and Pair Sets Index (PSI).

Intra-Consumer Variation: CROCS (using RLS) achieved significantly lower reconstruction errors compared to single-profile methods (RLP, DCP) and global multi-profile methods (GPF), particularly as the number of behavioural modes increased.
Robustness to Outliers: In synthetic experiments with increasing numbers of outlier days, WSMD consistently outperformed other set distances (SMD, MHD, Hausdorff). It required less overestimation of the cluster parameter $k$ to maintain accuracy compared to competitors.
Asynchronous Similarity: Experiments demonstrated that consumers with similar but asynchronously expressed patterns are common in real data (approx. 7.7% of pairs in the AG dataset vs. 0.6% for synchronised pairs). CROCS was the only method to consistently recover these asynchronous clusters, whereas methods relying on temporal alignment (e.g., Long Time Series, DCE) failed as the number of distinct patterns increased.
Scalability: CROCS demonstrated practical scalability, processing datasets of up to 80,000 consumers within two days on a single HPC node. The framework's first stage is fully parallelisable, and the second stage scales quadratically with the number of consumers but linearly with the number of prototypes, making it feasible for large populations.
Real-World Application: Applied to Australian summer and winter data, CROCS identified distinct consumer clusters with interpretable hyperprototypes. The results highlighted seasonal variations in behavioural diversity and provided actionable insights for DSM program design (e.g., targeting specific load profiles).

5. Significance and Claims

The paper claims that CROCS represents a significant advancement in behaviour-centric consumer segmentation by unifying methodological rigour with practical scalability.

Paradigm Shift: The framework challenges the assumption that consumer similarity must be temporally anchored, demonstrating that asynchronous similarity is prevalent and critical for effective DSM/DR strategies.
Operational Utility: By providing interpretable, multi-profile summaries (RRLS) and robust handling of real-world data imperfections (missing data, anomalies, irregular schedules), CROCS offers a more reliable foundation for designing and evaluating demand-side interventions.
Extensibility: The modular nature of the framework allows it to be adapted to various time-series domains beyond smart meters, such as water consumption, EV charging, or machine operation cycles, wherever long series can be decomposed into recurring periods.

The authors conclude that CROCS strengthens the analytical foundations of DSM and DR strategies, offering a robust tool to maximise the value of smart meter investments in increasingly complex electricity systems.