💻 computer science

What the Detector Can See: Evaluating CPS Anomaly Detectors Independently of the Decision Rule

This paper proposes a decision-rule-free evaluation framework that isolates a detector's ability to represent physical processes from its threshold calibration, revealing that traditional metrics like ROC-AUC can mask significant performance disparities and varying failure modes across different cyber-physical system benchmarks and anomaly detectors.

Original authors: Peiran Shi, Jian Xiang, Xiang Zhang, Chenglong Fu

Published 2026-08-05

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Peiran Shi, Jian Xiang, Xiang Zhang, Chenglong Fu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard at a massive, high-tech factory where robots build cars and mix chemicals. Your job is to spot anything weird happening on the assembly line. If a robot arm starts moving too fast or a valve opens when it shouldn't, you need to know immediately. In the world of science, this is called a Cyber-Physical System (CPS). It's where computer code meets the real, physical world. But here's the tricky part: sometimes the factory just has a "bad day." Machines vibrate, temperatures wiggle, and sensors get a little jittery. How do you tell the difference between a harmless glitch and a sneaky hacker trying to blow something up?

For a long time, scientists have tried to solve this by building "anomaly detectors." Think of these as super-smart cameras that watch the factory. They learn what "normal" looks like, and then they scream "ALARM!" if something looks different. But there's a problem with how we test these cameras. Usually, we just look at the final result: Did the alarm go off? Did it miss anything? It's like grading a student only on whether they raised their hand, without checking if they actually understood the question. This paper argues that we need to look inside the camera before it decides to scream. We need to see what the camera actually "saw" in the first place, before it gets confused by the rules it uses to decide when to yell.

The Two-Stage Detective

The authors of this paper, a team from the University of North Carolina at Charlotte, decided to stop grading detectors just on their final "screams" (alarms). Instead, they treated every detector like a two-step process, like a detective solving a crime.

Stage 1: The Evidence Collector.
Imagine a detective looking at a crime scene. They don't just shout "Criminal!" immediately. First, they gather clues. They measure the mud on the shoes, the temperature of the room, and the time on the clock. In the paper's language, this is the residual representation. The detector takes the factory's data and calculates the "gap" between what happened and what should have happened. If the robot arm moved 5 inches but the computer expected 4, the "residual" (the gap) is 1 inch. This stage is all about gathering the raw evidence.

Stage 2: The Decision Maker.
Once the detective has the clues, they have to decide: Is this a crime? Maybe they have a rule like, "If the mud is more than 2 inches, call the police." This is the alarm rule. It's a threshold. If the gap is big enough, you scream; if it's small, you ignore it.

The big problem the paper points out is that most scientists only look at the final result of Stage 2. They say, "Oh, Detector A had a 90% success rate, and Detector B had 80%." But that's like saying two detectives are equally good just because they both caught the same number of criminals, without asking how they found them. Maybe Detector A had amazing clues but a terrible rule for calling the police. Maybe Detector B had terrible clues but got lucky with a low bar for screaming.

The New Way to Look: The "Energy" Meter

To fix this, the authors invented a new way to measure the detectors. They created a tool called Normalized Residual Energy.

Think of this like a "weirdness meter." Instead of just asking "Did you scream?", they ask, "How weird did the evidence look before you decided to scream?" They measure the "energy" of the gap between the real world and the expected world. If the factory is acting normally, the energy is low. If a hacker is messing with the valves, the energy spikes.

The cool thing about this meter is that it doesn't care about the "scream rule." It just measures the raw evidence. The authors found that this energy meter has a special connection to a math concept called Kullback–Leibler (KL) divergence. In plain English, this is a way to measure how different two things are. It's like measuring how much a "normal" factory day sounds different from a "hacked" factory day. By using this energy meter, the authors could see the "truth" of the evidence without the noise of the decision rules.

What They Found: The Great Detective Swap

The team tested five different types of detectors (like GDN, FuSAGNet, TranAD, and others) on three famous factory testbeds (SWaT, WADI, and HAI). These are like different "levels" in a video game, each with its own layout and types of attacks.

Here is what they discovered, and it's pretty surprising:

The "Score" Lie: Two detectors can have almost the same final score (like an F1 score or ROC-AUC), but when you look at the raw evidence, they are totally different. One might have great evidence but a terrible alarm rule, while the other has weak evidence but a lucky rule. The paper showed that on the SWaT testbed, detectors with similar overall rankings could have alarm success rates that differed by more than ten times (an order of magnitude) when using a standard rule!
The "One Size Fits All" Myth: You might think a detector that is great at Level 1 (SWaT) would be great at Level 2 (WADI). The authors found this is not true. A detector named TranAD was the top performer on the HAI testbed but came in last place on SWaT. Meanwhile, NSIBF was the king of WADI but the loser on HAI. It turns out that different factories need different kinds of eyes.
The "Dilution" Problem: Some detectors look at the whole factory at once, pooling all the data together. The authors found that on the WADI testbed, this was a bad idea. If a hacker only messes with one small valve in a huge plant, and the detector looks at the whole plant, that small signal gets "diluted" (watered down) by all the normal, boring data from the rest of the factory. The signal gets lost in the noise. But detectors that looked at smaller, specific parts of the factory (like just the sensors) could see the attack clearly.
The "Drift" Trap: Sometimes, a detector misses an attack not because it's bad, but because the factory changed. If the factory runs slightly differently during the test than it did during training (like a machine warming up), the "normal" baseline shifts. The authors found that some detectors were failing because their "normal" reference was drifting away, making the attacks look like normal noise.

Why This Matters

The paper doesn't just say "Detector X is better." It acts like a diagnostic tool for doctors. If a patient (the detector) is sick, you don't just say "they are sick." You need to know why. Is the problem that the eyes (Stage 1) can't see the disease? Is the problem that the brain (Stage 2) is too slow to react? Or is the problem that the patient's body has changed so much that the old medical charts don't work anymore?

By separating the "evidence gathering" from the "decision making," the authors showed that a failed alarm can come from many different places:

Weak Evidence: The detector simply couldn't see the attack.
Bad Threshold: The detector saw the attack but set the alarm bar too high.
Drift: The factory changed, and the detector's memory of "normal" was outdated.
Dilution: The attack was too small and got lost in the big picture.

The Bottom Line

This paper suggests that we need to stop judging cyber-security detectors only by their final alarm bells. Instead, we should look at the "residual energy"—the raw, unfiltered evidence they gather. The authors showed that when you do this, the rankings of detectors change completely, and you can finally understand why a detector failed. It's a move from just counting the "alarms" to understanding the "vision" of the machine.

The study didn't invent a new detector to win a prize; it invented a new way to grade the detectors we already have. And the grade they gave is clear: if you want to know if a detector is actually good, you have to look at what it sees before it decides to scream.

Technical Summary: Evaluating CPS Anomaly Detectors Independently of the Decision Rule

1. Problem Statement

Cyber-physical systems (CPS) rely on anomaly detectors as a final line of defense against adversarial manipulation of sensor and actuator telemetry. However, the standard evaluation of these detectors is flawed because it conflates two distinct components:

Stage 1 (Representation): The mapping of observations to a residual signal (e.g., prediction errors, reconstruction errors, or invariant violations) that represents the gap between observed behavior and expected normal behavior.
Stage 2 (Decision Rule): The mapping of residuals to binary alarms via thresholds, smoothing, or sequential tests (e.g., CUSUM).

Current benchmarks typically report precision, recall, and F1 scores at a single operating point. These metrics mix the quality of the residual representation with the calibration of the decision rule. Consequently, a low F1 score cannot distinguish whether a missed attack resulted from:

Weak residual evidence (the attack is invisible to the model).
Train-test drift (the normal distribution shifted between training and testing).
Dimensional dilution (localized attacks are masked when pooling evidence across all channels).
Poor threshold calibration (the evidence exists but the alarm rule fails to extract it).

This issue is exacerbated when comparing heterogeneous detectors (e.g., graph predictors vs. invariant checkers) whose residuals live in different spaces and are paired with independently tuned decision rules.

2. Methodology

The authors propose a decision-rule-free evaluation framework that treats a CPS anomaly detector as a two-stage pipeline and evaluates Stage 1 directly.

The Two-Stage Decomposition

Stage 1 ( $\phi$ ): Maps a window of observations to a residual vector $r_t$ .
Stage 2 ( $\pi$ ): Maps residuals to alarms $a_t$ .
The framework isolates $\phi$ for evaluation before any alarm rule is applied.

The Residual Information-Energy Framework

The core metric is Normalized Residual Energy ( $e_t$ ), defined as the squared Mahalanobis distance of a residual from the trained-normal reference, scaled by one-half:
$e_t = \frac{1}{2} (r_t - \mu_0)^\top S_0^{-1} (r_t - \mu_0)$
where $\mu_0$ and $S_0$ are the mean and covariance of the residual distribution estimated from attack-free training data.

Theoretical Foundation:
Under a moment-matched Gaussian view, the expectation of this energy has an exact connection to the Kullback–Leibler (KL) divergence between the test distribution and the training-normal reference. Specifically:
$E[e_t | P] - \frac{p}{2} = D_{KL}(P_G || P_0) + \frac{1}{2} \log \frac{\det S_P}{\det S_0}$
This identity allows the energy scale to simultaneously measure:

Attack Separation: How well the attack distribution separates from normal.
Train-Test Drift: How much the normal distribution has shifted.
Residual Space Structure: The compactness of the representation.

Evaluation Metrics

The framework derives several diagnostic metrics from the energy stream:

Rank Metrics: ROC-AUC and pAUC@1% (focusing on low false-alarm rates) to measure point-level separation.
Attack-Window Energy: Integrated KL estimates ( $D_{FULL}$ ) and the probability ( $P_{FULL}$ ) that an attack window dominates a duration-matched normal window.
Drift Diagnostics: Train-test drift ( $D_{drift}$ ) and drift-corrected evidence ( $D_{eff}$ ) to distinguish attack signals from normal distribution shifts.
Compactness: The ratio of effective dimension ( $p_{eff}$ ) to nominal dimension ( $p$ ), indicating how much of the residual space is utilized.
Controlled Stage-2 Probe: A fixed threshold at the 99th percentile of training-normal energy ( $F1@p99$ ) is used only as a probe to see how much evidence a shared rule can extract, not as a primary ranking metric.

3. Experimental Setup

The framework was applied uniformly to five detectors across three CPS benchmarks without per-detector tuning:

Detectors: GDN, FuSAGNet, TranAD (graph/transformer predictors), NSIBF (observer-based), and GeCo (algebraic invariants).
Benchmarks: SWaT (water treatment), WADI (larger water distribution), and HAI (power/boiler).
Protocol: Detectors were trained on attack-free data and evaluated on labeled attack traces. The evaluation used a fixed protocol for estimating the normal reference (Ledoit–Wolf shrinkage covariance) and computing energy metrics.

4. Key Results and Findings

The study answers four diagnostic research questions (RQs) regarding the nature of detection failures:

RQ1: Is attack evidence present before the alarm rule?

Finding: Similar ROC-AUC values can hide order-of-magnitude differences in alarm performance. For example, on SWaT, TranAD had an ROC-AUC of 0.750 but an $F1@p99$ of only 0.025, while other detectors with similar ROC-AUC achieved much higher F1. This indicates that the residual representation contained ranking information, but the separation was weak in the low-false-alarm region.

RQ2: Is the normal residual reference stable?

Finding: Train-test drift is a major source of failure. Detectors with similar attack rankings exhibited vastly different drift values. For instance, on SWaT, GDN showed massive drift ( $1.9 \times 10^5$ ), while NSIBF showed minimal drift ( $6.2 \times 10^2$ ). High drift reduces the margin available for fixed thresholds, explaining missed alarms even when the model is "correct."

RQ3: Is the evidence expressed in the residual space or diluted?

Finding: The choice of residual space dictates what attacks are visible. On WADI, localized attacks were successfully detected by NSIBF (which scores a sensor-innovation subspace) but missed by full-observation detectors. The pooling of evidence across all channels diluted the localized signal. This explains why rankings change across testbeds: NSIBF ranked first on WADI but last on HAI, while TranAD ranked first on HAI but last on SWaT.

RQ4: Does the Stage-2 rule extract available evidence?

Finding: A controlled Stage-2 probe revealed that similar missed alarms can stem from different causes. Some failures were due to weak residual evidence (attack barely moves the residual), others to drift-dominated evidence (attack energy < drift), and others to threshold extraction failures (evidence exists but the fixed threshold misses it).

5. Significance and Contributions

The paper claims four main contributions:

Decision-Rule-Independent Evaluation Object: It formalizes the evaluation of the Stage-1 residual representation separately from the alarm rule, allowing researchers to ask if attack evidence is present before any thresholding or smoothing is applied.
Residual Information-Energy Scale: It introduces a normalized residual energy metric with an exact KL-divergence interpretation, providing a common scale to compare heterogeneous detectors (graph, transformer, observer, invariant) across different residual spaces.
Failure-Attribution Diagnostics: It provides a suite of metrics to distinguish between specific failure modes: weak evidence, reference instability, dimensional dilution, and Stage-2 calibration failure. This moves evaluation beyond a single leaderboard number to a diagnostic tool.
Uniform Cross-Detector Study: It demonstrates that detector rankings do not transfer reliably across testbeds and that similar alarm-level failures often have distinct underlying causes (e.g., representation redesign vs. reference stabilization).

Conclusion:
The authors argue that detection failure in CPS is multifaceted. By separating the residual representation from the decision rule, the proposed framework reveals that "detection failure" is not a monolithic concept. It can arise from the detector's inability to see the attack, the instability of the normal baseline, the dilution of signals in high-dimensional spaces, or the suboptimal extraction of evidence by the alarm policy. This approach enables more targeted improvements, such as redesigning representations for localized attacks or stabilizing references against drift, rather than simply tuning thresholds.