ECoLAD: Deployment-Oriented Evaluation for Automotive Time-Series Anomaly Detection

The paper introduces ECoLAD, a deployment-oriented evaluation protocol that reveals how throughput constraints on automotive hardware often render deep learning anomaly detectors infeasible before they lose accuracy, whereas lightweight classical methods maintain both coverage and detection performance.

The Gist

Imagine you are the chief engineer for a fleet of self-driving cars. Your job is to install a "health monitor" system that listens to the car's engine, brakes, and sensors 24/7 to spot tiny glitches before they cause a crash.

You have a list of 10 different "detective algorithms" (software programs) that claim to be the best at spotting these glitches. On a powerful desktop computer in your office, they all seem amazing. They catch almost every problem.

But here's the catch: You can't put a super-computer inside a car. The car's computer is small, slow, and can only do one thing at a time (it's "single-threaded"). If you install a detective that is too heavy, it will slow the car down, miss the glitch, or even crash the car's system.

This paper, ECoLAD, is a new way to test these detectives. Instead of just asking, "Who is the smartest?" it asks, "Who can do the job without breaking the car?"

Here is the breakdown using simple analogies:

1. The Problem: The "Gym" vs. The "Backpack"

Most researchers test these algorithms in a "Gym" (a powerful workstation with unlimited power). They rank them based on who finds the most errors.

• The Reality: Putting that algorithm in a car is like asking a bodybuilder to run a marathon while carrying a heavy backpack.
• The Issue: Some algorithms are like bodybuilders. They are strong (very accurate) but heavy. When you force them into the car's "backpack" (limited CPU power), they collapse. They become too slow to keep up with the car's speed.
• The Result: A leaderboard that only shows "Smartest Detective" is misleading. It might pick a bodybuilder who can't run.

2. The Solution: The "Staircase Test" (The Ladder)

The authors created a protocol called ECoLAD (Efficiency Compute Ladder for Anomaly Detection). Imagine a staircase with four steps, each getting harder:

• Step 1 (Top): A super-fast GPU (The Gym).
• Step 2: A multi-core CPU (A decent laptop).
• Step 3: A limited-core CPU (A standard laptop).
• Step 4 (Bottom): A single-core CPU (The Car's brain).

They take every detective and force them to walk down this staircase. As they go down, they have to shrink their brain size (reduce their computing power) to fit.

• The Rule: If a detective gets too slow or stops working on the bottom step, they are out, no matter how smart they were at the top.

3. The Findings: Who Survived the Climb?

When they ran this test on real car data, they found some surprising things:

• The "Heavyweights" (Deep Learning Models): Some fancy, complex AI models (like OmniAnomaly or TimesNet) were great at the top of the stairs. But as soon as they hit the "Car Step," they became too slow. They were like a Ferrari trying to drive through a muddy field; they just got stuck.
• The "Lightweights" (Classical Methods): Simpler, older algorithms (like HBOS or COPOD) were already fast. When they went down the stairs, they didn't just survive; they actually got faster because they had less weight to carry. They are like a nimble hiker who can run through the mud easily.
• The "Fragile" Ones: Some algorithms (like LOF) were fast but their accuracy dropped drastically when they had to shrink. They were like a glass statue: fast, but they broke when the pressure got high.

4. The "Throughput" Trap

The paper introduces a concept called Throughput. Imagine a conveyor belt of car data moving at 500 items per second.

• If your detective can only check 100 items per second, you have a bottleneck. The belt keeps moving, and your detective misses the bad items.
• ECoLAD measures exactly how many items per second each detective can handle before they start missing things.

5. The Big Takeaway

The paper concludes that accuracy isn't everything.

If you are building a system for a car, you shouldn't just pick the "smartest" algorithm. You need to pick the one that is fast enough to run in real-time on the car's limited hardware.

• Old Way: "Look, this AI has 99% accuracy! Buy it!" (Then you realize it takes 5 seconds to analyze 1 second of data. Useless for a car.)
• ECoLAD Way: "This AI has 85% accuracy, but it can analyze 1,000 seconds of data in 1 second on a cheap chip. Buy it."

Summary Analogy

Think of it like hiring a chef for a busy food truck.

• The Old Test: You hire the chef who can make the most complex, 10-course gourmet meal in a fancy kitchen.
• The ECoLAD Test: You realize the food truck only has one burner and a tiny fridge. You need a chef who can make a good burger quickly on one burner.
• The Paper's Lesson: Don't hire the gourmet chef for the food truck. Hire the one who can actually cook the burger before the customer gets angry.

ECoLAD is the new hiring guide that ensures your "detectives" can actually do the job in the real world, not just in the lab.

Technical Summary

Here is a detailed technical summary of the paper "ECoLAD: Deployment-Oriented Evaluation for Automotive Time-Series Anomaly Detection."

1. Problem Statement

Current Time-Series Anomaly Detection (TSAD) research predominantly relies on accuracy-only leaderboards evaluated on high-performance workstation hardware under unconstrained execution. This approach fails to reflect the realities of in-vehicle deployment, where:

• Latency must be predictable: Real-time monitoring requires stable inference times.
• Resources are limited: Embedded automotive systems often operate with severely restricted CPU parallelism (sometimes near single-threaded execution) and limited compute budgets.
• Ranking instability: Methods that perform well on GPUs or multi-core CPUs may become infeasible or degrade significantly when constrained, yet current benchmarks do not capture this "feasibility drift."

The paper argues that accuracy alone is insufficient; a method is only viable if it meets specific throughput targets (scoring rates) while maintaining stable detection quality under constrained hardware conditions.

2. Methodology: The ECoLAD Protocol

The authors introduce ECoLAD (Efficiency Compute Ladder for Anomaly Detection), a deployment-oriented evaluation protocol designed to systematically test model robustness under resource constraints.

A. Experimental Design

• Compute Ladder: The study defines four distinct execution tiers to simulate a gradient of resource reduction:
  1. GPU: Reference configuration (unlimited threads, hardware acceleration).
  2. CPU-MT: Multi-threaded CPU (14 threads).
  3. CPU-LT: Limited-thread CPU (7 threads).
  4. CPU-1T: Single-threaded CPU (1 thread), simulating a conservative embedded stress test.
• Mechanical Scaling Rules: Instead of re-tuning hyperparameters for each tier, the protocol applies monotone, integer-only scaling rules to reduce model capacity as compute decreases.
  • Workload ($v_{work}$): Scaled linearly with the reduction factor $s$.
  • Width/Heads ($v_{width}, v_{heads}$): Scaled with $\sqrt{s}$.
  • Depth ($v_{depth}$): Scaled with $s^{1/4}$.
  • Window Size: Scaled with $\sqrt{s}$ (minimum 8).
  • Note: Parameters affecting decision semantics (e.g., contamination rates) are not scaled.
• Throughput Constraints: The protocol sweeps target scoring rates ($\tau$) and measures Coverage (fraction of entities meeting the target) and Achievable AUC-PR (best performance among configurations meeting the target).

B. Datasets and Metrics

• Datasets:
  • Telemetry (Proprietary): 80,000 datapoints from a vehicle powertrain/chassis system (19 features). Anomaly rate $\approx 0.022$.
  • SMD (Public): Server machine monitoring data (22 entities).
  • SMAP (Public): Spacecraft telemetry data.
• Metrics:
  • Primary: AUC-PR (Area Under the Precision-Recall Curve) due to class imbalance.
  • Runtime: Distinguishes between inference-only time ($t_{inf}$) and full-run time ($t_{e2e}$) to separate online scoring capacity from offline training overhead.
  • Throughput: Measured in windows per second (wps).

C. Evaluated Detectors

The study benchmarks 10 detectors across three families:

• Classical: Isolation Forest (IForest), Local Outlier Factor (LOF), HBOS, COPOD, PCA.
• Deep Learning: USAD, TranAD, OmniAnomaly, GDN, TimesNet.

3. Key Contributions

1. Standardized Evaluation Protocol: ECoLAD formalizes compute reduction and CPU parallelism caps as first-class variables, providing an auditable, reproducible framework for deployment-oriented evaluation.
2. Feasibility-First Filtering: Introduces a mechanism to identify "feasible" operating points based on throughput targets before considering accuracy, preventing the selection of models that are too slow for deployment.
3. Empirical Evidence of Rank Drift: Demonstrates that method rankings change significantly under constrained execution, often driven by architectural bottlenecks rather than accuracy degradation.

4. Key Results

A. Cross-Tier Detection Quality (RQ1)

• Ranking Instability: Top performers on the GPU tier (e.g., OmniAnomaly on SMD) do not necessarily remain top performers on the CPU-1T tier.
• Domain Sensitivity: On the proprietary Telemetry dataset, classical methods (specifically HBOS) achieved the highest AUC-PR ($\approx 0.055$ on CPU-1T), offering a $\approx 2.9\times$ lift over the random baseline. Deep methods clustered near $0.041$, showing limited separability on this specific signal.
• Stability: Some deep methods (USAD, TranAD) maintained stable AUC-PR across tiers, while others (LOF) degraded sharply in accuracy as resources were reduced.

B. Degradation Modes (RQ2)

The study identified three distinct failure modes:

1. Backend/Overhead-Limited: Models like TimesNet maintained accuracy but suffered massive throughput collapse on CPU (e.g., dropping from ~9,500 wps on GPU to ~1,400 wps on CPU-1T). Their feasibility is lost due to runtime overhead, not accuracy.
2. Quality-Drift-Limited: Models like LOF maintained high throughput but suffered significant accuracy drops (AUC-PR fell from 0.145 to 0.073) when capacity was reduced, indicating sensitivity to model size.
3. Graceful Degraders: HBOS and COPOD retained high throughput and near-flat AUC-PR across all tiers. Notably, HBOS throughput increased on CPU-1T because the reduced work scale resulted in fewer histogram bins to compute.

C. Throughput-Constrained Behavior (RQ3)

• Coverage vs. Target: As the target throughput ($\tau$) increased, deep models (OmniAnomaly, USAD) quickly became infeasible (coverage dropped below 50%), whereas classical methods (HBOS, COPOD) sustained high coverage even at high targets.
• Operating Points: For deep models, meeting high throughput targets often required reducing model capacity to the point where performance fell below the random baseline. Classical methods offered viable operating points across the entire throughput sweep.

5. Significance and Implications

• Shift in Evaluation Paradigm: The paper argues that accuracy-only leaderboards misrepresent feasibility. A model with 0.5 AUC-PR is useless if it cannot run at 500 Hz on an embedded ECU.
• Hardware Awareness: The results highlight that hardware choice (GPU vs. CPU) and parallelism caps can dominate practical feasibility independently of accuracy.
• Recommendation for Deployment: The authors advocate for a "feasibility-first" approach:
  1. Screen detectors for throughput feasibility under target constraints (e.g., CPU-1T).
  2. Select the best-performing model among the feasible set.
• Classical vs. Deep: In the context of constrained automotive telemetry, lightweight classical detectors (HBOS, COPOD) proved more robust and feasible than complex deep learning models, which often suffer from architectural bottlenecks when parallelism is restricted.

In conclusion, ECoLAD provides a critical framework for bridging the gap between theoretical TSAD performance and real-world automotive deployment, emphasizing that predictable latency and resource efficiency are as important as detection accuracy.