🤖 machine learning

Towards Principled Continual Anomaly Detection: A Systematic Framework and Benchmark Scenarios

This paper introduces a systematic framework for designing reproducible continual anomaly detection benchmarks in tabular domains by discovering, filtering, and ordering tasks to overcome the limitations of arbitrary splits, ultimately providing five new benchmark scenarios derived from large-scale cybersecurity datasets.

Original authors: Kamil Faber, Mateusz Smendowski, Roberto Corizzo

Published 2026-07-22

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Kamil Faber, Mateusz Smendowski, Roberto Corizzo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to spot trouble. In the world of computer science, this is called anomaly detection. Usually, you show the robot a million pictures of "normal" things—like a quiet street or a calm server room—and a few pictures of "bad" things, like a car crash or a hacker attack. The robot learns what normal looks like so it can scream "ALERT!" when something weird happens.

But here is the tricky part: the world doesn't stay still. The "normal" street changes; the "normal" server room gets busier. This is called a non-stationary environment. If you just keep showing the robot new problems without letting it forget the old ones, it gets confused. This is the challenge of Continual Anomaly Detection (CAD). It's like asking a student to learn a new language every week while still remembering how to speak the ones they learned last year, all without a teacher to grade them. The big question scientists are asking is: How do we build a robot that can keep learning new tricks without forgetting the old ones?

The problem is that most of the tests we use to check if these robots are smart are broken. Imagine trying to test a student's ability to learn new languages by just handing them random pages of text. If the pages are all about the same topic, the student looks smart but hasn't really learned anything. If the pages are too different, the student fails instantly. Until now, researchers have been guessing how to split up their data to create these tests, often making up rules that don't actually reflect real life. This paper, written by Kamil Faber, Mateusz Smendowski, and Roberto Corizzo, says, "Stop guessing. Let's build a proper test."

The "Recipe" for a Better Test

The authors realized that creating a good test for these learning robots is harder than just chopping up a dataset. They noticed that if you just slice data by time (like "Monday's data" vs. "Tuesday's data"), you might end up with two days that are actually identical. The robot would think it's learning something new, but it's just seeing the same thing twice. That's a boring test. On the other hand, if you slice it too randomly, the robot might face a problem so hard it gives up immediately.

To fix this, the team built a systematic framework. Think of it as a super-smart chef who doesn't just throw ingredients into a pot. Instead, this chef has a strict recipe for making the perfect "Continual Learning Stew."

Here is how their recipe works:

Finding the Ingredients (Task Discovery): First, the framework looks at a huge pile of data (like a cybersecurity log) and tries to find natural groups. It uses math tricks, like clustering, to see which pieces of data naturally hang together. It's like looking at a box of mixed Lego bricks and grouping them by color or shape, rather than just grabbing a handful randomly.
Tasting the Dish (Task Evaluation): Before serving the test, the framework runs a "taste test." It trains a simple, single-task robot on each group of data to see if the group is actually learnable. If a group is too easy (the robot solves it instantly) or too hard (the robot can't learn it at all), it gets thrown out.
Checking for Redundancy (Filtering): The framework checks if any two groups are basically the same. If Group A and Group B are twins, you don't need both in the test. It also checks if learning Group A makes Group B too easy. If the robot learns Group A and suddenly knows Group B without trying, that's a bad test because it's not actually testing "continual" learning.
Arranging the Menu (Ordering): This is the most creative part. Once they have a list of good, distinct tasks, they have to decide the order in which the robot sees them. The authors created six different "menus" or orderings:
- Smooth Drift: The robot sees changes that happen slowly, like a sunset.
- Abrupt Drift: The robot sees sudden, shocking changes, like a light switch flipping.
- Curriculum: The robot starts with easy tasks and gets harder (or vice versa).
- Generalization: The robot sees tasks that help it learn others, or tasks that are very specific.

The Five New "Games"

Using this framework, the authors didn't just write a theory; they actually built five new benchmark scenarios (which are like new video games for the robots to play). They used three massive datasets from the world of cybersecurity (CICIDS2017, CICIDS2018, and CIC-UNSW-NB15). These datasets are full of records of computer traffic, some normal and some representing cyber-attacks.

They created:

Three single-dataset scenarios: These test how a robot handles changes within one specific type of network.
Two multi-dataset scenarios: These are the "boss levels." They mix data from different sources, forcing the robot to adapt to completely different environments.

The result is a set of five distinct "games," each with 5 to 13 different "levels" (tasks). For each game, they provided the six different orderings mentioned above. This means researchers can now test their robots in a fair way, seeing if the robot can handle a slow drift, a sudden shock, or a mix of easy and hard levels.

What They Found (and What They Didn't)

The authors ran their new tests on several different types of anomaly detection models. They found that:

The tests are hard but fair: When they let the robots learn without any special memory tricks (a "Naive" approach), the robots forgot a lot of what they learned. This is good! It means the test is actually challenging. If the robots didn't forget anything, the test would be too easy.
Memory helps: When the robots were allowed to keep a small "replay buffer" (a tiny memory of old data), they performed much better. This confirms that these new scenarios are good at testing whether a robot can actually remember its past.
Not all robots are equal: Some models were great at learning new things but terrible at remembering old ones. Others were okay at both. The new framework helps us see these differences clearly.

However, the authors are careful not to claim they have "solved" the problem of continual learning. They didn't invent a new robot that wins every game. Instead, they built a better scoreboard and a better set of games. They showed that previous ways of testing were often flawed because the tasks weren't distinct enough or were ordered poorly.

They also noted a few limits. Their tests are currently only for cybersecurity data (computer traffic). They don't know yet if this framework works perfectly for medical data or stock market data, though they suspect it might. Also, building these tests took a lot of computer power because they had to train many "single-task experts" to check if the tasks were good.

Why This Matters

Imagine if you were trying to teach a child to play soccer. If you only let them play against a wall that never moves, they might think they are a pro. But if you put them on a field with real players who change their strategy every minute, you'll see if they are actually good.

This paper is like building that real field. It gives scientists a principled, reproducible way to test if their "robots" are truly learning and adapting, or if they are just memorizing a trick. By providing these five scenarios and six orderings, the authors are saying, "Here is a fair playing field. Now, let's see who can actually learn."

The paper concludes that while they haven't fixed the robots themselves, they have fixed the way we measure them. This is a crucial step forward because, as the authors point out, you can't improve a system if you can't measure it correctly. They have handed the community a new set of tools to ensure that the next generation of anomaly detectors is truly ready for the changing world.

Technical Summary: Towards Principled Continual Anomaly Detection

Problem Statement

Continual Anomaly Detection (CAD) investigates how models can adapt to evolving data distributions while retaining performance on previously observed regimes. However, a critical gap exists in the evaluation of CAD methods, particularly within tabular domains. Unlike vision-based CAD, where task boundaries are often grounded in semantically meaningful units (e.g., object categories), tabular anomaly detection datasets rarely provide validated task boundaries.

Current approaches often construct CAD scenarios using arbitrary chronological windows, metadata fields, or manual partitions. These methods risk creating unlearnable, redundant, or overly transferable tasks that obscure genuine continual-learning dynamics. Consequently, tabular CAD lacks reliable, reusable benchmark scenarios and principled mechanisms for transforming existing static anomaly-detection datasets into validated continual settings. This paper addresses the scenario-design problem directly, treating scenario construction not as an experimental detail but as a first-class benchmark-design challenge.

Methodology

The authors propose a systematic, reproducible framework for transforming existing tabular anomaly-detection datasets into validated CAD scenarios. The framework operates through six distinct stages:

Task Discovery: The framework generates candidate task sets from preprocessed datasets using three mechanisms:
- Natural Splits: Leveraging inherent temporal, contextual, or acquisition-based structures (e.g., partitioning by capture day).
- Clustering-Based Discovery: Inferring task structure from data distribution using variants such as joint clustering of normal/anomalous samples, clustering only normal samples with random anomaly assignment, or clustering normal samples with nearest-centroid anomaly assignment.
- Multi-Dataset Option: Constructing scenarios across multiple datasets, treating each dataset as a single task or decomposing them further.
Task Evaluation: Candidate tasks are evaluated using Single-Task Experts (STEs). A cross-task performance matrix ( $M^f$ ) is generated where models are trained on one task and tested on all others. This empirical view exposes learnability, transfer, and redundancy.
Task Selection (Filtering): Five filtering criteria (FCs) are applied iteratively to remove unsuitable tasks:
- FC1 (Self-learnability): Retains only tasks learnable by at least one model above a random baseline.
- FC2 & FC3 (Transfer Coverage): Filters tasks that are too easy (covered by many other tasks) or too dominant (allowing a model trained on them to solve too many other tasks trivially).
- FC4 & FC5 (Redundancy): Removes tasks with nearly identical source or target transfer profiles.
Scenario Orderings: For the retained task sets, the framework defines six principled ordering families to expose different continual-learning dynamics:
- Drift-based: Smooth drift (minimizing cumulative distance between tasks) and Abrupt drift (maximizing shifts).
- Curriculum-based: Ascending (easy-to-hard) and Descending (hard-to-easy) based on empirical difficulty.
- Generalization-based: Ascending and Descending based on a task's ability to serve as a source of transferable knowledge.
Scenario Selection: The final scenario is selected via cross-model concordance (using Kendall's W) to ensure the ordering structure is consistent across models with different inductive biases. A consensus ordering is derived using the Borda count.
Scenario Validation: The final scenarios are validated against three properties:
- Feasibility: The scenario must be learnable by strong reference strategies (e.g., Single-Task Experts or Cumulative training).
- Non-triviality: The scenario cannot be solved by naive sequential training alone; it must require retention mechanisms.
- Forgetting: The scenario must induce measurable forgetting under naive training, distinguishing it from weak or redundant task sequences.

Key Contributions

A Principled Framework: A formalized, reproducible pipeline for converting static tabular anomaly detection datasets into validated continual scenarios, encompassing task discovery, filtering, ordering, and validation.
Six Ordering Families: Definition of complementary task orderings (drift, curriculum, generalization) that expose diverse continual-learning dynamics from the same task set.
Benchmark Suite: The delivery of five benchmark-ready CAD scenarios derived from three large-scale cybersecurity datasets (CICIDS2017, CICIDS2018, and CIC-UNSW-NB15). This includes:
- Three single-dataset scenarios (CAD-CICIDS2017, CAD-CICIDS2018, CAD-CICUNSW).
- Two multi-dataset scenarios (MCAD-CIC-3x1 and MCAD-CIC-3xN) covering cross-dataset continual adaptation.
Reproducible Artifacts: Release of task splits, train-test splits, exact orderings, and validation statistics to establish a foundation for future CAD benchmarking.

Experimental Results

The authors validated the five released scenarios using four diverse anomaly detection models (VAE, Deep SVDD, DAGMM, NeutralAD) and four continual learning strategies (Naive, Replay, Cumulative, MSTE).

Feasibility: Strong reference strategies (MSTE, Cumulative) achieved meaningful ROC-AUC scores (often $\ge 0.8$ ) across all scenarios, confirming that the tasks are not intrinsically degenerate.
Non-triviality: Naive sequential training consistently underperformed compared to Replay, Cumulative, and MSTE strategies. This gap confirms that the scenarios are learnable but require specific retention or adaptation mechanisms, rather than being solvable by simple fine-tuning.
Forgetting: Naive training exhibited significant Forgetting Measure (FM) values, particularly in multi-dataset scenarios (e.g., FM $\approx 46.7$ for VAE on MCAD-CIC-3xN). This demonstrates that the scenarios induce genuine continual-learning pressure.
Diversity: The scenarios cover varying difficulty regimes. For instance, MCAD-CIC-3x1 presents challenges for single-task experts (low MSTE performance for some models), while MCAD-CIC-3xN combines severe continual difficulty with strong single-task learnability.

Significance and Claims

The paper positions itself as a methodological foundation rather than a definitive, domain-complete benchmark. Its primary significance lies in addressing the "scenario-design problem" in tabular CAD. By moving away from arbitrary splits, the framework ensures that benchmark scenarios are empirically meaningful, non-redundant, and capable of exposing genuine forgetting and transfer dynamics.

The authors claim that their work provides a reproducible methodological foundation for future CAD benchmarking. They emphasize that the released scenarios are not interchangeable variants of a single setting but represent a diverse set of challenges (within-dataset regime shifts vs. cross-dataset adaptation). The paper concludes that more principled scenario construction can improve the reliability of anomaly detection research by reducing the risk of overstated claims caused by weak or misleading benchmarks. However, the authors modestly note limitations: the benchmark is currently restricted to cybersecurity datasets, and the framework is computationally demanding due to the extensive single-task expert training required for validation.