🤖 machine learning

DQS: A Low-Budget Query Strategy for Enhancing Unsupervised Data-driven Anomaly Detection Approaches

This paper introduces the Dissimilarity-based Query Strategy (DQS), a novel active learning approach that enhances unsupervised time series anomaly detection by selecting diverse samples for oracle labeling to refine threshold selection, demonstrating superior performance in low-budget scenarios despite potential mislabeling.

Original authors: Lucas Correia, Jan-Christoph Goos, Thomas Bäck, Anna V. Kononova

Published 2026-08-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lucas Correia, Jan-Christoph Goos, Thomas Bäck, Anna V. Kononova

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the guardian of a massive, humming machine factory. Every day, thousands of sensors record the heartbeat of the machines: temperature, vibration, pressure, and speed. Most of the time, these machines hum along in a "nominal" (normal) rhythm. But sometimes, a gear slips or a bearing overheats, creating a weird, "anomalous" glitch. If you catch these glitches early, you save the factory; if you miss them, things break. The problem is that the data is so huge and complex that humans can't watch every single second. So, we use computers to listen for the weird beats.

But here's the catch: computers are great at spotting weirdness, but they are terrible at knowing how weird is "too weird." It's like a smoke detector that beeps at the smell of toast but stays silent during a fire because you never told it what a real fire smells like. To fix this, we usually need to show the computer examples of "normal" and "broken" machines so it can learn the line in the sand. But in the real world, we rarely have a pile of labeled "broken" examples ready to go. This paper lives in the corner of science called unsupervised anomaly detection, where we try to find the bad apples without a pre-made list of what bad apples look like. The big question is: if we can't label everything, how do we teach the computer just enough to get it right without spending a fortune on human experts?

The authors of this paper, Lucas Correia and his team, decided to tackle this by mixing two ideas: unsupervised learning (letting the computer guess on its own) and active learning (asking a human expert for help only when absolutely necessary). They call their new method DQS (Dissimilarity-based Query Strategy).

Think of the computer as a detective trying to find a thief in a crowd. The detective has a "suspicion score" for everyone. If the score is too high, the person is a thief; if it's low, they are innocent. But the detective doesn't know where to draw the line. Usually, the detective just guesses a line, which often leads to mistakes. The authors suggest a smarter way: instead of asking the human expert to label random people, or just the people the detective thinks are most suspicious, the detective should ask about people who are completely different from each other.

Here is how their new strategy, DQS, works: Imagine the detective has a list of "suspicion scores" for everyone. To pick the next person to ask the human expert about, DQS looks at the scores and asks, "Who here is the most unlike the people I've already asked about?" They use a mathematical tool called Dynamic Time Warping (DTW) to measure this "unlikeness." You can think of DTW as a way to compare two songs that might be played at different speeds. Even if one song is fast and the other is slow, DTW can figure out if they are the same tune or totally different. DQS uses this to find anomaly scores that are as different as possible from the ones already checked. By picking the most diverse, "weirdly different" samples to show the human, the detective learns the line in the sand much faster and more accurately.

The paper tests this idea on a dataset called PATH, which simulates the complex, shifting behavior of a car's engine over time. They compared DQS against three other ways of asking for help:

Random: Just picking people to ask about by rolling a dice.
Top: Only asking about the people with the highest suspicion scores.
Uncertainty: Asking about people whose scores are right on the edge of the current guess.

The results, measured by a score called F1 (which balances finding all the bad apples while not crying wolf too often), showed some interesting things. When the human expert has a very small budget (meaning they can only label a few samples, like 1 or 5), DQS is the clear winner. It finds the best threshold much better than the other methods. However, the authors also tested what happens if the human expert makes mistakes (mislabeling). They simulated scenarios where the expert was wrong 10%, 20%, or even 30% of the time. In these messy, realistic situations, DQS still did well, but the "Top" strategy (asking about the most suspicious ones) turned out to be a bit more stubborn and robust against the human's errors.

Crucially, the paper shows that no single strategy is perfect in every situation. DQS isn't a magic bullet that beats everything forever. In fact, when the budget is huge (10 samples), the other methods sometimes catch up or even beat DQS on specific days. But the most important finding is that any of these smart asking strategies are better than just guessing the line without asking anyone at all. Even when the human expert is making mistakes, using a strategy to pick who to ask about still leads to better results than the computer trying to guess on its own.

The authors conclude that if you have the chance to ask a human expert for help, you should definitely do it, but you should be smart about who you ask. If you have very little time or money for labeling, use the DQS method to pick the most diverse samples. If you are worried your expert might get tired and make mistakes, the "Top" strategy might be safer. The paper doesn't claim to have solved the problem of anomaly detection forever, but it suggests that by being strategic about how we ask for help, we can get much closer to the perfect answer without needing a mountain of labeled data.

Technical Summary: DQS – A Low-Budget Query Strategy for Enhancing Unsupervised Data-driven Anomaly Detection Approaches

1. Problem Statement

Truly unsupervised approaches for time series anomaly detection are scarce in the literature. Existing methods often suffer from poorly set thresholds, which significantly hampers detection performance. While some approaches claim to be unsupervised, they frequently rely on labeled data subsets for calibration or hyperparameter tuning, a resource often unavailable in real-world scenarios. Even when truly unsupervised methods exist (e.g., using a Temporal Variational Autoencoder, TeVAE), the threshold selection process remains a critical bottleneck. Correia et al. [12] have demonstrated a significant performance gap between unsupervised thresholding and an optimal threshold, indicating untapped potential.

Furthermore, existing active learning literature in this domain typically assumes a "perfect oracle" (100% labeling accuracy), an idealized assumption that ignores the reality of human error, fatigue, and the difficulty of labeling certain samples. There is a lack of research investigating how mislabeling rates impact anomaly detection performance when using active learning for threshold selection.

2. Methodology

2.1 Active Learning Framework

The proposed approach integrates active learning into an existing unsupervised anomaly detection pipeline. The system operates in a loop involving:

The Oracle: A domain expert who labels unlabelled samples as nominal or anomalous.
The Model: An unsupervised anomaly detector (specifically TeVAE) trained on unlabelled data.
The Query Strategy: A mechanism to select which samples from a candidate pool should be sent to the oracle for labeling, constrained by a query budget $B$ .
The Feedback Strategy: The process of using the newly labeled data to refine the decision threshold, thereby improving detection performance.

2.2 Dissimilarity-Based Query Strategy (DQS)

The core contribution of this work is the Dissimilarity-Based Query Strategy (DQS). This is a non-parametric strategy designed to maximize the diversity of queried samples without requiring prior parameterization.

Mechanism: DQS evaluates the similarity between anomaly scores of different time series using Dynamic Time Warping (DTW). Unlike metrics such as Mean Squared Error, DTW can handle sequences of variable lengths and accounts for temporal misalignment.
Selection Logic:
1. Initialization: For the first query in a round, a random sample is selected from the candidate score set ( $C_{as}$ ).
2. Diversity Maximization: For subsequent queries within a round, the algorithm identifies the candidate sample that is most dissimilar to the most similar sample already in the query set ( $Q_{as}$ ).
3. Process:
  - Calculate the DTW distance between all candidates in $C_{as}$ and all samples in $Q_{as}$ .
  - Identify the candidate ( $o$ ) in $C_{as}$ that has the minimum distance to any sample in $Q_{as}$ (the "most similar").
  - Identify the candidate ( $r$ ) in $C_{as}$ that has the maximum distance to $o$ (the "most dissimilar").
  - Move $r$ to $Q_{as}$ .
Threshold Refinement: Once the budget $B$ is exhausted for a round, the labeled samples in $Q_{as}$ are used to perform a grid search for an optimal threshold $\tau$ , replacing the standard unsupervised threshold ( $\tau_{us}$ ).

2.3 Experimental Setup

Dataset: The PATH dataset, representing automotive powertrain dynamics, was used. It consists of multivariate time series with non-trivial anomalies.
Simulation of Real-World Conditions: The data was split into increasing temporal windows (1 day to 28 days) to mimic gradual data accumulation.
Mislabeling Simulation: To test robustness, ground-truth labels were flipped based on a Bernoulli distribution with mislabeling probabilities ( $p_m$ ) of 0.1, 0.2, and 0.3.
Baselines:
- Unsupervised Threshold ( $\tau_{us}$ ): Maximum error from the validation set.
- Hypothetical Best Threshold ( $\tau_{best}$ ): The optimal threshold found via grid search on the test set (unobservable in practice).
- Random Query Strategy (RQS): Random selection.
- Top-Based Query Strategy (TQS): Selection of samples with the highest anomaly scores (uncertainty-based in some literature).
- Uncertainty-Based Query Strategy (UQS): Selection of samples closest to the current threshold.

3. Key Contributions

Novel Query Strategy (DQS): Introduction of a non-parametric query strategy that leverages DTW to maximize sample diversity in the context of discrete-sequence anomaly detection.
Mislabeling Analysis: A novel testing methodology that explicitly accounts for oracle mislabeling probabilities, addressing a gap in the literature where perfect labeling is often assumed.
Integration of Active Learning: Demonstrating a practical framework for combining active learning with unsupervised models specifically to refine threshold selection, rather than retraining the model itself.

4. Results

4.1 Performance vs. Query Budget

Small Budgets ( $B < 10$ ): DQS consistently outperformed RQS, TQS, and UQS across various time splits (1 to 28 days). It achieved F1 scores significantly closer to the hypothetical best threshold ( $\tau_{best}$ ) than the unsupervised baseline.
Larger Budgets ( $B = 10$ ): DQS remained competitive, often matching or outperforming other strategies, though TQS showed strong performance in specific splits.
General Trend: All query strategies outperformed the unsupervised threshold ( $\tau_{us}$ ) once the training subset grew large enough for the model to learn meaningful representations.

4.2 Impact of Mislabeling

Robustness: While DQS performed best in low-budget, clean-label scenarios, TQS demonstrated higher robustness when faced with higher rates of mislabeling ( $p_m = 0.2, 0.3$ ).
DQS Sensitivity: DQS performance degraded more noticeably with increasing mislabeling rates compared to TQS, particularly in later time splits.
Overall Finding: Despite mislabeling, all active learning strategies outperformed the purely unsupervised threshold ( $\tau_{us}$ ). Even with a 30% error rate, the active learning-based thresholds yielded better F1 scores than the unsupervised baseline.

5. Significance and Claims

The authors claim that this work provides a practical solution for scenarios where labeled data is scarce but an oracle (human expert) is available for limited queries. The significance lies in:

Low-Budget Efficiency: DQS is particularly effective when the cost of labeling is high (small budget), maximizing the information gain per query by ensuring diverse sample selection.
Feasibility of Active Learning: The study validates that employing an active learning-based threshold is recommended whenever feasible, as it consistently outperforms unsupervised thresholding even in the presence of imperfect labeling.
Realistic Evaluation: By introducing mislabeling probabilities, the paper offers a more realistic assessment of active learning strategies in time series anomaly detection, moving beyond the idealized "perfect oracle" assumption.

The paper concludes that while no single strategy dominates all categories (DQS excels in low-budget/clean scenarios; TQS in high-mislabeling scenarios), the integration of active learning for threshold selection is a viable and superior approach to purely unsupervised methods. Future work is suggested to focus on standardizing benchmarking procedures and developing query strategies that balance low-budget efficiency with robustness to mislabeling.