Surprised by Attention: Predictable Query Dynamics for Time Series Anomaly Detection

The paper introduces AxonAD, an unsupervised time series anomaly detector that improves sensitivity to structural dependency shifts by combining reconstruction error with a novel query mismatch score derived from predicting the evolution of multi-head attention vectors.

The Gist

Imagine you are driving a car. Usually, when you turn the steering wheel to the left, the car turns left. When you press the gas, the car speeds up. These actions happen in perfect harmony.

Now, imagine a glitch where you turn the steering wheel hard to the left, but the car keeps driving straight. The speed is normal, the engine is humming, and the steering wheel isn't broken. But the relationship between your steering and the car's movement is broken.

Most computer programs designed to find "bad" data (anomalies) look for things that are obviously wrong, like a speedometer reading 500 mph or a steering wheel stuck at 90 degrees. They miss the subtle "coordination breaks" where everything looks normal individually, but the team is failing to work together.

This paper introduces a new detective called AxonAD that is specifically designed to catch these "coordination breaks." Here is how it works, using some simple analogies:

1. The Problem: The "Good Actor" Trap

Imagine a theater troupe. If an actor forgets their lines, the play stops. But what if the actor remembers their lines perfectly but starts improvising a completely different scene that doesn't match the other actors? The audience (the computer) might think, "Wow, that actor is great at memorizing lines!" and miss the fact that the scene is falling apart.

In data terms, old methods try to reconstruct the data (like memorizing the script). If the data looks "plausible" (the actor sounds good), the old methods say, "Everything is fine." They miss the fact that the connection between the steering and the turning is gone.

2. The Solution: Predicting the "Next Move"

AxonAD doesn't just look at what the data is; it looks at what the data should do next.

Think of a conductor leading an orchestra.

• The Old Way: The conductor listens to the music and checks if the notes are in tune. If the notes are in tune, the music is good.
• The AxonAD Way: The conductor has a crystal ball. They know exactly what the violin should be playing next based on what the drums just played.
  • Normal: The violin plays the note the conductor predicted. The "prediction" and the "reality" match perfectly.
  • Anomaly: The violin plays a totally different note. The conductor is surprised! "Wait, that's not what I expected!"

In the paper, this "crystal ball" is called Predictable Query Dynamics. The system learns the "rhythm" of how different data channels (steering, speed, acceleration) talk to each other. It predicts the next "move" in this conversation.

3. The Two-Part Score

AxonAD gives the data a score based on two things:

1. The "Amplitude" Check (Reconstruction Error): "Does the data look weird on its own?" (e.g., Is the speed too high?)
2. The "Surprise" Check (Query Mismatch): "Did the data do something unexpected in its relationship with other data?" (e.g., Did the steering turn without the car turning?)

If a car is driving at 200 mph (weird amplitude), the first check catches it.
If a car is driving at 60 mph but the steering is locked while the car drifts sideways (weird relationship), the second check catches it.

By combining these two, AxonAD catches the "silent killers"—the glitches that look normal on the surface but are actually dangerous.

4. How It Learns (The "Shadow Teacher")

To learn this rhythm without needing a teacher to point out every mistake, AxonAD uses a clever trick called EMA (Exponential Moving Average).

Imagine a student (the AI) and a teacher (the "Target").

• The teacher is a "slow-moving" version of the student. The teacher updates their knowledge very slowly, smoothing out the noise.
• The student tries to predict what the teacher would do next.
• If the student guesses wrong, they learn.
• Because the teacher is "slow," the student learns the true, stable rhythm of the data, rather than just memorizing random noise.

5. The Results

The authors tested this on real car data and a huge collection of other time-series data (like energy grids and server logs).

• The Result: AxonAD found the "coordination breaks" that other methods missed.
• The Analogy: If other methods were like security guards looking for people carrying big weapons, AxonAD is like a body language expert who spots a spy because they are walking slightly out of sync with the crowd, even though they aren't carrying anything.

Summary

AxonAD is a smart anomaly detector that stops asking "Is this number weird?" and starts asking "Is this number behaving like it should with its friends?"

It's like having a co-pilot who knows exactly how your car should react to your inputs. If you turn the wheel and the car doesn't turn, the co-pilot doesn't wait for the car to crash; they immediately yell, "Something is wrong with the connection!" before anything bad happens.

Technical Summary

1. Problem Statement

Multivariate time series anomalies, particularly in complex systems like autonomous vehicles, often manifest not as simple amplitude excursions (values going out of range) but as shifts in cross-channel dependencies (coordination breaks).

• The Limitation of Current Methods: Traditional residual-based detectors (e.g., VAEs, LSTMs, Transformers) rely on reconstructing input signals. If a flexible sequence model can still plausibly reconstruct individual channel amplitudes despite a broken coordination pattern (e.g., steering input no longer matching lateral acceleration), these models fail to detect the anomaly because the reconstruction error remains low.
• The Core Insight: While the values might look normal, the internal attention mechanisms required to model the relationship between channels should evolve predictably over short horizons under normal dynamics. Structural anomalies disrupt this predictability.

2. Methodology: AxonAD

AxonAD is an unsupervised detector that treats the evolution of multi-head attention query vectors as a predictable process. It combines a reconstruction pathway with a predictive attention pathway.

Architecture Components

1. Online Reconstruction Pathway:
   • Uses bidirectional self-attention to reconstruct the input window $X$.
   • Generates a standard reconstruction error score ($d_{rec}$) based on Mean Squared Error (MSE).
2. Predictive Attention Pathway (The Core Innovation):
   • History-Only Predictor: A causal Temporal Convolutional Network (TCN) forecasts future query vectors ($\hat{Q}_{pred}$) based only on past context (time-shifted embeddings). It does not predict keys, values, or attention maps.
   • EMA Target Encoder: A target encoder with parameters updated via Exponential Moving Average (EMA) provides stable ground-truth query vectors ($Q_{tgt}$) for the current window.
   • Training Objective: The model is trained using a masked JEPA-style (Joint Embedding Predictive Architecture) objective. It minimizes the cosine distance between the predicted queries and the EMA target queries on masked future timesteps.
3. Inference Scoring:
   • Query Mismatch Score ($d_q$): At inference, the model calculates the cosine distance between the predicted queries and the EMA target queries on the tail (most recent) timesteps of the window.
   • Final Score: The final anomaly score $S$ is the sum of the robustly standardized reconstruction error and the query mismatch score:
     $$S = r_z(d_{rec}) + r_z(d_q)$$
   • This additive approach allows the system to detect anomalies that are either amplitude-based (high $d_{rec}$) or coordination-based (high $d_q$).

Key Design Choices

• Query Prediction: Predicting queries specifically is chosen because they control attention routing. If the coordination between channels changes, the model's "intention" (query) to attend to specific features will deviate from the stable target, even if the raw data looks normal.
• Tail Aggregation: The mismatch is aggregated over the last $k$ timesteps to focus on the most recent dynamics, making the detector sensitive to sudden shifts.
• No Label-Based Calibration: The method uses robust statistics (Median and IQR) from nominal training data to standardize scores, avoiding the need for labeled thresholds.

3. Key Contributions

1. Predictive Query Dynamics: A novel approach that treats attention query vectors as temporally predictable signals rather than one-shot routing decisions. This provides sensitivity to structural dependency shifts that residual-based methods miss.
2. Complementary Scoring: Introduces a "query mismatch" score based on cosine distance in query space, which complements reconstruction residuals. This dual-signal approach captures both amplitude and coordination anomalies.
3. Stable Training Scheme: Utilizes an EMA target encoder with masked supervision in query space, avoiding direct supervision on attention maps or value outputs, which can be noisy.
4. State-of-the-Art Performance: Demonstrates significant improvements over strong baselines on both proprietary automotive telemetry and the standard TSB-AD benchmark.

4. Experimental Results

The authors evaluated AxonAD on two datasets:

1. Proprietary In-Vehicle Telemetry: 80,000 timesteps, 19 channels (steering, throttle, acceleration, etc.). Anomalies were primarily "coordination breaks."
2. TSB-AD Multivariate Suite: 17 datasets, 180 time series, covering various domains.

Key Findings:

• Proprietary Telemetry: AxonAD achieved an AUC-PR of 0.285, significantly outperforming the next best baseline (SISVAE at 0.128). It showed massive gains in Event-F1 (0.420 vs 0.255) and Range-F1, indicating superior temporal localization of coordination anomalies.
• TSB-AD Benchmark: AxonAD achieved the highest mean AUC-PR (0.437), VUS-PR (0.493), and Range-F1 (0.471) across all 180 series. It won on a clear majority of series against all baselines (Wilcoxon signed-rank test $p < 10^{-4}$).
• Ablation Studies:
  • Removing the query branch ("Recon only") significantly dropped performance, confirming the necessity of the query mismatch signal.
  • Predicting keys, values, or attention maps was inferior to predicting queries.
  • The combination of reconstruction and query scores was essential for the best aggregate performance.
• Efficiency: Inference latency is extremely low (0.069 ms per window), making it suitable for real-time fleet monitoring (10 Hz processing).

5. Significance

• Solving the "Coordination Break" Problem: This paper addresses a critical gap in time series anomaly detection where standard reconstruction models fail because they can "hallucinate" plausible values even when the underlying system logic is broken.
• Interpretability of Attention: By monitoring the predictability of the attention mechanism itself, AxonAD offers a new diagnostic signal that reflects the model's internal understanding of system dynamics.
• Practical Deployment: The method is fully unsupervised, requires no threshold tuning on labeled data, and has low inference latency, making it highly viable for safety-critical applications like autonomous driving and industrial IoT.
• Generalizability: The success on the diverse TSB-AD suite suggests the approach is robust across different domains, not just automotive telemetry.

In summary, AxonAD shifts the paradigm from "reconstructing the data" to "predicting the model's attention," effectively detecting anomalies where the relationship between variables breaks down, even if the variables themselves appear normal.