🤖 AI

Strictly Causal Streaming Video Anomaly Detection with a Theoretically-Grounded State-Space Core

This paper introduces a strictly causal, O(1) streaming video anomaly detector based on a theoretically-grounded state-space model with an input-dependent decay gate, which achieves ultra-low latency on edge hardware while demonstrating that its responsiveness is governed by event boundaries rather than base decay, though it currently trails non-causal baselines in accuracy on smaller datasets.

Original authors: Yogesh Kumar

Published 2026-08-26

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yogesh Kumar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quiet hum of a security camera's lens, a constant stream of images arrives, second by second. For decades, computers have struggled to watch this stream in real time, looking for the single moment when something goes wrong—a fall, a fight, a theft. The challenge is not just seeing the event, but seeing it the instant it happens, without waiting to look back at a recording or buffering a chunk of video to analyze later. This is the realm of video anomaly detection, a field where machines must learn the rhythm of normal behavior and instantly flag the beat that breaks. Traditionally, systems have relied on looking at groups of frames at once, like reading a paragraph to understand a sentence, which creates a delay between the event and the alert. But for cameras mounted on robots or mobile devices with limited power, that delay is too long, and the memory required to hold those chunks of video is too heavy. The goal is a system that processes every single frame the moment it arrives, using a tiny, fixed amount of memory, and reacts immediately.

Researchers at the Indian Institute of Technology Jodhpur have built a new kind of detector designed to meet this strict requirement. Instead of storing clips of video, their system treats the incoming stream as a continuous flow, updating its internal understanding with every new picture it receives. At the heart of this system is a mathematical structure that remembers the past but forgets the distant past quickly, unless something important happens. The team trained this system using only videos of normal activity, teaching it to predict what the next frame should look like based on what came before. When the actual frame arrives and does not match the prediction, the system raises an alarm. What makes this work unique is that it does not just work in theory; the researchers proved mathematically how the system's memory settings control its reaction speed, and they tested it on real hardware found in consumer devices, not just powerful supercomputers.

The core of their invention is a mechanism that acts like a gatekeeper for memory. Under normal circumstances, the system holds onto information for a while, smoothing out small changes. However, the researchers designed a specific gate that can slam shut instantly if the incoming image clashes with what the system expects. When this happens, the system effectively resets its memory in a fraction of a second, allowing it to react to an anomaly almost immediately. To understand how fast this should be, the team derived a rule that links the system's memory settings to its reaction time. They found that if the system were to rely only on its standard memory settings, it would take nearly sixty frames to fully react to a sudden change. Yet, when they watched the system in action, it reacted in far less time—sometimes as quickly as one or two frames. This discrepancy revealed that the gate mechanism was doing the heavy lifting, overriding the slow memory settings the moment an irregularity appeared.

The researchers tested their system on two standard datasets used for training security cameras: one featuring a university campus walkway and another showing a busy avenue. They ran the software on an Apple M3 Pro chip, a processor found in modern laptops, to measure how fast it could actually run in the real world. The results were striking for their speed. The system processed each frame in less than one millisecond, achieving a speed of over 1,300 frames per second. This is far faster than the speed of a standard video feed, meaning the system has a massive margin of safety and could easily run on much weaker hardware. However, the accuracy of the system was not perfect. In its initial, un-tuned state, it correctly identified anomalies about 68% of the time on the campus dataset and 70% on the avenue dataset. While this is lower than the 90% or higher scores often seen in other studies, those other studies typically use methods that look at video clips after the fact or require powerful graphics cards that cannot run on edge devices. The author was careful to report these numbers honestly, noting that their method is a first step toward a strictly real-time solution rather than a final, perfected product.

A deeper look at the results revealed a surprising nuance about how the system learns. The researchers tested whether the special "gate" that allows for instant memory resets was always helpful. On the smaller dataset with fewer training videos, removing the gate actually improved the system's accuracy, suggesting the gate was causing the system to over-learn the few examples it had. But on the larger dataset with many more videos, the gate became essential, and removing it caused the accuracy to drop sharply. This suggests that the gate is a powerful tool, but it requires enough data to learn how to use it correctly without getting confused. The team also tested different sizes for the system's internal memory and found that on the smaller dataset, a smaller memory worked better, while on the larger one, a bigger memory helped slightly. These findings indicate that the system's design is not a one-size-fits-all solution; its components interact with the amount of data available in complex ways.

The study concludes that while the system is not yet the most accurate detector available, it successfully bridges a critical gap between theory and practice. It proves that a strictly causal system—one that never looks ahead and never buffers clips—can run on consumer hardware with incredible speed. The researchers acknowledge that their work is incomplete; they have not yet tested the system on a third, larger dataset, and they have not yet optimized the system's settings to close the gap in accuracy with older methods. They also note that their current evaluation of exactly where an anomaly happens in a video is an approximation, and a more precise test would require different tools. Nevertheless, the work provides a clear blueprint for how to build video understanding systems that are fast, efficient, and truly real-time, moving the field away from heavy, delayed processing toward a future where cameras can think and react the instant something unusual occurs.

Technical Summary: Strictly Causal Streaming Video Anomaly Detection with a Theoretically-Grounded State-Space Core

Problem Statement
Video anomaly detection (VAD) systems are increasingly deployed on edge devices (cameras, robots, mobile) where compute, memory, and power are constrained. Existing State-Space Model (SSM) approaches, such as VADMamba, address efficiency but fail to meet three critical constraints simultaneously:

Strict Causality: They often buffer clips or windows internally, violating the requirement that a prediction at time $t$ depends only on frames $1, \dots, t$ with bounded per-step compute.
Theoretical Latency: They offer no analytical account of how a model's temporal memory relates to detection latency.
Hardware Reality: They benchmark efficiency via GPU throughput rather than measuring performance on the target edge hardware.

Methodology
The author proposes a strictly causal streaming VAD detector with a fixed-size state updated in $O(1)$ time and memory per frame, requiring no lookahead or clip buffering.

Architecture:
- Frozen Backbone: Frames are mapped to embeddings via a frozen visual backbone (ResNet-18 or DINOv2). The temporal model operates solely in this embedding space.
- Causal Diagonal SSM Core: The core consists of a stack of causal diagonal state-space layers. The state update follows a linear recurrence: $s_t = a_t \odot s_{t-1} + u_t$ .
- Event-Boundary Gate: A key innovation is the input- and state-dependent decay gate $g_t = \sigma(\text{MLP}([LN(x_t), s_{t-1}]))$ . The effective decay is $a_t = \bar{a} \odot g_t$ . This gate allows the model to "fast forget" (drop the effective decay $a_t$ ) when the current input is inconsistent with the current state, enabling rapid reaction to anomalies without globally sacrificing stability.
- Training: The model is trained self-supervised on normal-only data using a causal next-embedding prediction objective ( $\min \|\hat{e}_{t+1} - e_{t+1}\|^2$ ).
Theoretical Analysis (Decay–Delay Relationship):
The paper derives a closed-form relationship between the recurrence's decay rate ( $a$ ) and detection delay.
- Proposition 1: For a step-change anomaly, the settling delay $\delta(\epsilon)$ required for the state to move within a fraction $\epsilon$ of the new steady state is $\delta(\epsilon) = \lceil \ln \epsilon / \ln a \rceil$ .
- Implication: Channels with slower decay (larger $a$ ) take strictly longer to react. The author uses this to predict a theoretical delay bound based on the learned base decay $\bar{a}$ .

Key Contributions

Strictly Causal Formulation: A concrete architecture where every operation is an $O(1)$ per-frame update, ensuring hard latency guarantees.
Theoretical Latency Analysis: A derivation linking the SSM's decay spectrum to detection delay and minimum detectable anomaly duration, validated empirically.
Real-World Edge Evaluation: End-to-end latency and throughput measurements performed directly on consumer edge silicon (Apple M3 Pro), rather than simulated from GPU metrics.
Transparent Reporting: Full reporting of ablation results, including findings that complicate the initial design hypothesis (specifically regarding the utility of the gating mechanism across different dataset sizes).

Experimental Results

Datasets: Evaluated on UCSD Ped2 and CUHK Avenue. (ShanghaiTech was intended but excluded due to data availability).
Accuracy: With an initial, untuned configuration, the method achieves 67.9% AUC on Ped2 and 70.2% AUC on Avenue. The author notes these trail prior non-causal SSM baselines (often in the 90s), attributing the gap to the lack of hyperparameter tuning, motion signals, and backbone optimization rather than the streaming formulation itself.
Latency & Throughput: Measured directly on an Apple M3 Pro, the system achieves 0.74 ms/frame (1,343 FPS) on Ped2 and 0.77 ms/frame (1,290 FPS) on Avenue.
Theory Validation: The theoretical settling delay bound (derived from base decay alone) predicts 57–59 frames. However, the empirical detection delay is significantly lower (1.6 frames on Ped2, 18.4 on Avenue). This confirms that the event-boundary gate, not the base decay, governs the reaction speed.
Ablation Insights (Data-Size Dependence):
- Ped2 (Small dataset): Disabling the gate improved accuracy (76.6% vs 67.9%), and smaller state sizes performed better. This suggests the gate's parameters overfit on small datasets.
- Avenue (Larger dataset): Disabling the gate hurt accuracy sharply (60.0% vs 70.2%), and larger state sizes helped.
- Conclusion: The utility of the gating mechanism is data-size dependent, not uniformly helpful.

Significance and Claims
The paper claims to close the gap between theoretical efficiency, strict causality, and real-world edge deployment. Its significance lies in:

Providing a design tool (the decay-delay theory) rather than just an empirical curiosity.
Demonstrating that strictly causal, $O(1)$ streaming is feasible on edge hardware with real-time performance.
Offering a modest, honest evaluation that reports accuracy gaps and dataset-dependent behaviors (the gate's overfitting on small data) rather than smoothing over results to fit a narrative.

Limitations Acknowledged by the Authors

Accuracy Gap: Current results trail tuned, non-causal baselines; closing this requires backbone and hyperparameter tuning.
Hardware Specificity: Latency is measured only on Apple M3 Pro (MPS backend); CoreML/Neural Engine export and non-Apple edge benchmarks are pending.
Theoretical Bound: The settling-delay theorem assumes fixed decay; the deployed model uses time-varying decay via the gate, making the theoretical bound a loose upper limit.
Evaluation Metrics: Region- and track-based criteria (RBDC/TBDC) are currently approximated; official implementation is needed for future localization metrics.
Reproducibility of Baselines: A same-protocol comparison with VADMamba is impossible without substantial reimplementation, as the baseline relies on CUDA-only optical flow kernels.

Technical Summary: Strictly Causal Streaming Video Anomaly Detection with a Theoretically-Grounded State-Space Core

More like this