Frame-Level Evaluation in Weakly Supervised Video Anomaly Detection Mostly Measures Video-Level Ranking

In the world of computer vision, there is a constant race to teach machines to watch video feeds and spot trouble. Imagine a security camera scanning a busy street, a train station, or a factory floor. The goal is for the software to notice when something goes wrong—a fight breaking out, a person falling, or a vehicle driving the wrong way—and to pinpoint exactly when and where it happens. For years, researchers have trained these systems using a method called weak supervision. Instead of showing the computer every single second of a video and labeling the exact moment an event occurs, they simply tell the computer whether the entire video contains an anomaly or not. It is a practical shortcut, much like a teacher telling a student "this essay has a mistake" without pointing to the specific sentence. To see if these systems are working, scientists use a standard test that ranks every single frame of every video in a database, from the most suspicious to the most normal, and checks how well the system separates the bad moments from the good ones.

Two researchers from SungKyunKwan University in South Korea, Inpyo Song and Jangwon Lee, decided to look closer at this standard test. They asked a simple but profound question: when a computer scores highly on this ranking test, does it actually know when the trouble happened, or is it just good at guessing which videos contain trouble? Their investigation, published in a recent study, reveals a startling truth about how these systems are evaluated. They found that the standard test is so heavily weighted toward comparing different videos against each other that a computer can achieve a near-perfect score without ever identifying a single specific moment of danger within a video. It is as if a student could ace a history exam by correctly guessing which textbooks contain errors, without ever reading a single page to find the typo.

The researchers began by breaking down the math behind the standard ranking score. They realized that this score is built from two different types of comparisons. The first type compares a bad moment in one video against a good moment in the same video; this is the test of true localization, checking if the system knows the difference between a safe second and a dangerous second within the same recording. The second type compares a bad moment in one video against a good moment in a completely different video. This is the test of video-level sorting, checking if the system can tell that Video A is more dangerous than Video B. When they counted how many of these comparisons actually happened within the same video, the number was shockingly small. Across three major datasets used by the field, less than 0.4 percent of the comparisons were between frames of the same video. The vast majority of the score comes from comparing frames across different videos.

This imbalance creates what the authors call "temporal dilution." As the number of videos in a test set grows, the chance that the system is being tested on its ability to find the exact moment of an event shrinks rapidly. The system is mostly being rewarded for assigning a high overall score to a video that contains an anomaly and a low score to a video that does not. To prove that this was the case, the researchers ran a series of controlled experiments. They trained simple computer models to do nothing more than assign a single, constant score to every frame of a video based on whether that video was labeled as dangerous or safe. These models had no ability to distinguish one moment from another; the score for the first second of a video was identical to the score for the last second. Despite having zero ability to localize an event in time, these simple models still achieved very high scores on the standard test, ranging from 81 percent to 97 percent depending on the dataset.

The researchers then took the outputs of the most advanced, complex systems currently available and performed a "video-mean replacement." They took the detailed, second-by-second scores these complex systems produced and replaced every single second's score with the average score for that entire video. This process erased all the fine-grained timing information, leaving only the overall video rating. If the standard test truly measured the ability to find the exact moment of an anomaly, this change should have caused the scores to crash. Instead, the scores barely moved. In 69 out of 72 controlled runs, the system retained more than 90 percent of its original score even after all the timing details were stripped away. The same pattern held true for the official results released by the authors of other leading methods. Even when the researchers shuffled the order of the frames within a video, scrambling the timeline while keeping the same set of scores, the ranking performance remained largely intact.

The findings suggest that the current standard for evaluating these systems is not measuring what researchers think it is measuring. A high score on the pooled ranking test does not prove that a system can locate an event; it primarily proves that the system can rank videos by their overall risk. The researchers argue that this distinction matters deeply for real-world applications. If a security guard receives an alert that a video contains an anomaly, they need to know when to look, not just which video to watch. The current evaluation protocol rewards the ability to pick the right video but fails to verify the ability to find the right moment. The authors propose that the field must change how it reports results. They recommend that researchers always report a separate score that measures how well the system orders frames within a single video, alongside the standard video-level ranking. They also suggest that the design of future tests should ensure that the number of videos does not dilute the importance of finding the exact moment of an event.

This work does not claim that the current systems are useless or that they cannot learn to localize events. It simply shows that the scoreboard they are playing on does not force them to do so. The high scores seen in recent years are largely a reflection of the system's ability to sort videos, a capability that is useful but distinct from the ability to pinpoint time. By separating these two skills in the evaluation process, the field can ensure that future systems are not just good at guessing which videos are dangerous, but are truly capable of watching the clock and finding the exact second when something goes wrong. The path forward requires a shift in focus, moving away from a single, aggregated number that masks these differences, toward a clearer picture that values the precision of time as much as the identification of the event itself.