A VLM Answer Is Not an Anomaly Score: Rank Compression in Training-Free Video Anomaly Detection

In the quiet hum of a surveillance system, a camera watches a street corner, a factory floor, or a subway station, waiting for something to go wrong. For decades, computers have struggled to spot these moments of trouble, known as anomalies, because they are rare, unpredictable, and often look like nothing at all until they happen. Traditional systems required engineers to teach them what a crime or an accident looks like by feeding them thousands of examples, a process that is slow and rigid. Recently, a new kind of artificial intelligence called a vision-language model has offered a different path. These are powerful systems that can see an image and understand it through the lens of human language. Instead of being trained on specific examples of violence or theft, they can be asked a simple question: "Is something dangerous happening here?" If the model says "Yes," the system flags the moment. It sounds like a straightforward solution, but a new study reveals that the way we listen to the answer matters just as much as the answer itself.

Researchers at SungKyunKwan University in South Korea investigated how these models translate their thoughts into a score that a computer can use to rank events. They found that the standard way of using these models was throwing away a vast amount of useful information. When a model is asked to judge a video clip, it does not just pick a single word like "Yes" or "No" from a list. Instead, it calculates a probability for every possible answer it could give. Think of it as a scale of certainty. The model might be 94% sure a clip is dangerous and 56% sure another clip is dangerous. Under the old method, both clips would simply receive the label "Yes," and the computer would treat them as identical. The researchers discovered that this loss of detail, which they call rank compression, causes the system to miss the subtle differences that separate a near-miss from a clear danger.

To fix this, the team tested a different approach. Instead of waiting for the model to pick a single word, they looked at the full spread of probabilities the model generated for every possible answer. They turned this entire distribution of certainty into a single, precise number. This method, which they call a probability readout, allowed the system to see that the 94% clip was significantly more dangerous than the 56% clip, even though both were labeled "Yes." When they tested this method across four different large models and two major benchmarks for video anomalies, the results were striking. The new approach outperformed the old one in every single test. On one dataset, the improvement in accuracy was as high as 13 percentage points, and on another, it reached nearly 19 points. These gains were not small fluctuations; they were consistent and significant, appearing regardless of the specific model used or the type of question asked.

The researchers also explored why the old method failed so often. They found that even when they gave the models a very fine scale to choose from, allowing them to pick from 91 different numbers instead of just two, the models still collapsed their answers into a handful of repeated values. The system would still end up treating many different clips as identical, creating a "tie" in the ranking. The new method avoided this trap entirely. By using the full probability distribution, the system could distinguish between almost every single moment in the video, creating a smooth, detailed ranking rather than a jagged list of ties. This difference was crucial because in security, the ability to rank the most dangerous moments at the very top is what matters most. The study showed that the new method could double the number of actual dangers found at a low rate of false alarms, effectively making the system twice as useful without requiring any new training or extra computing power.

The team also checked to ensure this advantage was real and not just an artifact of the question wording or the way the model was asked to explain itself. They tried asking the models to describe the scene before giving a score, or to explain their reasoning after. In every case, the method that looked at the full probability distribution remained superior. They even tested if larger, more powerful models would close the gap, but found that the advantage persisted and sometimes grew. The only time the advantage shrank was when the model was forced to write an explanation before giving its score, which seemed to compress its internal certainty. This suggests that the key to unlocking the full potential of these AI systems lies in how we read their output.

This work does not just offer a small tweak; it redefines a critical step in how we use artificial intelligence for safety. The researchers demonstrated that the interface between a model's internal thought process and the final score is not a minor detail but a central part of the system's performance. By simply changing how the answer is read—looking at the whole picture of certainty rather than just the final word—engineers can turn the same frozen model into a much sharper detector. The study concludes that for anyone building systems to watch for trouble, the choice of how to interpret the model's answer is as important as the model itself. It is a reminder that in the world of artificial intelligence, the most powerful signal is often hidden in the nuance of what is almost said, not just in the final word spoken.