💻 computer science

Interactive Query based Abnormal Events Synopsis Generation in Surveillance Video

This paper proposes an interactive query-based algorithm for generating abnormal event synopses in surveillance videos that utilizes a rule-based classifier to handle complex user queries and introduces an "improved overlapping ratio" metric for evaluation, demonstrating superior accuracy and quality over existing methods on the PETS09 dataset.

Original authors: Judi Vennila Thangaswamy, Balamurugan Vaniappan

Published 2026-07-28

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Judi Vennila Thangaswamy, Balamurugan Vaniappan

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific moment in a movie, but the movie is actually a 24-hour security camera feed that never stops rolling. Watching every single second of that footage to find the one time someone dropped their keys or walked the wrong way would be like trying to drink from a firehose. This is the world of surveillance video analytics. In this field, computers act as the eyes, constantly watching for "abnormal events"—things that don't fit the normal pattern, like a person loitering too long, a sudden collision, or someone sneaking into a restricted zone.

To make sense of this flood of data, scientists use video summarization. Think of this as a "highlight reel" or a "movie trailer" for security footage. Instead of showing you the boring parts where nothing happens, the computer picks out only the interesting frames and stitches them together into a short, fast-paced summary. However, there's a catch: sometimes the computer gets confused. It might show you two people walking past each other at the exact same time, making them look like a single, giant blob. This "overlapping" makes the summary messy and hard to read. The big question researchers are trying to solve is: How do we create a summary that listens to exactly what the user wants to see, keeps the story in the right order, and doesn't get messy when people bump into each other?

The Detective's Query: A New Way to Summarize Security Footage

In this paper, researchers Judi Vennila Thangaswamy and Balamurugan Vaniappan propose a new method called Interactive Query-based Abnormal Events Synopsis Generation (IQAES). Imagine you are a detective investigating a crime. Instead of watching hours of grainy footage, you can ask the computer specific questions like, "Show me everyone who entered the lobby from the East between 2 PM and 3 PM," or "Find the moment two people walked in together."

The paper suggests that existing methods are a bit like a rigid librarian who only hands you books based on a fixed list, ignoring your specific questions. The new IQAES algorithm, however, acts like a super-smart assistant who understands complex requests. It can handle simple queries (like "Who entered?") and complex queries (like "Who entered from the North while someone else exited to the South?").

How the Magic Works

The system works in a few clever steps, using a set of rules to track people:

The Invisible Grid: First, the computer draws an invisible box (called a Region of Interest, or ROI) on the screen where it cares about. It tracks the coordinates of every person's body (top, bottom, left, right) as they move.
The "0" and "1" Code: The system creates a list for every person. If a person is inside the box, they get a "1". If they are outside, they get a "0".
The Query Engine: When you ask a question, the system scans this list.
- Entry/Exit: It looks for the moment a person's code flips from "0" to "1" (entering) or "1" to "0" (exiting).
- Loitering: If a person stays inside the box for a long time, the system flags them as "loitering."
- Direction: By comparing where a person was in the previous frame versus the current frame, the system knows if they are moving North, South, East, or West.
- Simultaneity: It can even catch when two people do things at the same time, like entering together, by checking if their "entry" moments happen within a tiny window of each other.

The "Messy Blob" Problem and the New Solution

One of the biggest headaches in video summaries is overlapping. Imagine two people walking side-by-side; to a computer, they might look like one giant, wide person. Old methods tried to measure the quality of a summary by counting how many "pixels" were overlapping. The authors argue this is like judging a crowded party by measuring the total area of the floor covered by people's feet—it doesn't tell you how many people are actually squished together.

To fix this, the paper introduces a new measuring stick called the Improved Overlapping Ratio (IOR). Instead of counting pixels, IOR counts the actual number of people who are overlapping. It asks, "In this summary, how many distinct humans are getting squished together?" This gives a much clearer picture of how "clean" the summary is.

What Did They Find?

The researchers tested their new system on a famous dataset called PETS09, which contains 794 frames of pedestrian video. They set up a specific "Region of Interest" with coordinates (160, 260) to (250, 280) and asked the system to find various events.

The results were promising. When compared to an older method (a "Visualization Framework"), the new IQAES system performed better in almost every category:

Accuracy: For simple entry/exit questions, the new system was 99% accurate.
Precision: It correctly identified 97% of the relevant events for simple queries, compared to 86% for the old method.
Recall: It found 100% of the actual events in some complex directional tests, whereas the old method only found 86%.
Cleanliness: The new system produced summaries with much less "mess." The Improved Overlapping Ratio (IOR) was significantly lower (e.g., 7.00% for complex directional queries vs. 43.00% for the old method), meaning fewer people were squished together in the final video.

They also asked five human volunteers to rate the summaries. The participants gave high marks for how pleasant the videos were to watch and how well they preserved the important events.

The Bottom Line

The paper concludes that this interactive approach is a strong step forward. It suggests that by letting users ask complex questions and by using the new IOR metric to check for "squished" people, we can create surveillance summaries that are faster to watch, more accurate, and easier to understand. While the system isn't perfect (it still had a few false alarms when people were heavily overlapping), the authors show that it outperforms the current standard, offering a clearer window into the chaotic world of surveillance video.

Technical Summary: Interactive Query based Abnormal Events Synopsis Generation in Surveillance Video

Problem Definition

Surveillance systems generate voluminous video data, the majority of which is redundant and uninformative. While video summarization (or synopsis generation) aims to condense this data into a shorter version containing unique frames, existing methods face significant challenges in surveillance contexts. Specifically, current techniques struggle with:

Processing Complex Queries: Handling user queries that involve multiple conditions, such as simultaneous interactions between multiple objects or specific directional movements.
Preserving Chronological Order: Maintaining the temporal sequence of events, which is critical for investigative analysis.
Evaluating Synopsis Quality: Existing metrics, such as the standard Overlapping Ratio (OR), often rely on pixel counts of overlapping regions. This approach is flawed because pixel counts vary based on the location of overlap and do not accurately reflect the number of distinct humans involved in the overlap, leading to misleading quality assessments.

The paper addresses the need for a system that can generate content-based abnormal event synopses by processing interactive queries (including simple entry/exit and complex multi-object directional queries) while preserving temporal order and providing a more accurate quality metric.

Methodology: The IQAES Algorithm

The authors propose the Interactive Query based Abnormal Events Summarization (IQAES) algorithm. The methodology operates on pre-processed surveillance video where human detection and tracking have already been performed to generate a positional coordinate vector ( $P$ ) and an abnormal event vector ( $A$ ).

1. Input Pre-processing

The system utilizes a Rough Set Framework for Multi-Human Tracking (RSFMHT) to derive a positional coordinate vector ( $P$ ) for humans in each frame. This vector contains four coordinates ( $X_{min}, X_{max}, Y_{min}, Y_{max}$ ) defining the bounding box of each human. An Abnormal Event Vector ( $A$ ) is then generated by applying Algorithm 1, which marks frames as '1' if a human is present within a defined Region of Interest (ROI) and '0' otherwise.

2. Query Processing

The IQAES algorithm categorizes user queries into two types and processes them using specific sub-algorithms:

Simple Queries: These involve single parameters, specifically the entry or exit of a single human.
- IHE (Interested Human's Entry): Detects transitions from '0' to '1' in the event vector to identify entry frames.
- IHX (Interested Human's Exit): Detects transitions from '1' to '0' to identify exit frames.
Complex Queries: These involve multiple conditions and objects.
- SLH (Simultaneous Loitering): Calculates the common frame intervals where multiple humans are simultaneously present within the ROI.
- SEH/SXH (Simultaneous Entry/Exit): Identifies entry or exit events occurring within a specific frame interval ( $x$ ) to account for detection delays, allowing for the detection of simultaneous movements of multiple humans.
- DEH/DXH (Directional Entry/Exit): Determines the moving direction (North, South, East, West) of humans by comparing their positional midpoints against the ROI corners. This supports queries regarding the specific direction of movement during entry or exit.

3. Synopsis Generation

The algorithm retrieves relevant frames from the input video based on the detected events and query conditions. It ensures the chronological order of frames is preserved in the final synopsis ( $S$ ), where $S \ll N$ (the synopsis is significantly shorter than the original video).

Key Contributions

1. Interactive Query-Based Framework

The paper introduces a rule-based classifier capable of handling complex, multi-condition queries that existing methods often fail to address. Unlike previous works that focus on scene-based summaries or simple trajectory analysis, IQAES specifically targets content-based summaries driven by user-defined constraints regarding object interaction (simultaneity) and movement direction.

2. Improved Overlapping Ratio (IOR)

A novel metric, the Improved Overlapping Ratio (IOR), is proposed to evaluate the quality of the generated synopsis.

Limitation of Existing Metrics: Standard Overlapping Ratio (OR) measures the ratio of overlapped pixels to the total frame area. The authors argue this is inaccurate because pixel counts depend on the specific location of the overlap and do not scale linearly with the number of humans involved.
IOR Definition: IOR measures the ratio between the number of overlapped humans and the total number of humans in the ROI across all synopsis frames.
Significance: This metric provides a more accurate representation of visual clutter and information loss caused by occlusion, independent of pixel density variations.

3. Comprehensive Sub-Algorithm Suite

The paper details a suite of eight specific algorithms (Algorithm 1 through 8) that systematically handle the logic for event detection, simultaneous occurrence analysis, and directional classification, providing a structured approach to interactive video summarization.

Experimental Results

The IQAES algorithm was evaluated using the PETS09 dataset, consisting of 794 pedestrian keyframes. The experiments covered various scenarios including single/multiple human entry/exit, simultaneous loitering, and directional movements.

Quantitative Performance:
- The algorithm achieved high accuracy (99%–100%) and precision (87%–100%) across different query types.
- Recall was particularly high for directional queries (100% for DEH and DXH).
- Frame Condensation Ratio (FCR) and Temporal Disorder Ratio (TDR) were measured to ensure compactness and chronological integrity.
Quality Evaluation:
- The proposed IOR yielded significantly lower values (indicating better quality) compared to the standard OR. For example, in complex directional queries, the IOR was 7.00% compared to 43.00% for the visualization framework [10].
- Subjective Evaluation: Five participants rated the synopsis on criteria such as pleasantness, clarity of posture, compactness, and content preservation. The scores indicated high satisfaction with the generated summaries.
Comparative Analysis:
- When compared to an existing visualization framework [10], the proposed IQAES outperformed the baseline in both Precision and Recall for both simple and complex queries.
- IQAES demonstrated lower Overlapping Ratio (OR) and Improved Overlapping Ratio (IOR) values, confirming superior handling of occlusion and visual clarity.

Significance and Claims

The paper claims that the proposed IQAES algorithm effectively addresses the gaps in current surveillance video summarization, specifically the inability to process complex multi-object queries and the lack of accurate quality metrics for overlapping events.

Handling Complexity: The system successfully prioritizes frames with abnormal events based on complex user queries involving direction and simultaneity, which previous methods struggled to maintain without losing temporal order.
Metric Improvement: The introduction of IOR is presented as a significant advancement over pixel-based metrics, offering a more reliable measure of synopsis quality that aligns with human perception of overlapping objects.
Investigative Utility: By generating a concise, chronologically ordered synopsis that adheres to specific user constraints, the system enables investigators to rapidly analyze voluminous surveillance footage, focusing only on relevant abnormal events.

The authors conclude that while the system performs robustly, minor false positives and negatives can occur due to severe occlusion in input frames. However, the overall results demonstrate that IQAES generates higher-quality synopses than existing visualization-based methods.