💻 computer science

Text-guided Fine-Grained Video Anomaly Understanding

The paper proposes T-VAU, a text-guided framework that leverages an Anomaly Heatmap Decoder and a Region-aware Anomaly Encoder to enable large vision-language models to perform unified fine-grained video anomaly detection, precise pixel-level localization, and interpretable semantic reasoning.

Original authors: Jihao Gu, Kun Li, He Wang, Kaan Akşit

Published 2026-04-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jihao Gu, Kun Li, He Wang, Kaan Akşit

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard watching a live feed of a busy city square. Your job is to spot anything weird happening.

The Problem with Current Guards
Most current computer systems acting as security guards are like overworked interns who only shout "ALARM!" or "ALL CLEAR!"

If they see something slightly off, they might just say, "Something is wrong here," without telling you what it is, where it is, or why it's weird.
They are great at spotting big, obvious things (like a car crashing), but they miss the subtle stuff (like a person running the wrong way or a bag being dropped).
On the other hand, newer "smart" AI systems (Large Vision-Language Models) are like chatty detectives. They can tell you a story: "A man in a red hat is acting suspiciously." But they are terrible at pointing their finger. They might say "Look over there!" while pointing at a tree, because they can't actually see the tiny details in the pixels.

The Solution: T-VAU (The "Super-Guard")
The paper introduces a new system called T-VAU (Text-guided Fine-Grained Video Anomaly Understanding). Think of T-VAU as a Super-Guard that combines the sharp eyes of a sniper with the storytelling skills of a novelist.

It does this using two special tools:

1. The "X-Ray Glasses" (Anomaly Heatmap Decoder)

Imagine the security camera feed is a painting. The old systems just look at the whole painting and guess if it's "bad."
T-VAU puts on X-Ray glasses that highlight exactly where the weirdness is happening in the picture.

It scans the video and draws a glowing, red "heat map" over the specific pixels where something is wrong.
If a person is running the wrong way, the glasses glow bright red around that specific person, ignoring the crowd around them.
This solves the problem of the "chatty detective" pointing at the wrong thing. Now, the system knows exactly where to look.

2. The "Detective's Notebook" (Region-aware Anomaly Encoder)

Once the X-Ray glasses find the weird spot, T-VAU doesn't just leave it there. It takes that glowing red spot and turns it into a structured note for the AI detective.

Instead of just saying "Look at the red spot," it translates the visual data into a prompt like: "Focus on the person in the red shirt moving fast from left to right."
This note is fed into the AI's brain, forcing it to pay attention to the specific evidence the X-Ray glasses found.
This allows the AI to answer complex questions like: "Who is doing it?" "What do they look like?" and "How did they move?"

The Training Ground (The Dataset)

To teach this Super-Guard, the researchers didn't just show it videos; they built a special training school.

They took existing video datasets and added detailed annotations.
Instead of just labeling a video "Abnormal," they labeled it: "A cyclist with a backpack entered from the right, turned sharply, and exited."
This taught the AI to connect the visual dots (the heat map) with the story (the text), so it learns to be both accurate and descriptive.

Why This Matters

In the real world, knowing that an accident happened isn't enough. You need to know:

Where did it happen? (The X-Ray glasses)
Who was involved? (The Detective's Notebook)
What exactly did they do? (The Storytelling)

The Result:
T-VAU is like a security system that doesn't just scream "Fire!" but instead says: "There is a fire in the kitchen, specifically near the stove. It started 10 seconds ago, and the smoke is drifting toward the hallway."

It bridges the gap between seeing (pixels) and understanding (words), making AI much more reliable for safety, security, and industrial inspection.

1. Problem Statement

Video Anomaly Detection (VAD) faces a critical limitation in real-world scenarios: subtle anomalies (e.g., slight motion deviations, brief interactions, or fine-grained appearance changes) are often obscured by background clutter, occlusions, and compression artifacts.

Limitations of Traditional VAD: Conventional methods output coarse binary scores or frame-level heatmaps. They lack semantic interpretability, cannot explain why an event is anomalous, and fail to localize subtle cues precisely.
Limitations of Large Vision-Language Models (LVLMs): While LVLMs can generate textual explanations, they struggle to ground their reasoning in pixel-level evidence. When applied directly to VAD, they often produce hallucinated descriptions or fail to localize subtle visual signals accurately.
The Gap: There is a disconnect between low-level visual evidence (pixels) and high-level semantic reasoning (language). Existing approaches lack a unified framework that provides both precise localization and faithful, evidence-grounded textual explanations.

2. Methodology: T-VAU Framework

The authors propose T-VAU, a framework built upon a frozen Large Vision-Language Model (LVLM) that bridges the gap between pixel-level evidence and language-based reasoning. The system consists of three main components:

A. Anomaly Heatmap Decoder (AHD)

The AHD is responsible for extracting precise, threshold-free spatio-temporal anomaly heatmaps.

Mechanism: It takes multiscale visual features from the LVLM's vision encoder and aligns them with binary text prompts (representing "normal" vs. "abnormal") generated by a text encoder.
Process:
1. Computes cosine similarity between intermediate visual tokens and sentence-level text embeddings.
2. Fuses similarity maps across multiple transformer layers using learnable weights.
3. Outputs a pixel-level anomaly heatmap ( $H$ ) that highlights subtle deviations without relying on manual thresholds.
Goal: To provide high-spatial-precision visual evidence of where and when an anomaly occurs.

B. Region-aware Anomaly Encoder (RAE)

The RAE acts as a bridge, converting the visual heatmaps into structured prompts that the LVLM can understand and reason with.

Mechanism: It processes the anomaly heatmaps to extract both local and global features.
- Temporal Dynamics: Computes differences between consecutive frames to capture motion.
- Spatial Partitioning: Divides frames into grids (e.g., 3x3) to extract region-specific features.
- Prompt Generation: Generates three types of embeddings: a learnable base prompt, region-specific prompts, and a global prompt.
Integration: These embeddings are concatenated with the visual features and dialogue history, then injected into the LVLM decoder. This allows the model to perform multi-turn reasoning (e.g., "Where is the anomaly?", "What does the target look like?", "How is it moving?") grounded in the visual evidence.

C. Fine-Grained Supervision Pipeline

To train the system effectively, the authors constructed a specialized dataset and pipeline:

Data Source: Derived from ShanghaiTech and UBnormal benchmarks.
Annotation Strategy:
1. Frame-level Extraction: Uses structured prompts to extract target attributes (appearance) and bounding boxes.
2. Temporal Aggregation: Links frames to create target-level timelines (tracking identity, location, and evolution).
3. Anomaly-Focused Refinement: Applies anomaly masks and Gaussian blurring to background regions to suppress noise and emphasize abnormal evidence.
4. Cross-Modal Verification: Ensures consistency between appearance descriptions and motion trajectories.
Output: A dataset with aligned annotations for appearance, spatial localization, and motion trajectories.

3. Key Contributions

T-VAU Framework: A novel architecture that couples an Anomaly Heatmap Decoder (AHD) for pixel-level localization with a Region-aware Anomaly Encoder (RAE) for injecting evidence into LVLM reasoning. It enables unified detection, localization, and multi-turn explanation.
Fine-Grained Dataset: Construction of a target-level video-text anomaly dataset based on ShanghaiTech and UBnormal. It provides explicit supervision for object appearance, localization, and motion trajectories, addressing the lack of fine-grained labels in existing benchmarks.
State-of-the-Art Performance: The framework achieves superior results in both anomaly localization metrics and textual reasoning quality, demonstrating that grounding language in pixel-level evidence significantly improves interpretability and accuracy.

4. Experimental Results

The model was evaluated on UBnormal (synthetic, open-set) and ShanghaiTech (real-world) datasets.

Anomaly Localization (UBnormal):
- Achieved 94.8% Micro-AUC and 87.8% Macro-AUC after fine-tuning.
- Significantly outperformed baselines in localization metrics: 67.8% RBDC (Region-Based Detection Criteria) and 76.7% TBDC (Track-Based Detection Criteria).
- Compared to the strongest baseline (Georgescu et al.), T-VAU improved Micro-AUC by +26.6 points and RBDC by +39.1 points.
Textual Reasoning & Dialogue (ShanghaiTech & UBnormal):
- BLEU-4 Scores: Achieved 62.67 for Target descriptions and 88.84 for Trajectory descriptions on ShanghaiTech, outperforming strong LVLM baselines (e.g., InternVL2-8B) by ~7-6 points.
- Decision Accuracy: Achieved 97.67% accuracy on Yes/No anomaly judgments.
- Qualitative Analysis: The model successfully generated self-consistent narratives (e.g., identifying a "silver SUV" accelerating from the left) with precise spatial grounding, reducing hallucinations common in standard LVLMs.
Ablation Studies:
- Removing AHD resulted in the loss of spatial interpretability (RBDC/TBDC became inapplicable).
- Removing RAE allowed for heatmap generation but failed to translate visual evidence into high-quality language reasoning (lower BLEU-4 and accuracy).
- This confirms the complementarity of both modules.

5. Significance and Impact

Bridging the Gap: T-VAU successfully closes the loop between low-level visual perception and high-level semantic reasoning, solving the "black box" problem of traditional VAD and the "hallucination" problem of direct LVLM application.
Interpretability: By providing pixel-level heatmaps alongside textual explanations, the system offers evidence-grounded reasoning, which is crucial for safety-critical applications like public security and industrial inspection.
Fine-Grained Understanding: The framework moves beyond simple binary classification to support complex queries about who is involved, what they look like, where they are, and how they move, aligning with the emerging field of Subtle Visual Computing (SVC).
Generalizability: The approach demonstrates that injecting structured, region-aware prompts derived from visual evidence can significantly enhance the reasoning capabilities of frozen LVLMs without requiring massive retraining.

In summary, T-VAU represents a significant step forward in making video anomaly detection not just accurate, but also explainable, localized, and interactive.