💻 computer science

VTO: Visual Tool Orchestration for Video Anomaly Detection

The paper proposes VTO, a process-supervised reinforcement learning framework that leverages a foundation model-driven cognitive evaluator and fine-grained step-wise supervision to enable multimodal agents to dynamically orchestrate specialized visual tools for improved video anomaly detection, validated by a new benchmark and toolset called VAD-Tool.

Original authors: Rui Wang, Yeteng Wu, Xianling Zhang, Mengshi Qi

Published 2026-08-11

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Rui Wang, Yeteng Wu, Xianling Zhang, Mengshi Qi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery in a busy, chaotic city. You have a notebook and a pen, but the city is full of moving cars, falling objects, and people fighting. To solve the case, you can't just look at the whole scene and guess; you need to use specific tools. Maybe you need a magnifying glass to track a specific person, a speedometer to check if a car is speeding, or a fire detector to spot smoke. In the world of computers, this is called Video Anomaly Detection. It's the job of teaching machines to spot weird or dangerous things in video footage, like a fight breaking out or someone falling down.

For a long time, computers tried to do this by memorizing patterns, like a student cramming for a test. They would look at a video and say, "This looks like a fight!" But if the fight happened in a new place or looked slightly different, the computer would get confused and fail. Recently, scientists tried giving computers a "brain" (a Large Language Model) that could talk and think. They hoped the computer could figure out which tool to use and when. But here's the problem: these smart computers often stop thinking and give their answer immediately after seeing one small problem. They miss the bigger picture, like how a small fight could turn into a stampede. They need a way to learn how to keep investigating, step-by-step, without giving up too soon.

This is where the new paper comes in. The researchers, led by Rui Wang and Mengshi Qi from Beijing University of Posts and Telecommunications, built a new system called VTO (Visual Tool Orchestration). Think of VTO as a strict but brilliant coach for a detective robot. Instead of just letting the robot guess, the coach watches every single step the robot takes. If the robot picks the wrong tool, or stops investigating before it's sure, the coach gives it a "thumbs down" right then and there. If the robot follows a perfect logical chain, checking one clue after another, the coach gives it a "thumbs up."

The team created a special training ground called VAD-Tool, which is like a giant playground with 12 different "magic tools" for the robot to learn. These tools can do things like count crowds, recognize faces, spot weapons, or detect fire. The researchers taught their robot using a method called "process-supervised reinforcement learning." In simple terms, this means they didn't just grade the robot on the final answer; they graded the process. They made sure the robot learned that stopping too early is a mistake. They even used a super-smart AI (a 72-billion-parameter model) to act as a judge, checking if the robot's reasoning made sense at every step.

The results were impressive. When they tested their new VTO system, it got much better at solving these complex, multi-step mysteries than the old methods. While other systems often gave up after finding one clue, VTO kept going, connecting the dots until it found the whole story. In their tests, the new system improved the accuracy of picking the right tools and the right order by up to 10.2% compared to the previous best methods. It even beat some of the biggest, most powerful AI models out there, proving that teaching a computer how to think step-by-step is more important than just making it bigger. The paper suggests that by forcing the AI to follow a strict, logical path and rewarding it for not giving up, we can build safer, smarter systems to watch over our cities.

Technical Summary: VTO – Visual Tool Orchestration for Video Anomaly Detection

1. Problem Definition

Video Anomaly Detection (VAD) is a critical task for applications ranging from public safety to urban surveillance. However, real-world anomalies are rarely isolated; they are complex, diverse, and dependent on specific physical contexts (e.g., a local altercation escalating into a weaponized assault). Traditional deep learning approaches treat VAD as a closed-set classification or scoring task, suffering from poor generalization in open-world scenarios.

Recent attempts to leverage Large Language Models (LLMs) and Vision-Language Models (VLMs) for explainable VAD have introduced multimodal agents. However, these systems face two primary limitations:

Static Orchestration: Systems relying on Supervised Fine-Tuning (SFT) or heuristic behavior cloning treat tools as isolated capsules. They struggle to dynamically orchestrate distinct visual expert models to decipher intricate causal dependencies.
Premature Termination: Standard Reinforcement Learning (RL) approaches, which rely on coarse-grained, outcome-based rewards, often cause agents to halt execution immediately after detecting a primary anomaly. Without fine-grained supervision, agents fail to complete the necessary multi-step causal chains required to identify cascading secondary hazards.

2. Methodology: The VTO Framework

The authors propose VTO (Visual Tool Orchestration), a process-supervised reinforcement learning framework designed to transform VAD into a complex, multi-step cognitive tool orchestration problem.

2.1 VAD-Tool Benchmark

To support the framework, the authors constructed VAD-Tool, a hierarchical visual toolset and benchmark comprising:

12 Specialized Vision Tools: Organized into three functional categories:
- Entity Identity & Tracking: Person Re-ID, Gait Recognition, License Plate Recognition, Car Re-ID.
- Behavior & Scene Understanding: Pose Estimation, Human Fall Detection, Crowd Counting, Scene Recognition.
- Security Event & Hazard Detection: Violence Detection, Weapon Detection, Fire/Smoke Detection, and a general Anomaly Detection expert.
Dataset: A large-scale training dataset (~15.48M tokens) and a rigorous testing set (1,841 samples) featuring interrelated anomaly queries (e.g., Pose-based Violence Detection) that require multi-step reasoning. The data was curated via a human-in-the-loop pipeline using strong VLMs (Qwen2.5-VL) for initial trajectory generation and expert review for logical correctness.

2.2 Training Pipeline

The VTO framework operates in two distinct stages:

Stage 1: Supervised Fine-Tuning (SFT)
The base model is fine-tuned on the VAD-Tool dataset to establish foundational capabilities:

Syntactic Tool Invocation: Adhering to valid ReAct formatting (Thought, Action, Action Input, Observation).
Anomaly Event Modeling: Adapting internal representations for high-stakes security events.
Observation Integration: Learning to contextualize specific outputs from visual experts (e.g., tracking IDs, spatial coordinates).

Stage 2: Process-Supervised Cognitive Alignment
To overcome the limitations of static behavior cloning and outcome-based RL, the authors introduce a dynamic learning loop optimized via Group Relative Policy Optimization (GRPO). This stage employs a dual-reward system to provide dense, step-wise supervision:

Rule-Based Objective Rewards ( $R_{rule}$ ):
- Task Completion ( $R_{comp}$ ): Positive scores for successful completion, negative penalties for errors.
- Tool/Parameter Matching ( $R_{tool}, R_{param}$ ): Exact match enforcement for tool names and parameters.
- Efficiency Penalty ( $R_{eff}$ ): Discourages redundant tool invocations.
LLM-as-a-Judge Rewards ( $R_{LLM}$ ):
- Utilizing a 72B-parameter model (Qwen2.5-VL) as an expert evaluator to score trajectories on:
  - Logicality ( $R_{logic}$ ): Step-by-step causal reasoning.
  - Relevance ( $R_{rel}$ ): Alignment of actions with the user query.
  - Completeness ( $R_{complete}$ ): Ensuring all necessary logical steps are covered, explicitly penalizing premature termination.

The agent optimizes its policy by maximizing the group-relative advantage, effectively learning to construct goal-driven causal chains without premature termination.

3. Key Contributions

VTO Framework: A dynamic multimodal agent framework powered by process-supervised reinforcement learning. It explicitly addresses the premature termination problem in VAD by shifting from outcome-based rewards to fine-grained, step-wise cognitive alignment.
VAD-Tool Benchmark: A comprehensive evaluation platform integrating 12 specialized vision models. It is specifically designed to test single-step and multi-step interrelated anomaly reasoning, filling a gap in existing datasets that focus primarily on outcome-driven metrics.
Performance Breakthrough: The framework demonstrates that dynamic policy optimization can significantly outperform static SFT baselines and even surpass the performance of massive 72B-parameter models in complex tool orchestration tasks.

4. Experimental Results

Extensive experiments were conducted on the VAD-Tool benchmark using various base models (Vicuna-7B, Llama-3-8B, Qwen3-VL-8B) and a strong 72B baseline (Qwen2.5-VL-72B).

Baseline Limitations: Zero-shot and few-shot baselines, as well as SFT-only models, showed catastrophic degradation on "Interrelated Tools" tasks. For instance, Vicuna-7B dropped to 0% accuracy on interrelated whole-response tasks, and Qwen2.5-VL-72B plateaued at ~67.40%.
VTO Performance: VTO significantly outperformed all baselines.
- On the Qwen3-VL-8B backbone, VTO achieved 95.14% accuracy on the "Whole" metric for interrelated tools, compared to 89.74% for the SFT baseline.
- On the Llama-3-8B backbone, VTO improved interrelated performance from 65.93% (SFT baseline) to 72.16%.
- VTO achieved perfect scores (100%) in Decision (Decis) and Tool selection (Tool) metrics for the Qwen3-VL-8B variant.
Ablation Studies: Removing the process-supervised components (specifically the task completion reward or the LLM judge reward) led to significant performance drops, confirming the necessity of both deterministic rule-based anchors and qualitative logical supervision to prevent premature termination and hallucination.

5. Significance and Claims

The paper claims that VTO redefines Video Anomaly Detection as a complex, multi-step cognitive tool orchestration problem rather than a static classification task. By introducing Process-Supervised Cognitive Alignment, the authors demonstrate that agents can be trained to dynamically explore environments, construct observable multi-step causal chains, and rigorously adhere to safety constraints without terminating prematurely.

The work bridges the gap between static imitation learning and dynamic reinforcement learning in physical security domains. It provides a rigorous infrastructure (VAD-Tool) and a scalable framework (VTO) that enables agents to handle unpredictable, cascading hazards, moving beyond the limitations of current training-dependent and training-free paradigms. The authors emphasize that their approach allows smaller models (8B parameters) to achieve performance levels that rival or exceed much larger models (72B parameters) in specific, complex reasoning tasks.