Technical Summary: SENSE-VAD

Problem Statement

Autonomous vehicles (AVs) have achieved significant progress in detecting kinematic hazards (e.g., lane departures, stationary obstacles) but remain structurally blind to a critical class of safety-critical scenarios: socially complex anomalies. These are situations where the danger is not encoded in motion statistics or appearance alone, but in the inter-agent relationships and social context. Examples include a child running away from a guardian, a person being carried by another, or a pursuer chasing a pedestrian. In these scenarios, individual agents may exhibit normal motion trajectories, yet the relational configuration constitutes an anomaly requiring immediate AV intervention.

Current Video Anomaly Detection (VAD) datasets for autonomous driving (e.g., DADA, DoTA, CADP) focus primarily on vehicle-centric accidents, kinematic outliers, or single-object anomalies. They lack the egocentric perspective, fine-grained social-relational annotations, and diversity of subtle behavioral cues necessary to address this "long tail" of safety-critical events. Furthermore, existing methods often rely on motion reconstruction or appearance-based signals, which fail when the motion itself is statistically plausible but socially aberrant.

Methodology

Dataset Construction: SENSE-VAD

The authors introduce SENSE-VAD, the first synthetic video anomaly detection benchmark for autonomous driving explicitly designed around socially complex anomalies.

• Generation Platform: The dataset is generated using the CARLA simulator across three versions (0.9.12, 0.9.15, and 0.10.0) running on Unreal Engine 4 and 5. Version selection is scenario-specific to accommodate different agent-control requirements (e.g., skeletal control for carrying scenes in 0.9.15, animal locomotion in 0.10.0).
• Anomaly Taxonomy: The dataset defines 15 anomaly classes across six conceptual categories:
  1. Child Safety: (e.g., Child Running from Guardian, Kidnapping, Kid Occlusion).
  2. Group Behavior: (e.g., Police Chase, Crowd Chaos).
  3. Individual Behavior: (e.g., Abnormal Person Behavior, Person Being Carried).
  4. Person–Object Interaction: (e.g., Bag Theft, Unattended Bag, Object Drop).
  5. Cyclist Interaction: (e.g., Cyclist Hazard, Cyclist–Bag Abandonment, Cyclist–Vehicle Collision).
  6. Vehicle & Agent: (e.g., Vehicle–Pedestrian Collision, Dog Leaving Owner).
• Design Principle: A core tenet is the separation of social anomaly from motion-statistics anomaly. Many scenarios involve motion that appears normal in isolation (e.g., a child running) but is anomalous due to relational context (e.g., running away from a guardian toward a road).
• Data Composition:
  • Synthetic Clips: 65 anomaly clips (21 training, 44 test) with per-frame binary labels. Each anomaly class is paired with a "behaviorally matched" normal clip (e.g., a child walking calmly beside a guardian) to prevent models from learning simulator-specific visual artifacts.
  • Real-World Probes: The test set includes real-world normal and anomalous clips (e.g., persons acting erratically in public) to evaluate sim-to-real transfer.
  • Total Scale: 540,888 frames across 108 clips, recorded at 800×600 resolution with an egocentric dashcam perspective.

Evaluation Protocol

The authors benchmark 11 state-of-the-art VAD methods representing four supervision regimes:

1. Semi-supervised: Trained only on normal data (e.g., Future Frame Prediction).
2. Weakly-supervised: Trained with video-level anomaly labels (e.g., VadCLIP, VERA, REWARD).
3. Unsupervised: No labels used (e.g., NP-VAD, CLAP).
4. Training-free: Foundation model reasoning using LLMs/VLMs (e.g., LAVAD, AnomalyRuler, VADTree).

Additionally, a VLM Probing Study was conducted using Qwen3-VL-4B-Instruct with structured questions (Perception, Prediction, Planning, Safety) to determine if failures stem from object detection limitations or a lack of social reasoning.

Key Results

Benchmark Performance

• Overall Failure: No method exceeded 57.2% AUC on the test set. The best performing method was the unsupervised NP-VAD (57.15%), followed by CLAP (56.12%).
• Kinematic Methods Inverted: Methods relying on motion prediction or distance from nominal distributions (e.g., FFP, REWARD, BN-WVAD) performed below chance (37–44% AUC). This confirms that socially complex anomalies are kinematically ordinary and thus indistinguishable from normal motion using standard statistical features.
• Foundation Models: Training-free methods using VLMs/LLMs (e.g., LAVAD, AnomalyRuler) clustered near chance performance (39–49% AUC).
• Supervision Hierarchy Reversal: Contrary to typical VAD benchmarks, unsupervised methods outperformed supervised ones, suggesting that label-induced bias in supervised models actively misleads them on social anomalies.

VLM Probing Findings

The probing study revealed that the failure of VLMs is representational, not perceptual:

• Object Detection vs. Social Context: The model successfully detected agents (high recall in Perception questions) but failed to register the relational context that made the scene anomalous.
• Safety Question Failure: The model answered "dangerous" for 100% of frames (both normal and anomalous), indicating a dominance of language priors over visual grounding.
• Reasoning Breakdown: When asked to plan or predict, the model was actually less likely to flag the anomaly on anomalous frames than to raise false alarms on normal frames. The anomaly did not propagate into the model's decision-making process.

Significance and Claims

The paper claims that socially complex anomalies constitute a distinct and currently unsolved challenge for autonomous driving perception.

1. Gap Identification: Current AV perception systems are optimized for appearance and trajectory signals, rendering them blind to anomalies defined by inter-agent relationships.
2. Dataset Contribution: SENSE-VAD fills the gap by providing the first egocentric, per-frame labeled benchmark specifically for social anomalies, distinguishing them from motion-based outliers.
3. Methodological Insight: The results demonstrate that refinements to existing kinematic or appearance-based models are insufficient. Reliable detection requires explicit reasoning about inter-agent social relationships.
4. Future Direction: The authors argue that future AV perception systems must move beyond zero-shot foundation model queries and instead utilize training protocols that provide direct social-context supervision, potentially leveraging scene-graph or object-centric models to represent relational structures.

The paper concludes that while SENSE-VAD is synthetic, its design principles and the inclusion of real-world probes provide a necessary foundation for developing AVs that can anticipate social anomalies rather than merely reacting to kinematic hazards.