Technical Summary: PaSBench-Video

Problem Statement

Current Multimodal Large Language Models (MLLMs) possess video understanding capabilities but lack the ability to function as proactive safety monitors. While accidents are often preceded by visually recognizable warning signs, existing benchmarks fail to evaluate the critical window between the first visible sign of danger and the moment harm occurs. Current safety benchmarks suffer from three primary limitations:

1. Static Inputs: They rely on static images or image sequences rather than continuous streaming video.
2. Lack of Temporal Calibration: They do not distinguish between premature warnings (issued before risk is visible) and late warnings (issued after harm occurs).
3. Absence of False-Positive Measurement: They omit evaluation on benign scenes, ignoring the critical issue that frequent false alarms degrade user trust and cause alarm fatigue.

Consequently, there is no controlled testbed to determine if MLLMs can causally observe a video stream, identify the precise moment a risk becomes actionable, issue a timely warning, and remain silent on safe scenes.

Methodology

Dataset Construction: PaSBench-Video

The authors introduce PaSBench-Video, a benchmark comprising 740 videos across four domains: driving, healthcare, daily life (smart home), and industrial production.

• Risk Split (481 videos): Manually verified clips containing evolving safety risks. Each video is annotated with frame-level boundaries:
  • Risk-start ($r$): The earliest frame where visual evidence supports an actionable warning.
  • Accident ($a$): The frame where the adverse event begins.
  • Risk Source & Avoidance Action: Specific hazard agents and recommended interventions.
• No-Risk Split (259 videos): Benign clips from normal driving, surveillance, and home monitoring, used exclusively to measure false-positive rates.
• Data Sources: Includes real-world datasets (e.g., DADA-2000, Nexar, OOPS) and synthetic data generated via Kling v3 Omni for healthcare and industrial scenarios where real-world data with specific annotation granularity was unavailable.
• Filtering: A rigorous two-stage filter (model-based using Qwen3.5 and Gemini-3.1-Pro, followed by human adjudication) ensures clips are predictable (risk inferable from past/current frames), avoidable (intervention possible), non-intentional (not staged stunts), and visually grounded.

Evaluation Protocol

The benchmark employs a streaming evaluation protocol:

• Causal Observation: Models receive only past and current frames within a sliding window (3-second windows, 0.3s stride).
• Task: The model must decide whether to issue a structured warning (identifying risk source and action) or remain silent.
• Metrics: A hierarchy of progressively stricter metrics parameterized by intervention lead time $\tau$ (default 1.0s):
  1. Any-Det@$\tau$: Detection of any warning before the accident.
  2. Any-Hit@$\tau$: Detection of a warning within the valid risk window $[r, a-\tau]$.
  3. First-Hit@$\tau$: The first warning of the video falls within $[r, a-\tau]$.
  4. First-Act@$\tau$: The first warning is correctly timed and correctly identifies the risk source and content.
• False-Positive Rate: Measured as the fraction of no-risk videos triggering $\ge K$ warnings.

Experimental Setup

The authors benchmarked 13 MLLMs, including open-weight models (Qwen3.5, Gemma) and proprietary models (Gemini, Claude, GPT). All models were evaluated with identical prompts, temperature 0, and frame sampling.

Key Results

1. Low Performance and Qualitative Failure

No model exceeded 20.0% on the strictest metric (First-Act@1). This indicates that fewer than one in five videos receives a correctly timed warning that also correctly identifies the risk source. The gap between detection (Any-Det) and actionable warning (First-Act) is massive; for example, Gemini 3.1 Pro detects risk in 93.6% of videos but achieves a First-Act score of only 20.0%.

2. Tight Coupling of Recall and False Positives

There is a strong positive correlation (Pearson $\rho=0.64$) between recall and false-positive rates. Models cannot improve detection without proportionally inflating false alarms. This suggests current models function as sensitivity-threshold detectors rather than systems performing genuine risk reasoning. Increasing sensitivity shifts warnings from "missed" to "too-early" but does not increase the proportion of "valid" warnings.

3. Domain-Dependent Performance

Performance varies sharply based on the visual discriminability of the domain:

• Daily Life: Models achieve moderate recall at low false-positive rates because risks are often inherently anomalous (e.g., a person falling, open flames).
• Driving: Models perform poorly, firing indiscriminately. Routine scenes (merging, crossing) and pre-collision scenes look visually similar, causing models to trigger warnings before the risk is actually visible (90% of failures for high-sensitivity models were "too-early").

4. Failure Modes

Even when timing is correct, models frequently fail to identify the risk source:

• Wrong Risk Object (36.9%): Identifying the correct entity class but the wrong specific instance (e.g., predicting a bus instead of a sedan).
• Entity-Type Swap (33.9%): Hallucinating a different category of agent (e.g., predicting a pedestrian when the hazard is a vehicle).
• These errors indicate a reliance on scene-level activity cues rather than causal reasoning about whether an activity will escalate into harm.

5. Deployment Constraints

Beyond capability gaps, engineering constraints preclude immediate deployment:

• Latency: Even the fastest models (GPT-5.4 mini) have median latencies (~3.5s) exceeding the 0.3s window stride by an order of magnitude.
• Cost: Continuous monitoring costs for top-tier proprietary models exceed $2,000/day per camera, while open-weight models suffer from prohibitive latency.

Significance and Claims

The paper claims that PaSBench-Video provides the first controlled testbed for evaluating proactive video safety warnings that isolates algorithmic progress from infrastructure improvements. By removing speed pressure and restricting models to causal observation, the benchmark reveals that current MLLMs rely on surface-level activity cues rather than reasoning about emerging harm.

The authors emphasize that the proactive warning task is not yet ready for deployment due to both engineering constraints (latency, cost) and insufficient capability (low recall, high false positives). However, they argue that benchmarking this capability now is "timely and necessary" to guide algorithmic progress toward genuine temporal risk reasoning, moving beyond simple activity detection toward risk anticipation. The dataset is released to facilitate this research, with the explicit goal of developing models that can distinguish between routine activity and imminent danger.