Technical Summary: MAVEN (Multi-stage Agentic Video Event aNnotation)

1. Problem Statement

Training Vision-Language Models (VLMs) for video event reasoning requires high-quality, structured annotations that capture temporal dynamics, spatial relationships, causal factors, and multi-step reasoning (Chain-of-Thought, or CoT). Current approaches face three primary limitations:

• Scalability vs. Detail: Manual labeling cannot support the scale required for training, while existing auto-labeling methods often rely on flat, single-pass video descriptions that lose fine-grained detail.
• Rigidity: Many pipelines are constrained to fixed taxonomies or specific sensor inputs (e.g., ego-centric driving), limiting generalizability to new domains like surveillance or public safety.
• Lack of Reusable Intermediates: Existing methods often generate Q&A directly from raw video or single captions, creating a "lossy" process where information is irrecoverably discarded, preventing the generation of diverse downstream task formats from a single annotation pass.

2. Methodology: The MAVEN Pipeline

MAVEN is a multi-stage agentic pipeline designed to transform raw videos into structured, multi-task training data centered around an Event of Focus (EoF). The EoF is defined as any notable event (anomalous or routine) that the pipeline characterizes. The pipeline operates through three sequential stages:

Stage 1: Three-Level Video Captioning

To avoid information loss, MAVEN generates three complementary caption levels using a VLM (e.g., Gemini 3.1 Pro):

1. Global Caption: A holistic description of scene layout, weather, time of day, and pre-event baseline behavior.
2. Dense Caption: A temporally grounded, event-level description pairing major events with precise start/end timestamps to establish a causal chain.
3. Chunk Captions: Fine-grained descriptions of short video segments (5–30s) to recover subtle behaviors, small objects, and decisive moments missed by global captions.
   These levels are designed to be mutually corrective, where global context disambiguates vague details, and dense captions impose temporal structure on chunk observations.

Stage 2: MSTED Synthesis

An LLM (e.g., Gemini 3.1 Flash) consolidates the three caption levels into a Multi-Scale Spatio-Temporal Event Description (MSTED). This explicit intermediate representation serves two critical functions:

• Verification Checkpoint: It allows human or automated review for completeness and accuracy before Q&A generation, preventing error propagation.
• Sole Input for Q&A: Downstream Q&A generators receive only the MSTED, not the raw video or original captions. This ensures the model cannot hallucinate details absent from the structured representation.
  The MSTED comprises: (1) Holistic Scene Description, (2) Temporal and Spatial Localization, and (3) Event of Focus Description (category, properties, root cause, consequences).

Stage 3: Multi-Task CoT Q&A Generation

A second LLM pass generates three task formats grounded solely on the MSTED:

• MCQ: 4-option questions with step-by-step reasoning on timestamps and spatial locations.
• Binary Verification: Yes/No questions with supporting reasoning.
• Open-Ended QA: Free-form questions requiring causal or descriptive reasoning.

3. Key Contributions

A. Agentic Domain Adaptation

MAVEN packages the pipeline as a single Agent Skill. Given a new domain description and target question examples, the agent performs backward inference to determine what the MSTED must capture to make those questions answerable. It then redesigns all prompts top-down across the pipeline stages (captioning, synthesis, Q&A) to adapt to new camera views, event types, and question styles without manual prompt engineering.

B. Hierarchical Pipeline Refinement

Beyond one-shot adaptation, MAVEN supports iterative refinement via human feedback. When systematic errors are identified, the agent:

1. Classifies Errors: Uses a structured taxonomy (misinformation, hallucination, missing info, temporal/spatial/attribution errors).
2. Traces Root Causes: Maps errors back through the pipeline hierarchy to the originating stage (e.g., a specific captioning level).
3. Applies Targeted Fixes: Distinguishes between prompt gaps (fixed by rewriting prompts) and system limitations (fixed by inserting new pipeline stages or modifying structure). For example, if uniform chunking splits events, the agent can introduce a new "event-centered highlight" stage.

C. Dataset and Training

The authors applied MAVEN to label 5,300+ traffic videos (3,841 CCTV and 1,500 dashcam). They fine-tuned Cosmos-Reason2-8B using a Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL, specifically DAPO) protocol.

4. Experimental Results

Private CCTV Evaluation Set

Fine-tuning on MAVEN-labeled CCTV data yielded significant improvements over zero-shot Cosmos-Reason2 and surpassed both Gemini 2.5 Pro and Gemini 3.1 Flash:

• MCQ Accuracy: +38.8 points over zero-shot (reaching 86.25% vs. 82.50% for Gemini 2.5 Pro).
• Verification Accuracy: +35.0 points (85.00% vs. 76.25% for Gemini 2.5 Pro).
• Open-Ended Score: +24.1 points (39.45 vs. 15.60 for Gemini 2.5 Pro).

AccidentBench (Public Benchmark)

• Cross-Domain Transfer: A model trained only on CCTV data (never seeing dashcam footage) matched Gemini 2.5 Pro (40.6% vs. 40.3%) and approached Gemini 3.1 Flash (42.8%) on the dashcam-heavy AccidentBench benchmark. This suggests the structured CoT reasoning capability is domain-general.
• Agent Adaptation: Adding agent-adapted dashcam annotations narrowed the gap to Gemini 3.1 Flash, particularly on short videos.
• RL Post-Training: RL further pushed overall performance to 44.2%, exceeding both Gemini baselines, with the most significant gains on "Hard" questions requiring multi-step causal inference.

Ablation Studies

Comparing MAVEN against a flat, single-pass captioning baseline (using the same teacher models) showed MAVEN outperformed the baseline by +6.25 MCQ points. This confirms that the gains stem from the multi-stage structure and explicit MSTED intermediate, not merely from the strength of the teacher models.

Qualitative Generalization

The agentic workflow successfully adapted the pipeline to warehouse surveillance and public safety domains using only domain descriptions, correctly capturing complex multi-actor interactions and suspicious behaviors without manual re-engineering.

5. Significance and Claims

The paper claims that MAVEN addresses the critical bottleneck of producing structured, causal reasoning data at scale. Its significance lies in:

1. Structured Intermediate Representations: By enforcing an explicit MSTED, the pipeline avoids the information loss inherent in single-pass auto-labeling, enabling diverse downstream tasks from a single annotation effort.
2. Domain Generalization: The structured reasoning induced by the pipeline allows models trained on one domain (CCTV) to generalize effectively to another (Dashcam), suggesting that reasoning patterns are more transferable than raw visual features.
3. Self-Improving Pipelines: The combination of agent-driven adaptation and hierarchical refinement allows the pipeline to evolve and fix its own structural limitations based on human feedback, moving beyond static, brittle annotation workflows.

The authors conclude that while current limitations include reliance on specific teacher models (Gemini) and the need for human review in the refinement loop, the approach demonstrates a viable path toward self-optimizing, domain-agnostic video reasoning training systems.