💻 computer science

AgentCVR: Active Multi-Agent Cross-Video Reasoning via Script-Simulated Reinforcement Learning

This paper introduces AgentCVR, a multi-agent framework that addresses the limitations of single-pass strategies in Cross-Video Reasoning by employing a Master Agent to iteratively coordinate specialized Visual and Audio agents for targeted evidence acquisition, utilizing a novel Script-Simulated Reinforcement Learning approach to optimize training efficiency while achieving state-of-the-art performance.

Original authors: Yilun Qiu, Jiahe Wang, Cilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, Chun Yuan

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yilun Qiu, Jiahe Wang, Cilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, Chun Yuan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Too Much Information" Trap

Imagine you are a detective trying to solve a mystery, but instead of one crime scene, you are given five different security camera tapes from different angles and times.

Current AI models (the "smart" computers we use today) try to solve this by shoving all five tapes into a single, compressed file and reading it all at once. It's like trying to find a specific needle in a haystack by squinting at a photo of the entire haystack. Because the AI has to compress so much data, it often misses the tiny, crucial clues (the needles) hidden in the background. It gets overwhelmed and makes guesses based on incomplete information.

The Solution: AgentCVR (The Detective Team)

The authors propose a new way called AgentCVR. Instead of one AI trying to do everything at once, they created a team of specialized detectives led by a Master Detective.

The Master Agent (The Team Leader): This AI doesn't look at the videos directly. Instead, it acts like a project manager. It reads the question, thinks about what it needs to know, and then sends specific instructions to its team members.
The Visual Agent (The Eye): When the leader says, "Go check the kitchen in Video 2 between 1:00 and 1:30," this agent zooms in only on that specific slice of time and reports back what it sees.
The Audio Agent (The Ear): If the leader thinks, "Maybe they said something about the fire," this agent listens only to that specific time slice and reports back the dialogue or sounds.

The Magic: Instead of reading the whole book at once, the team asks questions, looks at specific pages, listens to specific chapters, and pieces the story together step-by-step. This ensures they don't miss the rare, critical clues hidden in the noise.

The Training Challenge: The "Expensive Gym" Problem

To teach this team leader how to be smart, you usually need to let it practice millions of times. But in the real world, watching videos is expensive and slow (like paying a high hourly rate to watch a movie). If the AI has to watch a video, make a mistake, and try again, it costs a fortune in computer power.

The Innovation: Script-Simulated RL (The "Text-Based Simulator")

The authors came up with a clever trick to train the AI without burning money on video processing.

Imagine you want to train a pilot. Instead of making them fly a real, expensive plane for every practice session, you put them in a text-based flight simulator.

The Script: A powerful language model writes a "script" describing the video (e.g., "At 10 seconds, a red car drives by; at 20 seconds, a dog barks").
The Simulator: The AI agent interacts with this text script instead of the actual video. It asks, "What happens at 10 seconds?" and the simulator replies, "A red car drives by."
The Result: The agent learns the logic of how to investigate (when to look, what to listen for, when to stop) using cheap text.

Once the agent is an expert in the "text simulator," the authors simply swap the text simulator for the real video tools. Because the agent learned the strategy of investigation, it can immediately apply those skills to real videos without needing to relearn everything from scratch.

The Results: Smarter and Faster

When they tested this system on a massive benchmark called CrossVid (which is full of tricky questions requiring multiple videos):

Beating the Old Way: AgentCVR significantly outperformed the "single-pass" models that try to swallow all videos at once.
Rivaling the Giants: It performed almost as well as the most powerful, expensive, closed-source AI systems (like the top-tier models from big tech companies), even though AgentCVR uses smaller, open-source models.
Precision: It was particularly good at finding exactly when something happened (temporal grounding) and comparing details across different videos.

Summary

AgentCVR changes the game from "reading the whole library at once" to "hiring a team of specialists to find specific clues." It uses a clever "text-based practice field" to train the team leader efficiently, allowing it to solve complex video mysteries that previously stumped even the smartest AI.

Technical Summary: AgentCVR

Problem Definition

Cross-Video Reasoning (CVR) represents a critical frontier in multimodal intelligence, requiring models to retrieve, align, and aggregate evidence distributed across multiple independent video streams to answer complex queries. Unlike single-video tasks where evidence is locally confined, CVR involves sparse, temporally misaligned cues that often necessitate explicit cross-video comparison.

Current approaches, primarily Multimodal Large Language Models (MLLMs), typically employ a "passive single-pass" paradigm. In this strategy, all candidate videos are encoded into a shared compressed context to generate an answer in one step. The authors argue this design is fundamentally flawed for CVR because compressing long videos into fixed-size representations often obscures rare but critical evidence. Consequently, these models struggle with weak evidence attribution and unreliable cross-video comparison, particularly for temporal and comparative queries.

Methodology: AgentCVR

To address these limitations, the authors propose AgentCVR, a framework that reframes CVR from a context-length problem into an active evidence-acquisition task. The system operates on a multi-round, multi-agent paradigm rather than a single-pass inference.

1. Multi-Agent Architecture

AgentCVR utilizes a lightweight Master Agent as a central controller that iteratively coordinates specialized Visual and Audio Agents.

Master Agent: At each step, it reasons over the user query and the accumulated interaction history to dynamically decide which modality-specific agent to invoke, which temporal segment to inspect, and when to terminate.
Action Space: The action space is decomposed into three subspaces:
- Visual Query: Specifies a video ID, temporal interval, and focus prompt for the Visual Agent to extract localized textual observations.
- Audio Query: Triggers the Audio Agent to retrieve speech or sound descriptions from a specific interval, leveraging auditory signals for cross-video alignment.
- Answer Action: Terminates the process once sufficient evidence is collected to generate the final answer.
  This design separates evidence gathering from high-level reasoning, ensuring the reasoning module receives explicit, query-conditioned inputs rather than an overloaded context window.

2. Script-Simulated Reinforcement Learning (RL)

To optimize the Master Agent's policy for complex, long-horizon decision-making, the authors introduce Script-Simulated RL. Direct online RL with raw video inference is prohibitively expensive due to the computational overhead of repeatedly invoking heavy vision-language models.

Synthetic Script Construction: The training environment is a lightweight, text-based simulator. An LLM generates synthetic semantic scripts ( $W_{script}$ ) that define video sets, temporally grounded events, and cross-video relations. These scripts serve as a surrogate for raw video data.
Text-Based Simulator ( $M_{sim}$ ): During online exploration, the simulator replaces raw-video inference. Given an agent action and a script slice, $M_{sim}$ performs semantic lookup to generate natural-language observations that approximate feedback from visual and audio agents. Crucially, the simulator is constrained to return "no evidence" if no relevant event exists, preventing hallucination.
Training Objective: The Master Agent is optimized using Group Relative Policy Optimization (GRPO). The reward function consists of:
- Correctness Reward: A sparse signal based on the final answer's accuracy.
- Formatting Reward: An auxiliary reward encouraging valid tool invocation formats to ensure stable execution.
Sim-to-Real Transfer: The policy learned in the script-based simulator is directly transferred to the real-world inference pipeline, where the agent interacts with actual visual and audio tools over raw videos.

Key Contributions

Paradigm Shift: The paper proposes AgentCVR, shifting the CVR paradigm from passive single-pass context compression to active, multi-round evidence acquisition.
Script-Simulated RL: It introduces a novel training paradigm that utilizes LLM-generated semantic scripts and a lightweight text-based simulator to train multi-agent policies, bypassing the high costs of online multimodal exploration.
Empirical Validation: Extensive experiments demonstrate that AgentCVR achieves strong performance across comprehensive CVR tasks, outperforming single-pass baselines and matching state-of-the-art closed-source systems in specific alignment and localization tasks.

Experimental Results

The framework was evaluated on CrossVid, a large-scale benchmark specifically designed for CVR, covering four dimensions: Comparative Analysis, Temporal Understanding, Multi-view Reasoning, and Free-form QA.

Performance: AgentCVR (specifically the 8B variant) significantly outperforms open-source single-pass models (e.g., Qwen3-VL series) and adapted single-video agents. It achieves performance comparable to powerful closed-source frontier models (e.g., Doubao-1.5-VL-pro, GPT-4.1, Gemini-2.5-Pro), particularly in tasks requiring precise evidence localization and fine-grained cross-video alignment.
Ablation Studies:
- Architecture: Replacing the single-pass baseline with the multi-agent architecture yielded clear improvements at both 4B and 8B scales.
- Training: Introducing Script-Simulated RL further boosted performance, validating the necessity of explicit policy optimization for mastering long-horizon interactions.
- Modality Synergy: While visual perception is the primary driver of evidence extraction, the inclusion of the Audio Agent provided complementary cues, improving performance on tasks requiring cross-modal verification.
Efficiency: The Script-Simulated RL approach achieved approximately a 5x speedup in training latency compared to real-world video interactions while maintaining high alignment (Sim-to-Real consistency) in decision-making.

Significance and Claims

The authors claim that AgentCVR establishes a new, efficient, and highly interpretable path for multimodal agents to actively reason across complex, real-world video streams. By treating CVR as an evidence-acquisition problem, the framework overcomes the context-overload limitations of current MLLMs.

The paper modestly acknowledges that while AgentCVR narrows the gap with closed-source systems, a marginal performance gap remains. It also notes limitations inherent to multi-turn reasoning, such as potential instruction drift, premature cognitive closure in ambiguous scenarios, and higher inference latency compared to single-pass methods. The work highlights the importance of active perception and efficient policy learning via simulation as critical components for advancing video understanding beyond single-video analysis.