A Skill-augmented Agentic Framework and Benchmark for Multi-Video Understanding
This paper introduces MVX-Bench, a comprehensive multi-video benchmark covering 11 computer vision tasks, and proposes SAMA, a skill-augmented agentic framework that leverages visual tools and conflict-aware verification to significantly outperform existing models in complex multi-video reasoning.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a detective trying to solve a mystery, but instead of looking at a single crime scene photo, you have to watch five different security camera feeds simultaneously to figure out what happened.
That is the challenge this paper tackles. Current AI models are great at looking at one video at a time, but they struggle when asked to compare, contrast, and reason across multiple videos at once.
Here is the story of their solution, broken down into simple concepts and analogies.
1. The Problem: The "Blindfolded Marathon Runner"
Currently, if you want an AI to understand multiple videos, the standard method is to glue all the videos together into one giant, long strip and feed it to the AI.
- The Analogy: Imagine trying to read a book by taping 10 different chapters together into one massive scroll, then asking someone to read it while wearing a blindfold that only lets them see a tiny 2-inch window at a time.
- The Result: The AI misses details, gets confused about the order of events, and can't tell the difference between two very similar-looking videos. It's like trying to find a specific needle in a haystack by looking at the whole haystack at once without a magnet.
2. The New Benchmark: "The Olympic Gymnastics Course" (MVX-Bench)
The authors realized we can't just test AI on "easy" video questions. They built a new testing ground called MVX-Bench.
- What it is: Think of this as a 11-event Olympic gymnastics course for AI.
- The Events: Instead of just asking "What color is the car?", they ask complex questions like:
- Forensic Detection: "Which of these two videos is a deepfake?" (Spotting the fake).
- Re-Identification: "Is the person in Video A the same guy as in Video B, even though he's wearing a different hat?" (Identity matching).
- Counterfactual Reasoning: "If the driver in Video A had braked earlier, would they have hit the tree?" (Imagining "what if" scenarios).
- The Score: They tested 20 different AI models on this course. The results were shocking: even the smartest AI models only got about 50% of the answers right. It's a huge challenge that current technology hasn't solved yet.
3. The Solution: "The Detective Agency" (SAMA)
To fix the "glued video" problem, the authors created a new system called SAMA. Instead of one giant brain trying to do everything at once, SAMA acts like a Detective Agency with a team of specialists.
Here is how the agency works:
A. The Case Manager (The Planner)
This is the boss. It doesn't look at the videos itself. Instead, it reads the question and decides: "Okay, to solve this, I need to check the time, count the people, and compare the colors. Let's call the specialists."
B. The Specialists (The Tools)
The Case Manager hires different experts for specific jobs:
- The Counting Expert: Uses a specialized tool to count objects (like cars or people) frame-by-frame.
- The Similarity Expert: Uses a "fingerprint scanner" to see if two videos look mathematically identical.
- The Subtitle Reader: Reads the text on the screen or the spoken words to find clues.
- The Tracker: Follows a specific person across different camera angles to make sure it's the same person.
C. The "Truth Squad" (Conflict Resolution)
This is the most clever part. Sometimes, the specialists disagree.
- Scenario: The "Counting Expert" says, "I see 2 bags." But the "Visual Reader" says, "I see 3 bags."
- The Old Way: The AI would just guess or get confused.
- The SAMA Way: The "Truth Squad" steps in. It says, "Hold on, you two disagree. Let's zoom in on that specific 5-second window and look again." It forces the AI to re-examine the evidence until the specialists agree.
4. The Results: Why It Matters
When they put this "Detective Agency" (SAMA) on the "Olympic Course" (MVX-Bench):
- It didn't just get better; it leapfrogged the competition.
- It took a relatively small AI model (which usually scores poorly) and, by giving it this team structure, boosted its score from 32% to 52%.
- It even beat some of the massive, expensive "super-computer" AI models (like GPT-4o) that try to do everything alone.
The Big Takeaway
The paper teaches us that being smart isn't just about having a bigger brain; it's about knowing how to use the right tools.
Just like a human detective doesn't try to memorize every crime scene in the city, but instead uses magnifying glasses, fingerprint kits, and witness interviews to solve a case, this new AI framework uses a structured team of tools to solve complex video mysteries. It proves that for the hardest video problems, organization and verification are more important than raw computing power.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.