CRAFT: Critic-Refined Adaptive Key-Frame Targeting for Multimodal Video Question Answering
The paper introduces CRAFT, a query-conditioned pipeline that combines dynamic keyframe selection, multilingual ASR, and a hybrid critic loop to achieve state-of-the-art performance in grounded multi-video question answering by iteratively verifying claims and consolidating evidence with precise source attribution.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a news editor trying to write a single, accurate article about a major event (like an earthquake or an election) based on dozens of different video clips from around the world. Some clips are in English, some in Burmese, some are hours long, and many contain irrelevant footage.
Your goal is to write a report where every single sentence is backed up by a specific video clip, and you must cite exactly which clip proved that fact. If you guess a detail or cite the wrong video, your article is considered "hallucinated" (fake).
This is the challenge the paper CRAFT tackles. Here is how they solved it, explained simply:
The Problem: The "Needle in a Haystack"
Real-world news videos are messy.
- Too Long: A 2-hour video is too big for a computer to "watch" all at once. If you just pick random frames (like taking a photo every 10 seconds), you might miss the exact moment the earthquake happened.
- Missing Audio: A lot of the truth is spoken, not shown. If a reporter says, "The bridge collapsed at 3 PM," but the camera only shows the rubble, a computer looking only at pictures will miss that crucial detail.
- Lying Machines: Even smart AI models sometimes make things up. They might confidently state a fact that isn't actually in the video, or mix up two different events.
The Solution: CRAFT (The Smart Newsroom)
The authors built a system called CRAFT (Critic-Refined Adaptive Key-Frame Targeting). Think of it as a highly organized newsroom with a team of specialists rather than one overworked editor.
Here is the step-by-step workflow:
1. The "Smart Camera" (Dynamic Keyframe Selection)
Instead of watching the whole video or picking random frames, CRAFT acts like a smart camera operator.
- The Analogy: Imagine you are looking for a specific scene in a 2-hour movie. Instead of watching the whole thing, you ask the AI, "Show me only the frames where the earthquake is visible."
- How it works: The system scans the video and picks out only the frames relevant to your specific question. This saves time and ensures the AI focuses on the right evidence.
2. The "Translator" (ASR & Translation)
The system doesn't just look; it also listens.
- The Analogy: It's like having a translator in the room who listens to every word spoken in the video (even in foreign languages like Burmese or Nepali) and writes it down in English.
- How it works: It uses speech-to-text technology to turn spoken words into a transcript. If the audio is noisy or in a rare language, it has a "backup translator" ready. It also checks the transcript to make sure the AI didn't get stuck repeating the same word over and over (a common glitch).
3. The "Fact-Checker" Loop (The Critic)
This is the most important part. The system doesn't just write the article once; it revises it.
- The Analogy: Imagine a junior reporter writes a draft. Then, a senior editor (the "Critic") reads it and asks three tough questions:
- Time Check: "Did this actually happen at the time you said?" (Temporal Grounding).
- Contradiction Check: "Does this sentence contradict what you said in the previous paragraph?" (Cross-Claim Screening).
- Final Verdict: "Is this claim actually supported by the video, or are you making it up?" (Adjudication).
- How it works: If the Critic finds a mistake, it sends the draft back to the AI with instructions to fix it. This happens up to four times until the facts are solid.
4. The "Editor-in-Chief" (Citation Merging)
Finally, the system combines all the verified facts into one report.
- The Analogy: Imagine five different reporters found the same fact (e.g., "The bridge collapsed"). Instead of writing the sentence five times, the Editor-in-Chief writes it once but attaches all five video clips as proof.
- How it works: This ensures the final report is concise but fully supported by every piece of evidence found.
The Results: Did it Work?
The team tested CRAFT on a difficult benchmark called MAGMaR 2026, which involves real-world news events.
- The Score: CRAFT achieved the highest scores among all tested systems. It was better at finding the right facts (Recall) and correctly citing the videos (Citation F1) than other powerful AI models.
- The Secret Sauce: The paper found that the biggest improvements came from:
- Breaking facts down into tiny, atomic pieces (Atomic Claims).
- Listening to the audio (ASR).
- The "Critic" loop that forced the AI to check its own work.
Summary
CRAFT is like a super-efficient, multi-lingual newsroom that doesn't just watch videos, but listens to them, picks out the most important moments, argues with itself to find errors, and produces a final report where every single sentence is backed up by a specific video clip. It proves that by adding a "critic" to check the work, AI can become much more reliable at telling the truth about real-world events.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.