💻 computer science

TRACE: Evidence Grounding-Guided Multi-Video Event Understanding and Claim Generation

TRACE is an evidence grounding-guided framework that enhances multi-video event understanding and claim generation by constructing searchable timelines via OCR and object detection to enable precise, query-aware evidence localization before visual reasoning, thereby significantly improving factual completeness, citation recall, and state-of-the-art performance on benchmarks like MAGMaR 2026.

Original authors: Pengyu Yan, Akhil Gorugantu, Mahesh Bhosale, Abdul Wasi, Vishvesh Trivedi, David Doermann

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pengyu Yan, Akhil Gorugantu, Mahesh Bhosale, Abdul Wasi, Vishvesh Trivedi, David Doermann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery about a massive storm. You have a library containing thousands of hours of video footage from different news channels, social media, and government broadcasts. Your boss asks you a specific question: "How many people were missing, and did the government send aid?"

If you were a standard AI (like the ones described in the "old way"), you would be handed a giant stack of these videos and told, "Read them all and tell me the answer." The problem? The AI gets overwhelmed. It tries to watch every single second, but because it can't hold that much information in its "brain" at once, it has to skip through the videos quickly. In doing so, it misses the tiny, crucial details—like a blurry text on a news ticker saying "300 missing" or a scoreboard showing aid numbers—because it's too busy looking at the dramatic visuals of the storm itself.

TRACE is a new, smarter way to solve this. The authors call it a "Ground-Before-Reasoning" strategy. Think of it like this:

The Problem: The "Blind Search"

Current AI models are like a detective who walks into a room full of people shouting and tries to find one specific fact by just listening to the loudest voices. They miss the quiet whispers (the text on a screen) that actually contain the answer. They also get tired (run out of "context budget") if the room is too big.

The TRACE Solution: The "Index First" Approach

Instead of watching the videos blindly, TRACE acts like a librarian who first creates a searchable index before the detective even starts reading.

Here is how the process works, step-by-step:

1. The "Scanner" (Grounding)
Before the AI tries to "think" or "reason," it first runs a fast, cheap scanner over the videos.

What it does: It uses two simple tools: OCR (Optical Character Recognition, which reads text on screens like news tickers and scoreboards) and Object Detection (which spots things like microphones, podiums, or crowds).
The Analogy: Imagine a robot that quickly flips through every page of a book and writes down a list of every number, name, and object it sees, along with the time it saw them. It doesn't try to understand the story yet; it just builds a text-based timeline.
Why it helps: This turns a messy video into a clean, searchable list of facts.

2. The "Search Engine" (Localization)
Now, the human (or the user) asks their question: "Where is the info about missing people?"

What it does: A text-only AI (which is very fast and cheap) looks at that searchable timeline created in Step 1. It matches the question against the list of numbers and objects.
The Analogy: Instead of re-reading the whole book, the librarian uses the index to say, "Ah, the answer is on pages 45, 82, and 110." It ignores the rest of the book.
Why it helps: It finds the exact moments that matter without wasting time on irrelevant scenes.

3. The "Storyteller" (Claim Generation)
Now, the powerful video AI (the "Storyteller") gets to work.

What it does: It is given only the specific video clips identified in Step 2, plus the notes from the index. It is told, "Look at these specific moments and tell me the facts."
The Analogy: The detective is now handed only the three relevant pages of the book, with the important numbers highlighted. They can focus 100% of their attention on those pages to write a perfect, accurate report.
The Result: The AI generates a claim (e.g., "300 people are missing") and, crucially, cites exactly which video clips proved that fact.

4. The "Editor" (Consolidation)
Since you had 10 different videos, you might get 10 slightly different reports.

What it does: A final step combines these reports. If Video A says "300 missing" and Video B says "300 missing," the system merges them into one strong fact and keeps the citations from both videos.
The Analogy: An editor takes all the different reporters' notes, checks they agree, and writes one final, authoritative article that credits every single source.

Why This Matters (The Results)

The paper tested this system on real-world news events (MAGMaR 2026) and found that:

It found more facts: It didn't miss the small details hidden in text overlays.
It cited better: It was much better at pointing to the exact video evidence that proved its claims (improving "citation recall" significantly).
It was more accurate: It beat the previous best systems by a wide margin.

The Bottom Line

TRACE changes the game by saying: "Don't try to understand the whole video at once. First, scan it to find the clues, then use those clues to find the answer." It turns a chaotic, overwhelming task into a structured, step-by-step investigation, ensuring the AI doesn't miss the small but critical details that hold the truth.

Technical Summary: TRACE – Evidence Grounding-Guided Multi-Video Event Understanding and Claim Generation

Problem Statement

Multi-video event understanding requires models to not only recognize visual content but to locate and attribute specific pieces of evidence scattered across long, heterogeneous video corpora to answer user queries. Current Large Vision-Language Models (LVLMs) struggle in this regime due to two primary failure modes:

Context Saturation: LVLMs have limited context windows, forcing aggressive temporal subsampling that often omits brief, information-dense moments (e.g., news tickers, scoreboards, or broadcast overlays) critical for factual queries.
Misaligned Attention: When prompted directly with raw video, LVLMs tend to focus on visually salient content rather than query-relevant evidence, producing broad narrative summaries instead of precise, attributable claims.

Existing approaches often rely on brute-force context expansion or uniform frame sampling, which fails to address the core challenge of identifying which frames matter before processing them.

Methodology: The TRACE Framework

TRACE introduces a grounding-before-reasoning paradigm. Instead of asking an LVLM to jointly discover evidence and generate claims, the framework decomposes the task into a lightweight, text-based grounding stage followed by guided multimodal reasoning.

1. Structured Video Grounding

The system first converts long, unstructured videos into a text-searchable timeline using lightweight, non-visual processing:

Object Detection: YOLOv12 processes sampled frames to detect objects (e.g., people, microphones, podiums). Co-occurrence patterns provide contextual signals (e.g., identifying a press conference).
Text Recognition (OCR): An OCR module extracts visible text from broadcast overlays, scoreboards, and lower-thirds. This captures high-precision semantic signals (entities, statistics, locations) that object detectors miss.
Timeline Construction: These streams are merged into a chronological, text-serializable timeline $F = \{(t, D_t, T_t)\}$ , where $D_t$ represents detected objects and $T_t$ represents OCR text.

2. Query-Conditioned Evidence Localization

A text-only LLM (Qwen3-30B) operates on the structured timeline to perform evidence localization without invoking a vision encoder:

Alignment: The timeline is partitioned into windows and serialized. The LLM aligns these text representations with the user query and persona to identify relevant moments.
Bridging the Gap: This stage translates low-level detector outputs into semantic intent, identifying indirect evidence (e.g., a "vote percentage" overlay) even without explicit keyword matches.
Grounding Summary: The LLM synthesizes a coherent semantic description of the localized evidence, acting as a bridge between raw perception and downstream reasoning.

3. Grounding-Guided Claim Generation

A downstream LVLM (Qwen3-VL-30B) generates claims conditioned on explicit guidance:

Hybrid Frame Selection: The visual input combines uniformly sampled frames (for narrative coverage) with guidance-targeted keyframes identified during the localization stage.
Temporal Alignment: Frame indices are passed as explicit positional metadata to preserve temporal spacing in the model's rotary position embeddings, ensuring text annotations align with visual tokens.
Evidence Fusion: The LVLM receives the hybrid frames, the query, the persona, the grounding summary, and structured annotations (timestamps, objects, OCR). It is instructed to generate single-sentence claims grounded in observed evidence, prioritizing specific facts over vague paraphrases.

4. Cross-Video Claim Consolidation

Claims from multiple videos are aggregated to form a unified event report:

Semantic Clustering: Generated claims are embedded and clustered based on similarity.
Verification: An LLM verifies clusters under a strict "same-proposition" criterion to distinguish genuine paraphrases from factually distinct claims.
Citation Propagation: The system retains the most information-complete claim as the canonical representation and propagates the union of supporting video citations across all cluster members, maximizing citation recall.

Key Contributions

Grounding-Guided Pipeline: A novel architecture that constructs a structured, text-searchable video timeline via OCR and object detection, enabling query-conditioned evidence localization prior to expensive visual inference.
Ground-Before-Reasoning Paradigm: A strategy that separates evidence discovery from multimodal generation, addressing the context bottleneck and attention misalignment of current LVLMs.
Hybrid Framework: A system combining targeted keyframe selection with cross-video claim deduplication and citation propagation.
State-of-the-Art Performance: Achievement of top results on the MAGMaR 2026 benchmark and strong generalization on the WikiVideo dataset.

Experimental Results

The authors evaluated TRACE on the MAGMaR 2026 Oracle Track (8 real-world news event topics) and the WikiVideo dataset (52 queries, 398 videos).

MAGMaR 2026 Results

Overall Performance: TRACE achieved the highest scores on the official leaderboard, with an Avg. F1 of 0.499 (InfoF1: 0.551, CiteF1: 0.447), surpassing the second-ranked team (HAIVLab) by 0.049 points.
Comparison with Baselines: Compared to an unguided Qwen3-VL-30B baseline on the validation set, TRACE improved the macro-average MiRAGE F1 from 0.705 to 0.811.
Citation Recall: The most significant improvement was in citation recall, rising from 0.440 to 0.628 (+42.7% relative), demonstrating the framework's ability to identify and cite the full range of relevant sources rather than just the most salient video.
Human Evaluation: TRACE ranked second in human evaluation (3.825/5.0), trailing the top team by only 0.008 points.

WikiVideo Generalization

TRACE maintained the highest Avg. F1 (0.879) on WikiVideo, outperforming both Qwen3-VL-8B and Qwen3-VL-30B baselines.
While the absolute margin was smaller due to the shorter duration and uniform distribution of WikiVideo videos (which benefits uniform sampling), the method still achieved the highest citation recall (0.838).

Ablation Studies

Grounding Impact: All pipeline variants outperformed the strongest unguided baseline, confirming that multi-modal grounding is the primary source of improvement.
Aggregation Strategy: Embedding-based similarity aggregation with LLM verification (EMBED-SIM) outperformed pure LLM merging, particularly in citation F1.
Keyframe Augmentation: Adding guidance-targeted keyframes improved Information Recall (InfoR), showing that the text-based grounding successfully directs the model to visual evidence missed by uniform sampling.

Significance and Claims

The paper argues that structured perception-based grounding is an effective and transferable principle for video claim attribution. By shifting the burden of evidence localization from the expensive visual reasoning stage to a lightweight, text-serializable stage, TRACE mitigates context saturation and directs visual capacity toward evidentially relevant moments.

The authors claim that this approach:

Solves the "needle in a haystack" problem in long-video understanding without requiring massive context windows.
Improves factual completeness and attribution fidelity, which are critical for multi-video event understanding.
Provides a scalable alternative to brute-force long-context processing, as the grounding stage is fast, deterministic, and interpretable.

The paper concludes that while the method achieves state-of-the-art results, limitations remain, including the reliance on the COCO-80 vocabulary for object detection (limiting domain-specific entity recognition) and the non-differentiable nature of the pipeline, which prevents end-to-end error recovery. Future work is suggested to include open-vocabulary detection and adaptive frame sampling.