💻 computer science

MARQUIS: A Three-Stage Pipeline for Video Retrieval-Augmented Generation

MARQUIS is a three-stage pipeline that significantly enhances video retrieval-augmented generation by addressing limitations in complex query handling and multi-video synthesis through query expansion, structured evidence extraction, and controlled article generation, achieving state-of-the-art performance on the MAGMaR2026 shared task.

Original authors: Debashish Chakraborty, Dengjia Zhang, Jialiang Jin, Hanting Liu, Katherine Guerrerio, Hanxiang Qin, Tyler Skow, Alexander Martin, Reno Kriz, Benjamin Van Durme

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Debashish Chakraborty, Dengjia Zhang, Jialiang Jin, Hanting Liu, Katherine Guerrerio, Hanxiang Qin, Tyler Skow, Alexander Martin, Reno Kriz, Benjamin Van Durme

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a journalist tasked with writing a detailed news article about a complex event, like a massive storm or a political election. However, you don't have a single reporter on the ground; instead, you have a library containing 100,000 different video clips of the event, recorded by hundreds of different cameras. Your job is to find the right clips, figure out what actually happened in them, and write a coherent story that cites exactly which video proved each fact.

This is the challenge the MARQUIS system tackles. The paper argues that current AI tools are bad at this specific job. They either can't find the right videos when the question is complicated, or they get overwhelmed by trying to read too many videos at once and end up hallucinating (making things up).

MARQUIS solves this by acting like a three-stage newsroom assembly line, breaking the massive job into smaller, manageable steps.

Stage 1: The Detective's Search (Retrieval)

The Problem: If you ask an AI, "Tell me everything about the 2025 election results," a standard search engine might get confused. It tries to turn that whole long question into one single "search code" (embedding), which often misses the nuance. It's like trying to find a specific needle in a haystack by describing the whole haystack at once.

The MARQUIS Solution:
Instead of one big search, MARQUIS acts like a detective who breaks a big case into small clues.

Decomposition: It takes your big question and breaks it into 20+ tiny, specific questions (e.g., "How many seats did the Liberals win?" "What was the voter turnout?" "Which districts flipped?").
Parallel Search: It searches the video library for each of those tiny questions separately.
The Fusion: It takes all the lists of videos found for the tiny questions and merges them into one master list.
The Re-Ranker: Finally, a specialized "video judge" looks at the top candidates and re-orders them to make sure the most relevant ones are at the very top.

The Result: This method is like using a magnifying glass to find specific needles rather than guessing where the whole haystack is. It improved the system's ability to find the right videos from a score of 0.195 to 0.759 (a massive jump).

Stage 2: The Fact-Checkers (Information Extraction)

The Problem: Once you have the videos, you can't just feed the whole video file to a writer. The writer needs specific facts. Also, videos can be tricky; sometimes a person says something that isn't true, or the camera angle is misleading.

The MARQUIS Solution:
MARQUIS doesn't just "read" the video; it extracts specific "claims" and then calibrates them.

Three Types of Notes:
- General Notes: "A woman in a red coat is speaking." (Facts that might be useful later).
- Targeted Claims: "The video shows 50 people rescued." (Facts specifically answering your question).
- Q&A: The system asks the video specific questions and gets answers.
The Calibration (The Lie Detector): This is a crucial step. Before the writer sees a claim, a "calibrator" model looks at the claim and the original video side-by-side. It asks: "Does this video actually prove this sentence?" It assigns a probability score (0 to 1). If the video doesn't support the claim, it gets filtered out.

The Analogy: Imagine a team of fact-checkers who read every sentence a reporter writes and then go back to the raw footage to verify it. If the footage doesn't match, the sentence is thrown in the trash.

Stage 3: The Writers (Article Generation)

The Problem: Even with good facts, writing a smooth article is hard. If you give a writer 500 disconnected bullet points, they might write a messy list. If you give them a 2-hour video transcript, they might get lost in the details and forget the main point.

The MARQUIS Solution:
The system offers three ways to write the article, but the best one uses a structured approach:

Clustering: It groups the verified facts into themes (e.g., "Casualties," "Rescue Efforts," "Government Response").
Summarizing: It writes a short, cited sentence for each theme.
Polishing: It stitches those sentences together into a smooth, readable news story, ensuring every fact has a citation (like [Video #45, 0:12-0:15]).

The "Recursive" Option (MARQUIS-RLM):
The paper also introduces a special "Manager" AI (based on Recursive Language Models). Instead of just writing once, this Manager acts like a project manager with a persistent notebook.

It looks at the facts it has.
It realizes, "I'm missing the casualty count for the north district."
It goes back to Stage 1 and 2, specifically asks for that missing info, adds it to its notebook, and checks for conflicts.
Only when the notebook is full and consistent does it write the final article.

The Bottom Line

The paper claims that by breaking the problem down—searching smarter, verifying facts strictly, and organizing the writing process—MARQUIS produces much better results than current methods.

Retrieval: Found the right videos much more often.
Generation: Produced articles that humans rated higher (3.83 out of 5 vs. 3.09 for the baseline) because they were more coherent and better sourced.
Reliability: The system is designed to be "grounded," meaning it refuses to write facts that aren't backed up by the video evidence, avoiding the "hallucinations" common in other AI systems.

In short, MARQUIS turns a chaotic pile of 100,000 videos into a reliable, well-sourced news article by acting like a highly organized, multi-step newsroom rather than a single, overwhelmed writer.

Technical Summary: MARQUIS

Problem Statement

The paper addresses the challenge of grounded article generation from large-scale video corpora. While video archives offer immediate and broad documentation of real-world events, converting this raw audiovisual evidence into well-sourced, analytical articles remains a largely manual process. Current automated systems face significant limitations at both ends of the pipeline:

Retrieval: Existing methods struggle with complex, multi-faceted queries that cannot be captured by a single embedding vector. They often fail to surface all relevant videos when a query combines multiple implicit and explicit information needs.
Generation: Generation models face memory constraints when processing long, multi-video contexts. Furthermore, existing Vision-Language Models (VLMs) are typically designed for low-level tasks (captioning, entity-centric QA) rather than the high-level synthesis required for article generation. They often fail to reason across multiple videos or synthesize evidence coherently.

These limitations compound in a full pipeline: retrieval errors propagate missing or irrelevant evidence into the generation stage, where models already struggle to reason over the context they receive.

Methodology: The MARQUIS Pipeline

The authors present MARQUIS (Multimodal Article generation via Retrieval, Query decomposition, Uncertainty calibration, and Iterative evidence Synthesis), a three-stage pipeline designed to modularly address these limitations.

Stage 1: Video Retrieval

The retrieval pipeline operates in two stages to handle complex queries:

Query Decomposition and Expansion: Instead of encoding a complex, instructional query as a single vector, the system uses an LLM to decompose the query into $N$ atomic sub-queries. Each sub-query targets a single retrievable fact and is phrased as a concise search phrase.
Dense Retrieval: Both the original query and the decomposed sub-queries are retrieved against an omnimodal index using OmniEmbed, producing ranked lists of top candidates.
Rank Fusion: The ranked lists from the sub-queries are aggregated using strategies such as Reciprocal Rank Fusion (RRF), Sum/Max/Mean similarity, or Weighted RRF.
Reranking: The fused candidates (top 100) are passed to RankVideo, a video-native reranker, to reorder the list based on refined judgments, balancing performance and scalability.

Stage 2: Information Extraction and Calibration

Retrieved videos are converted into structured, calibrated evidence artifacts:

Extraction Streams: The system employs three complementary extraction components using Qwen3.5:
1. Query-Agnostic Notes: Captures directly observable visual events, OCR, and spoken content without conditioning on a specific query.
2. Query-Conditioned Claims: Extracts claims specifically relevant to the information need, avoiding generic descriptions or unsupported inferences.
3. Question-Answer (QA) Extraction: Decomposes information needs into sub-questions and answers them using retrieved video content and transcripts. This includes both single-shot and iterative variants.
Calibration: Each extracted artifact (note, claim, or QA pair) is scored against its source video by CLUE, a calibrated multimodal uncertain-inference model. This produces a support probability ( $s_\theta \in [0, 1]$ ) estimating whether the video supports the claim. This step separates evidence creation from support estimation, allowing for the filtering of unsupported claims before generation.

Stage 3: Article Generation

The pipeline synthesizes filtered evidence into a cited article. The authors compare three strategies:

BULLET: Renders evidence directly as a numbered list with inline citations (conservative baseline).
CAG (Collaborative Article Generation): A single-pass baseline that organizes evidence and generates a cited article in one synthesis pass.
GINGER: A staged approach that clusters evidence into thematic facets, ranks clusters by relevance, summarizes each cluster into a cited sentence, and finally rewrites them into a coherent article.

MARQUIS-RLM: Recursive Control

The authors introduce MARQUIS-RLM, an instantiation of Recursive Language Models (RLMs). This system treats the pipeline modules (retrieval, extraction, QA, calibration) as callable tools within a persistent, structured memory environment.

Think–Act–Observe Loop: The Root LM iteratively plans actions, executes tool calls (e.g., video_qa, transcribe), and observes results.
Structured Memory: Unlike standard RLMs that rely on context windows, MARQUIS-RLM maintains a dynamic memory bank (fact tables, keyword indexes) that can be searched, reused, and revised. This mitigates information forgetting and enables the resolution of cross-video conflicts and the identification of information gaps before the final article is written.

Key Contributions

MARQUIS Pipeline: A three-stage framework for large-scale video retrieval-augmented article generation that modularly decomposes retrieval, extraction, and generation.
Retrieval Improvements: A two-stage retrieval approach combining query expansion with rank fusion and video-native reranking, which significantly improves retrieval performance on the MAGMaR2026 benchmark.
Generation Strategies: A comparative analysis of claim-based, QA-based, and RLM-controlled generation, demonstrating that query decomposition and video-grounded QA improve human preference scores over baselines.
RLM Integration: The introduction of MARQUIS-RLM, which uses structured memory and iterative tool use to manage evidence, resolve conflicts, and curate facts, achieving the highest citation recall among non-QA systems.

Experimental Results

The system was evaluated on the MAGMaR2026 shared task (based on WikiVideo data).

Retrieval: The proposed retrieval pipeline improved nDCG@10 from 0.195 to 0.759 over a dense retrieval baseline (OmniEmbed). The combination of query decomposition, RRF fusion ( $K=10$ ), and RankVideo reranking yielded the best results.
Article Generation (Oracle Setting): In oracle generation (where systems receive ground-truth relevant videos), the ITER-QA-BASE system improved the average human score from 3.09 to 3.83 over the CAG baseline.
System Comparison:
- GINGER outperformed the CAG baseline in human scores (3.12 vs 3.09) and citation precision, suggesting that staged decomposition helps organize evidence.
- MARQUIS-RLM achieved a human score of 3.30 and the strongest citation recall (27.2%) among non-QA systems, though it showed lower precision, indicating a trade-off between coverage and relevance.
- QA-based systems (Iterative and Single-Shot) achieved the highest human scores (up to 3.83), but suffered from "zero-score" failures when the QA pipeline could not retrieve relevant videos for specific topics, leading to conservative empty outputs.

Significance

The paper argues that grounded generation from video is best framed as an evidence-management problem rather than a simple summarization task. The authors claim that effective systems should:

Retrieve broadly by decomposing complex queries.
Extract evidence atomically.
Estimate source support via calibration.
Synthesize only from selected, verified evidence.

The work demonstrates that explicit query decomposition and video-native reranking substantially improve retrieval, while modular generation strategies (clustering, QA, RLM control) offer complementary trade-offs between coherence, factuality, and citation coverage. The authors suggest that future work should focus on improving learned calibration, tighter integration of retrieval and extraction, and generation methods that combine structured evidence organization with iterative control.