SagaQA: A Multi-hop Reasoning Benchmark for Long-form Narrative Understanding in TV Series

Technical Summary of SAGAQA: A Multi-hop Reasoning Benchmark for Long-form Narrative Understanding in TV Series

Problem Statement
Current Multimodal Large Language Models (MLLMs) have demonstrated significant capabilities in short-term visual perception and causal reasoning within brief video clips. However, a critical disparity exists between these short-context abilities and the capacity for long-term narrative comprehension. Existing benchmarks typically focus on adjacent clips, low-level understanding, or single-file narratives, often relying on retrieval strategies that isolate key frames. These approaches fail to address the complexity of full-length TV series, where understanding requires connecting disparate events, character arcs, and multimodal cues across extended temporal windows (e.g., 20 hours of video). There is a lack of benchmarks that demand "big-hop" reasoning to synthesize information from completely different episodes within a coherent narrative chain.

Methodology
To address this gap, the authors introduce SAGAQA, a benchmark specifically designed for multi-hop narrative reasoning over full-length TV series.

• Dataset Construction: SAGAQA consists of 200 question-answer (QA) pairs derived from the soap opera As the World Turns (2007–2010). Each question targets a window of 20 consecutive episodes (approx. 20 hours of video). The dataset generation pipeline leverages the SummScreen3D dataset, using an iterative process where an LLM (Gemini 2.5 Flash) generates synthetic QA pairs based on episode summaries.
• Core Constraints: The dataset enforces three strict criteria:
  1. Multi-Episode: Answers cannot be derived from a single episode.
  2. Multi-Hop: Questions require a series of dependent reasoning steps (averaging 4.2 hops) to connect information across different episodes.
  3. Multi-Modal: Answers require analyzing both visual elements and dialogue; dialogue alone is insufficient.
• Evaluation Framework: The authors evaluate existing and novel video planning strategies, categorizing them into three classes:
  • Parallel Planners: Decompose a composite question into independent sub-questions executed simultaneously.
  • Sequential Planners: Answer questions through a step-by-step sequence, refining queries based on previous outputs.
  • Hybrid Planners: A novel approach combining both strategies. The agent selects actions (e.g., BREAKDOWN or MOVE_ON) but executes multiple independent queries in parallel during the BREAKDOWN phase, aiming to balance broad exploration with deep reasoning.
• Implementation: Experiments utilize the VideoRAG framework for retrieval, with planning and answer generation handled by LLMs such as Qwen3-30B-Instruct and Mistral-Small-3.1-24B.

Key Results
The evaluation on SAGAQA reveals distinct performance characteristics among the planning strategies:

• Hybrid Superiority: Hybrid planners consistently outperform purely Parallel or Sequential planners, particularly in Episode Grounding metrics (Precision, Recall, F1). The Hybrid approach achieves an F1 score of 41.02 (compared to ~32-34 for others), indicating a superior ability to identify the correct set of episodes required to answer a question.
• Exploration-Exploitation Trade-off: The results highlight a systematic trade-off. Parallel planners offer broad coverage but lack the depth for complex causal chains. Sequential planners excel at refinement but suffer from "reasoning drift," often getting stuck in repetitive loops or failing to explore alternative narrative paths. Hybrid planners mitigate these limitations by using parallel execution for exploration and sequential logic for dependency management.
• Multimodal Necessity: Replacing VideoRAG with TextRAG (using only subtitles) results in a significant drop in episode grounding metrics, confirming that visual information is essential for answering SAGAQA questions.
• Synthesis Bottleneck: While Hybrid planners improve retrieval accuracy, abstractive metrics (ROUGE, METEOR) remain relatively stagnant across all methods. This suggests that the primary bottleneck is not in retrieving relevant video segments, but in effectively synthesizing and aggregating this evidence into a coherent, high-quality answer.

Significance and Contributions
The paper claims three primary contributions:

1. SAGAQA Benchmark: It introduces a novel dataset that bridges the gap between short-term visual perception and full-episode narrative understanding, challenging models to reason over 20-hour windows with multi-hop dependencies.
2. Strategic Analysis: It systematically analyzes video planning strategies, formally distinguishing between Parallel, Sequential, and Hybrid planners, and demonstrating the efficacy of hybrid approaches in long-form reasoning.
3. Insight into Limitations: The study provides evidence that while retrieval mechanisms are improving, current models struggle with the downstream synthesis of evidence across extended narratives. It suggests that future research must move beyond single-trajectory reasoning toward structured, hybrid exploration strategies (e.g., Tree-of-Thought or graph-based planning) to handle complex, long-form video content.

The authors maintain a modest stance regarding generalization, noting that the dataset is built exclusively on soap operas, which have specific narrative structures (e.g., strong continuity, recurring characters) that may not generalize to other domains like documentaries or instructional videos. Furthermore, while 20 hours is considered long-form, the benchmark does not yet test ultra-long memory spanning the entire lifecycle of a TV series.