💻 computer science

Narrative Aligned Long Form Video Question Answering

This paper introduces NA-VQA, a new benchmark comprising 88 full-length movies and 4,400 question-answer pairs designed to evaluate narrative reasoning in long-form videos, alongside Video-NaRA, a framework that improves long-range reasoning by constructing and retrieving structured event-level chains.

Original authors: Rahul Jain, Keval Doshi, Burak Uzkent, Garin Kessler

Published 2026-03-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rahul Jain, Keval Doshi, Burak Uzkent, Garin Kessler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery, but instead of a single crime scene, the clues are scattered across a 2-hour movie. Some clues are in the first 10 minutes, some are in the middle, and the final piece of the puzzle is in the very last scene.

This is exactly the problem researchers at Purdue University and Amazon are tackling in their new paper. They are trying to teach AI to watch a whole movie and understand the story, not just the individual scenes.

Here is a breakdown of their work using simple analogies:

1. The Problem: The "Amnesiac" AI

Current AI models (like the ones that can chat with you or describe images) are great at looking at a single photo or a short 10-second clip. But when you give them a full movie, they tend to get "amnesiac."

The Analogy: Imagine reading a book where every time you turn the page, you forget the first 50 pages. If someone asks, "Why did the hero betray his friend in the final chapter?" the AI might look at the final chapter, see a betrayal, but have no idea why it happened because it forgot the argument that happened 30 minutes ago.
The Reality: Most AI benchmarks only test short videos or ask simple questions like "What color was the car?" They don't test if the AI can connect a character's motivation in Scene 1 to their action in Scene 50.

2. The New Test: NA-VQA (The "Movie Detective" Exam)

To fix this, the authors created a new benchmark called NA-VQA (Narrative-Aligned Video Question Answering).

What it is: They took 88 full-length movies and created 4,400 questions.
The Twist: These aren't simple questions. They are "multi-hop" mysteries.
- Example Question: "Why did the character refuse the gift in the final scene?"
- The Catch: To answer this, the AI must remember a conversation from the beginning of the movie, a secret revealed in the middle, and a promise made in the third act.
The Categories: They tested the AI on seven types of thinking, like understanding Causal links (A caused B), Social dynamics (who likes whom), and Hypothetical scenarios (what if X happened?).
The Evidence: They labeled the clues as Short (clues close together), Medium, or Far (clues separated by huge chunks of time). The AI struggled the most with "Far" clues.

3. The Solution: Video-NaRA (The "Story Librarian")

The authors realized that trying to force an AI to remember every single frame of a 2-hour movie is like asking a human to memorize a library by reading every book cover-to-cover without stopping. It's inefficient and leads to errors.

Instead, they built Video-NaRA, which acts like a Story Librarian.

How it works:
1. The Librarian (Narrative Memory): Instead of storing raw video frames, the AI first watches the movie and writes a "summary card" for every major event. It groups these cards into "Story Arcs" (e.g., "The Hero's Journey," "The Villain's Plan").
2. The Retrieval: When a question is asked, the AI doesn't re-watch the whole movie. It asks the Librarian: "Do you have any story cards about the hero's betrayal?"
3. The Answer: The Librarian pulls out the specific cards from the beginning, middle, and end that form a chain, and hands them to the AI to write the answer.
The Metaphor:
- Old Way: Trying to find a specific sentence in a 500-page book by reading every single word from start to finish, hoping you don't get tired and forget the beginning.
- New Way (Video-NaRA): Using the Table of Contents and Chapter Summaries to jump straight to the relevant pages, then reading just those pages to find the answer.

4. The Results

When they tested this new "Librarian" system against the best AI models available:

The Struggle: Even the smartest current AIs failed miserably on questions requiring "Far" evidence (clues far apart in time). They kept guessing or making up stories because they couldn't connect the dots.
The Success: Video-NaRA improved performance by about 3%. While that sounds small, in the world of AI, it's a huge leap. It proved that if you organize the video into a story structure first, the AI can actually reason about long-term events.

The Big Takeaway

This paper teaches us that to understand a long story, you can't just look at the pictures; you need to understand the plot.

Current AI is like a tourist who takes a million photos of a city but has no map. NA-VQA is the map, and Video-NaRA is the guide that teaches the tourist how to use that map to connect the dots between the Eiffel Tower (Scene 1) and the Louvre (Scene 50) to understand the whole journey.

The researchers are releasing their dataset to the public, hoping to help other scientists build AIs that can finally watch a whole movie and say, "Ah, I see what happened, and here is exactly why."

1. Problem Statement

Current Multimodal Large Language Models (MLLMs) have shown progress in understanding long-form videos, but existing benchmarks and models suffer from significant limitations:

Lack of Narrative Reasoning: Most benchmarks rely on localized cues or short videos (≤1 hour) and fail to test the ability to track intentions, connect distant events, and reconstruct causal chains across an entire movie (2+ hours).
Superficial Patterns: Existing datasets often use multiple-choice formats, allowing models to exploit answer priors or surface-level patterns without genuine comprehension.
Temporal Disconnection: State-of-the-art models struggle to link temporally distant events (e.g., events separated by hours of video), failing to establish causal links or maintain narrative coherence.
Data Scarcity: There is a lack of benchmarks with open-ended questions grounded in multiple, temporally dispersed evidence spans across full-length movies.

2. Methodology

The paper proposes two main components: a new benchmark dataset (NA-VQA) and a novel framework for reasoning (Video-NaRA).

A. The NA-VQA Dataset

Scale & Scope: Comprises 88 full-length movies (diverse genres) and 4,442 open-ended QA pairs.
Evidence Grounding: Each question is grounded in multiple evidence spans labeled by temporal distance: Short, Medium, and Far. This forces models to retrieve information scattered across the timeline.
Reasoning Taxonomy: Questions are categorized into 7 reasoning types:
1. Causal (Cause-and-effect)
2. Narrative (Plot progression)
3. Character-centric (Motivations/traits)
4. Thematic (Underlying themes)
5. Goal-based (Character objectives)
6. Social (Interactions)
7. Hypothetical (Counterfactuals)
Construction Pipeline:
1. Event Extraction: Uses LLMs (Claude Sonnet 4) to extract structured events from shot-level descriptions (Who does What to Whom/Why/Where).
2. QA Generation: Generates multi-hop reasoning questions based on these events.
3. Validation & Refinement: An automated pipeline (QA Validator and Refiner) ensures answers rely strictly on visual evidence (removing character names/textual identifiers) and verifies completeness, minimality, and faithfulness of the evidence.

B. The Video-NaRA Framework

To address the limitations of standard MLLMs, the authors propose Video-NaRA (Narrative Reasoning and Alignment), a memory-augmented architecture that mimics human narrative construction.

Narrative Memory Construction:
- Instead of processing frames uniformly, the model uses an MLLM (Qwen2.5-VL) to analyze clips and assign them to narrative slots based on temporal coherence and semantic similarity (shared characters, goals, and context).
- This creates a Narrative Memory Bank ( $M$ ) where clips are grouped into coherent storylines rather than a flat sequence.
Visual Feature Extraction:
- To ensure stability, frame-level visual features are extracted using a pretrained encoder (CLIP) and aggregated to form clip embeddings.
Retrieval Module:
- Given a query, the system computes relevance scores at two levels:
  - Clip-level: Cosine similarity between the query and frame embeddings.
  - Narrative-level: Aggregates clip scores within a narrative slot using a max-boosted weighted mean.
- The final relevance score combines clip relevance with the narrative slot's importance, allowing the model to retrieve temporally coherent evidence chains rather than isolated frames.
Reasoning:
- The retrieved top- $k$ clips (narrative chains) are passed to a fine-tuned reasoner (Qwen2.5-VL) to generate the final answer using Chain-of-Thought (CoT).

3. Key Contributions

NA-VQA Benchmark: A comprehensive dataset of 88 full movies with 4.4K open-ended QA pairs, explicitly designed to test long-range dependencies and narrative reasoning across 7 distinct categories.
Video-NaRA Architecture: A novel memory-based framework that constructs event-level narrative chains and stores them for retrieval, moving beyond uniform frame processing.
Extensive Benchmarking: A rigorous evaluation of state-of-the-art MLLMs (both open-source and proprietary) revealing systematic failures in "Far" evidence reasoning.
Open Release: Commitment to releasing the full NA-VQA dataset upon publication.

4. Results

Baseline Performance: State-of-the-art MLLMs (including GPT-4, Claude Sonnet 4.5, and various open-source VLMs) consistently fail on questions requiring Far evidence.
- Observation: Models struggle to establish causal links between events separated by extended durations.
- Proprietary Models: Claude Sonnet 4.5 achieved the highest average score (~62%), but still showed gaps in long-range reasoning.
Video-NaRA Performance:
- Video-NaRA outperformed all baseline models in the zero-shot setting.
- When fine-tuned on NA-VQA, Video-NaRA achieved the strongest overall performance, showing relative improvements of up to 3% over baselines.
- Ablation Studies: Removing the Narrative Memory Bank caused a significant drop in performance, particularly in the "Far" evidence category (Evidence score dropped from 35.20 to 32.70), proving that narrative grouping is crucial for long-range dependencies.
Reasoning Categories: Video-NaRA demonstrated stable performance across all 7 reasoning types, whereas baseline models showed high variance (e.g., performing well on Character reasoning but poorly on Causal reasoning).

5. Significance

Paradigm Shift: The paper argues that the bottleneck in long-video understanding is not just context length, but the lack of explicit narrative modeling. Models need to organize information into causal chains rather than processing frames sequentially.
Evaluation Standard: NA-VQA sets a new standard for evaluating "deep" temporal reasoning, moving beyond simple fact retrieval to complex narrative reconstruction.
Future Direction: The success of Video-NaRA suggests that memory-augmented, narrative-centric architectures are the necessary path forward for handling very long-form, complex video content, paving the way for more sophisticated multimodal systems.