🤖 machine learning

NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative Understanding

This paper introduces NarrativeTrack, the first benchmark designed to evaluate multimodal large language models' narrative understanding through a structured Compositional Reasoning Progression framework that exposes their inability to robustly track entities across temporal dynamics, revealing a fundamental trade-off between perceptual grounding and temporal coherence.

Original authors: Hyeonjeong Ha, Jinjin Ge, Bo Feng, Kaixin Ma, Gargi Chakraborty

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hyeonjeong Ha, Jinjin Ge, Bo Feng, Kaixin Ma, Gargi Chakraborty

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a movie. A standard "smart" AI might look at a single frame and tell you, "That's a man in a red shirt holding a coffee cup." It's good at spotting things in a snapshot.

But Narrative Understanding is different. It's like watching the whole movie and keeping a mental diary of the characters. You need to know: Is that the same man in the red shirt from the beginning? Did he change his shirt when he went to the kitchen? Did he disappear for a while and then come back? Is that person in the blue jacket actually the same guy, or just someone who looks similar?

This paper introduces NARRATIVETRACK, a new "exam" designed to test if AI models can actually do this kind of storytelling, rather than just taking snapshots.

Here is a breakdown of the paper using simple analogies:

1. The Problem: The "Snapshot" AI

Current AI models are like amnesiac tourists. They are great at describing the view right in front of their eyes (a beautiful sunset, a specific car). But if you ask them, "What happened to the tourist who was wearing a hat 10 minutes ago?" or "Did that tourist change into a swimsuit later?", they often get confused. They might forget the person entirely, or worse, they might hallucinate (make things up) and say the tourist is still there when they left the room.

Existing tests for AI video understanding are too easy. They often ask questions that can be answered by looking at just one single frame.

Bad Test: "What color is the car?" (Answer: Look at one picture).
Narrative Test: "The car was red at the start, but the driver changed it to blue halfway through. What color is it at the end?" (Answer: You have to remember the whole story).

2. The Solution: The "Character Tracker" Pipeline

To create a fair test, the researchers built a robot assistant that watches videos and creates a "character sheet" for every person in the video. This happens automatically, without humans needing to label every second.

Think of this pipeline as a super-efficient film crew:

The Spotter (Detection): Scans every frame to find people.
The Stalker (Tracking): Follows those people across the video, making sure "Person A" in the first scene is the same "Person A" in the last scene, even if they walk behind a tree (occlusion) or change clothes.
The Biographer (Context): Writes down what the person is doing, what they are wearing, and where they are at every moment.

3. The Exam: "Compositional Reasoning Progression" (CRP)

The researchers didn't just ask random questions. They designed a test that gets harder in three specific levels, like climbing a ladder:

Level 1: The "Is He Still There?" Test (Entity Existence)
- The Challenge: Can the AI remember that a character existed, disappeared, and then came back?
- The Metaphor: It's like playing "Where's Waldo?" but over a long movie. If Waldo leaves the room and comes back, does the AI know it's the same Waldo, or does it think he vanished forever?
Level 2: The "Detective" Test (Entity Changes)
- The Challenge: Did the character change their action, outfit, or location?
- The Metaphor: Imagine a spy who starts in a suit, then changes into a tuxedo, then runs into a kitchen. The AI must track these changes. If the AI says, "He's still in the suit," it failed the test.
Level 3: The "Imposter" Test (Entity Ambiguity)
- The Challenge: There are two people who look very similar. Can the AI tell them apart?
- The Metaphor: It's like a case of mistaken identity. If two people are wearing identical red coats, can the AI tell which one is the main character and which one is a stranger? Most AIs get this wrong and mix them up.

4. The Results: The "Specialist" vs. The "Generalist"

The researchers tested many AI models on this new exam. The results revealed a funny trade-off, like a Swiss Army Knife vs. a Scalpel:

General-Purpose AIs (The Swiss Army Knife): These are smart, all-around models (like GPT-4o). They are great at describing what they see right now (perception). They rarely hallucinate. But, they are bad at remembering the timeline. They often forget who was where 30 seconds ago.
Video-Specialized AIs (The Scalpel): These models are trained specifically on videos. They are good at understanding time and sequence. But, they are prone to "hallucinations." They might confidently say, "Yes, the man changed his shirt!" even if he never did, because they are trying too hard to fit a story together.

The Big Discovery:
To truly understand a story, an AI needs both the sharp eyes of the Generalist (to see details) and the memory of the Specialist (to track time). Currently, no model has mastered both perfectly.

5. Why This Matters

This paper shows that simply making AI models bigger or feeding them more video frames doesn't solve the problem. The AI isn't just "forgetting"; it's fundamentally struggling to link a person's identity across time.

NARRATIVETRACK is the first tool that forces AI to stop guessing and start tracking. It's a diagnostic tool that tells us exactly where AI fails:

Do they lose the character when the camera cuts?
Do they confuse two similar-looking people?
Do they get confused if we ask the story in reverse order?

The Takeaway

Right now, AI is like a very observant but short-term memory human. It can describe a scene perfectly but can't follow a plot. NARRATIVETRACK is the first step toward teaching AI to be a true storyteller, capable of following the complex, shifting lives of characters from the opening scene to the credits.

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have advanced significantly in static image understanding and short-video reasoning, they struggle with narrative understanding in long-form videos. Current benchmarks often rely on:

Static cues: Questions answerable from a single frame.
Coarse semantics: Global summaries that ignore fine-grained details.
Short temporal windows: Clips with minimal scene variation.

The core gap is the inability of MLLMs to maintain coherent entity representations over time. Humans track who is doing what, when, and where across scene transitions, occlusions, and temporal gaps. Existing models fail to integrate fine-grained perceptual grounding (recognizing specific visual details) with temporal reasoning (tracking state changes and continuity), leading to hallucinations and identity drift.

2. Methodology

A. NARRATIVETRACK Benchmark

The authors introduce NARRATIVETRACK, the first benchmark specifically designed to evaluate narrative understanding through fine-grained entity-centric reasoning.

Scope: Focuses exclusively on human entities (the primary agents in narratives) to isolate temporal-perceptual grounding.
Data Source: Constructed from raw video sources (AVA, Video-MME, LVBench) with an average clip length of ~55 seconds.
Automated Pipeline: To ensure scalability and avoid manual annotation costs, the authors developed a fully automated pipeline:
1. Entity Detection: Uses an ensemble of Detectron2 and Owlv2 with spatial consensus (IoU $\ge$ 0.5) to merge detections and reduce redundancy.
2. Entity Tracking: Clusters bounding box embeddings using ReID models (OSNet) to identify main characters. Refines trajectories using face recognition and a majority-voting mechanism across multiple MLLMs to ensure identity consistency.
3. Contextual Recognition: Uses Gemini-2.5-Pro to extract temporal attributes (Action, Outfit, Scene) for each entity at every timestep, creating a structured Temporally Grounded Entity Representation ( $\tau_{e_i}$ ).

B. Compositional Reasoning Progression (CRP)

The benchmark evaluates models using a structured framework that increases complexity across three dimensions:

Entity Existence (Level 1): Tests basic temporal continuity. Can the model track an entity's appearance, disappearance, and reappearance?
Entity Changes (Level 2): Tests the integration of temporal continuity with perceptual grounding.
- Action Changes: Tracking dynamic events.
- Outfit Changes: Tracking visual state transitions.
- Scene Changes: Tracking spatial-contextual shifts.
Entity Ambiguity (Level 3): Tests fine-grained disambiguation. Can the model distinguish the target entity from visually similar entities (distractors) appearing in the same or different clips?

C. QA Generation

Questions are programmatically generated from the extracted entity metadata using templates aligned with CRP.

Formats: Binary, Multiple-Choice (MC), and Ordering (chronological sequence).
Distractors:
- Real Distractors: Attributes of other entities within the same video (tests discrimination).
- Synthetic Distractors: Attributes from different videos (tests hallucination/robustness).
Reasoning Patterns: Includes Forward (start $\to$ end), Backward (end $\to$ start), and Agnostic (mid-point $\to$ start/end) to test directional bias.

3. Key Contributions

First Entity-Centric Narrative Benchmark: NARRATIVETRACK shifts the evaluation paradigm from global scene understanding to bottom-up, entity-level tracking over time.
Automated Annotation Pipeline: A novel, unsupervised pipeline that generates high-quality, temporally grounded entity representations (bounding boxes, actions, outfits, scenes) without human intervention.
Compositional Reasoning Framework (CRP): A systematic diagnostic tool that isolates specific failure modes (temporal continuity vs. perceptual grounding vs. ambiguity).
Comprehensive Evaluation: Evaluation of 12 open-source models (General-Purpose and Video-Specialized) and 1 proprietary model (GPT-4o).

4. Experimental Results

Quantitative Performance

GPT-4o achieved the highest average accuracy (72.27% with 128 frames), significantly outperforming open-source models.
General-Purpose vs. Video-Specialized:
- Open-Source General-Purpose (OGP-MLLMs) (e.g., Qwen-2.5-VL-32B: 56.96%) outperformed Open-Source Video-Specialized (OVS-MLLMs) (e.g., Video-LLaMA2-72B: 49.70%).
- Trade-off Identified: OGP models excel at perceptual grounding (static visual cues) but fail at temporal consistency. OVS models capture temporal context better but frequently hallucinate entity details or identities under ambiguity.
Scaling: Increasing model size (e.g., 7B $\to$ 32B/72B) yields inconsistent gains. Larger models do not automatically solve entity tracking; in some cases (LLaVA-NeXT-34B), performance dropped compared to smaller variants.

Qualitative Findings & Failure Modes

Temporal Directional Bias: Models exhibit a strong forward bias. Performance drops significantly (up to 20.65% gap) when reasoning backward (end $\to$ start) or bidirectionally (agnostic). This mirrors the "Reversal Curse" in LLMs.
Frame Density: Increasing frame density beyond 20 frames degrades performance for most models, suggesting they cannot effectively filter redundant temporal information or maintain coherence with dense inputs.
Distractor Sensitivity: Video-specialized models struggle significantly with synthetic distractors, showing a higher tendency to hallucinate entity presence compared to general-purpose models.
Visual Grounding: Removing visual inputs reduces GPT-4o's accuracy to near-random levels, confirming the benchmark requires true multimodal reasoning, not just language priors.

5. Significance and Conclusion

NARRATIVETRACK reveals a fundamental trade-off in current MLLMs: the difficulty of simultaneously maintaining temporal integration (long-range continuity) and perceptual precision (fine-grained visual details).

Diagnosis: The benchmark provides the first systematic framework to diagnose where models fail—whether in tracking persistence, grounding state changes, or disambiguating similar entities.
Future Directions: The authors suggest that future progress requires:
- Bidirectional temporal modeling to overcome forward bias.
- Entity-centric training objectives to enforce identity consistency.
- Architectural innovations that better integrate memory mechanisms with fine-grained visual perception.

In summary, NARRATIVETRACK demonstrates that true narrative understanding is a compositional capability that emerges only from the successful integration of temporal reasoning and perceptual grounding, a capability currently lacking in state-of-the-art MLLMs.