MovieTeller: Tool-augmented Movie Synopsis with ID Consistent Progressive Abstraction

MovieTeller is a training-free, tool-augmented framework that generates coherent and factually accurate movie synopses by leveraging external face recognition tools for ID-consistent character grounding and a progressive abstraction pipeline to overcome the context limitations of existing Vision-Language Models.

The Gist

Imagine you are trying to tell a friend the entire plot of a 3-hour movie, but you've only seen a few random screenshots of it. If you try to describe it from memory alone, you might say, "A guy in a suit fights another guy," and then later, "A man in a blue shirt saves the day." Your friend gets confused: Wait, is the guy in the suit the same as the guy in the blue shirt? Who is the villain? Why did the plot jump like that?

This is exactly the problem current AI faces when trying to summarize long movies. It's great at describing a single picture, but it gets lost in a long story. It forgets who is who and loses the thread of the plot.

MovieTeller is a new "smart assistant" designed to fix this. Think of it not as a single super-brain, but as a team of specialists working together to write a perfect movie review.

Here is how MovieTeller works, broken down into three simple steps:

1. The "Detective" Tool (Fact-Checking)

Current AI models are like artists who are good at painting but bad at remembering names. If you show them a picture of a famous actor, they might just say, "Here is a man."

MovieTeller brings in a specialist detective (a face recognition tool) before the AI starts writing.

• The Analogy: Imagine you are writing a story about a party. Instead of guessing who the guests are, you have a security guard at the door with a list of names and photos.
• How it works: Before the AI describes a scene, this "detective" scans the image, finds the faces, and says, "That's not just a man; that is Jack, the hero. That's Sarah, the villain." It even draws a box around them to prove it.
• The Result: The AI is forced to write, "Jack and Sarah argue," instead of "A man and a woman argue." This stops the AI from getting confused about who is who.

2. The "Summarizer" Pipeline (The Ladder of Abstraction)

Trying to summarize a whole movie in one go is like trying to drink a whole ocean in one sip. The AI gets overwhelmed and spills the story.

MovieTeller uses a step-by-step ladder approach, called "Progressive Abstraction."

• The Analogy: Imagine you are writing a biography of a person's life.
  1. First, you write a short paragraph about what happened on Monday.
  2. Then, you write a paragraph about what happened on Tuesday.
  3. Next, you combine Monday and Tuesday into a summary of the first week.
  4. Finally, you combine all the weeks into the whole life story.
• How it works: MovieTeller breaks the movie into small "scenes," summarizes those into "chapters," and then combines the chapters into the final "movie synopsis." This keeps the story logical and prevents the AI from forgetting the beginning by the time it reaches the end.

3. The "No-Training" Magic (Plug-and-Play)

Usually, to make an AI smarter, you have to feed it thousands of hours of movies and teach it from scratch (like going to school for four years). This is expensive and slow.

MovieTeller is training-free.

• The Analogy: Instead of teaching a new employee how to do a job, you just give them a toolkit and a manual.
• How it works: MovieTeller takes existing, powerful AI models (the "generalists") and simply connects them to the "detective" tool and the "summarizer" steps. It's like plugging a high-quality lens into a camera; you don't need to rebuild the camera to get better photos.

Why Does This Matter?

The paper tested MovieTeller on 100 different movies. The results were impressive:

• Accuracy: It got the character names right almost 100% of the time, whereas other AIs got them wrong half the time.
• Story Flow: The summaries it wrote felt like a real story, not a jumbled list of events.
• Human Preference: When humans read the summaries, they preferred MovieTeller's version over 60% of the time.

In short: MovieTeller is like hiring a director (the AI) who is assisted by a script supervisor (the face detector) and an editor (the summarizer). Together, they ensure the final story is accurate, the characters are consistent, and the plot makes sense, all without needing to spend years teaching the AI how to do it.

Technical Summary

1. Problem Definition

The paper addresses the critical challenges in automated long-form video summarization, specifically for movies and TV series. While Vision-Language Models (VLMs) excel at captioning single images, they fail when applied to full-length narratives due to two primary bottlenecks:

• Lack of ID-Consistent Character Identification: General-purpose VLMs struggle to track specific characters across long durations. They often describe the same protagonist inconsistently (e.g., "a man" in one scene, "a person" in another) or hallucinate identities, breaking narrative continuity.
• Fractured Narrative Coherence: The quadratic complexity ($O(n^2)$) of Transformer self-attention mechanisms makes processing hour-long video sequences computationally prohibitive. Simple frame sampling or concatenation often results in fragmented summaries that lack a unified story arc.

Existing solutions often require costly model fine-tuning or fail to provide factual grounding, leading to hallucinated descriptions.

2. Methodology: MovieTeller Framework

The authors propose MovieTeller, a training-free, modular framework that combines tool-augmentation with progressive abstraction. Instead of training a monolithic model, it orchestrates off-the-shelf models in a plug-and-play pipeline.

The framework operates in three key stages:

A. Scene Segmentation and Keyframe Extraction

• Segmentation: The raw video is partitioned into semantically coherent scenes ($S$) using PySceneDetect based on visual content changes.
• Quality Control: A Keyframe Quality Gate filters out non-informative transitional frames (e.g., black screens) by checking mean brightness and pixel standard deviation, ensuring only visually rich keyframes are processed.

B. Factual-Grounded Scene Description Generation (Tool-Augmentation)

This is the core innovation for solving ID consistency:

1. External Tool Invocation: A specialized face recognition model (powered by InsightFace with ArcFace) acts as an external "tool."
2. Factual Grounding: The system pre-constructs a face database ($B$) using cast metadata (actor names and reference images). For every detected face in a keyframe, the tool calculates the closest match in the database to determine the character's identity ($ID$) and bounding box ($BBox$).
3. Prompt Engineering: These factual groundings (Name + BBox) are injected into the VLM's prompt. This transforms the VLM's task from open-ended generation to fact-constrained narration, forcing the model to anchor descriptions to verified identities.
   • Prompt Example: "The actor '{Name}' is located within the bounding box {BBox}..."

C. Progressive Abstraction

To handle long-context limitations and ensure narrative flow, the process is hierarchical:

1. Scene to Chapter: Individual scene descriptions are grouped into sequential "chapters." The VLM summarizes each chapter, distilling plot progression and character motivations while maintaining the grounded identities.
2. Chapter to Synopsis: All chapter summaries are concatenated and fed into the VLM with a prompt instructing it to adopt a screenwriter persona. This final stage synthesizes the inputs into a cohesive, holistic movie synopsis that captures the global narrative arc from exposition to resolution.

3. Key Contributions

• Training-Free Framework: MovieTeller achieves state-of-the-art results without fine-tuning the underlying VLM, leveraging a "plug-and-play" architecture that integrates specialist tools with generalist models.
• ID-Consistent Generation: By using face recognition as an external tool to provide Factual Groundings (names and bounding boxes), the framework effectively eliminates character hallucinations and ensures consistent identity tracking throughout the synopsis.
• Progressive Abstraction Pipeline: The multi-stage approach (Scene $\to$ Chapter $\to$ Synopsis) mitigates context length constraints and mirrors human cognitive processes for understanding complex narratives.
• Comprehensive Validation: The method was validated on a diverse dataset of 100 full-length films (over 10,000 minutes), covering various genres and decades.

4. Experimental Results

The authors evaluated MovieTeller against baselines (No-Hint) and ablation studies (Name-Only Hint) using three evaluation strategies:

• Automatic Metrics (BERTScore): MovieTeller achieved the highest semantic similarity scores (e.g., 0.638 vs. 0.612 for the baseline on Qwen2.5-VL), indicating better alignment with reference synopses.
• LLM-as-a-Judge: Using Gemini 2.5 Flash to score on Faithfulness, ID Consistency, Coherence, and Conciseness:
  • ID Consistency: MovieTeller showed a massive 117% relative gain (3.80 vs. 1.75) compared to the baseline.
  • Final Score: Achieved a 39% gain in the final aggregated score (3.02 vs. 2.17) on the InternVL3 model.
• Human Evaluation: In a 3-way forced-choice task, human evaluators preferred MovieTeller in 60% to 62% of cases, significantly outperforming the Name-Only ablation (32-42%) and the baseline (4-6%).
• Qualitative Analysis: Case studies (e.g., The Bullet Vanishes) demonstrated that while baselines produced generic or confused narratives, MovieTeller successfully identified specific characters (e.g., "Director Jin," "Song Donglu") and captured complex plot dynamics and tensions.

5. Significance and Future Work

Significance:
MovieTeller demonstrates that tool-augmentation is a highly effective strategy for overcoming the limitations of current VLMs in long-context tasks. It proves that integrating specialized modules (like face recognition) to provide factual grounding is more efficient and effective than attempting to train a single massive model to learn these capabilities from scratch. This approach offers a scalable, cost-effective solution for content indexing, personalized recommendations, and media archiving.

Future Directions:

• Audio Integration: Incorporating speaker diarization to link dialogue to visually identified characters for richer, dialogue-aware summaries.
• Dynamic Tool Use: Developing protocols where the VLM learns to contextually decide when to invoke tools.
• Domain Expansion: Adapting the modular framework for other long-form video domains, such as sports analysis and documentary summarization.