🤖 AI

MultivationBench: A Benchmark for Multimodal Sequential Motivation Reasoning

The paper introduces MultivationBench, a new benchmark based on psychological frameworks that evaluates the ability of multimodal large language models to perform sequential motivation reasoning in story-driven visual narratives, revealing that current models struggle to maintain consistent reasoning across dynamic contexts despite their static recognition capabilities.

Original authors: Kawai Chung, Chunkit Chan, Yauwai Yim, Yuxuan Liu, Haochen Shi, Weiqi Wang, Qing Zong, Tianshi Zheng, Yixuan Fu, Kai Chung Wong, Hao Liang, Yifan Gao, Xi Yang, Janet Hui-wen Hsiao, Yangqiu Song

Published 2026-07-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kawai Chung, Chunkit Chan, Yauwai Yim, Yuxuan Liu, Haochen Shi, Weiqi Wang, Qing Zong, Tianshi Zheng, Yixuan Fu, Kai Chung Wong, Hao Liang, Yifan Gao, Xi Yang, Janet Hui-wen Hsiao, Yangqiu Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a movie, but instead of just seeing what happens, you are trying to guess why the characters are doing it. This is the heart of "social intelligence," a superpower humans have that lets us understand the hidden feelings and goals behind our actions. Scientists call this "motivation reasoning." It's not just about seeing someone pick up a phone; it's about figuring out if they are calling for help, checking the time, or looking at a photo of a lost loved one. Usually, we don't guess this from a single frozen picture. We watch the story unfold, gathering clues over time. If a character looks sad in scene one, then sees a photo in scene two, our guess about their motivation changes. This paper dives into whether the newest, super-smart computer brains (called Multimodal Large Language Models) can do this same kind of "story detective work." While these computers are great at looking at a picture and describing it, or reading a story and answering questions, this research asks a harder question: Can they keep track of a character's changing feelings as a whole story plays out, just like a human does?

The researchers behind this study, led by a team from the Hong Kong University of Science and Technology and Amazon, decided to build a giant test to find out. They created something called MultivationBench. Think of it as a massive library of 1,000 short visual stories, like a mix of movie clips and social media posts, containing over 4,000 specific moments where a character does something. But this isn't just a test of "what happened"; it's a test of "why it happened." To make the test fair and scientific, the team used two famous psychological rulebooks: Maslow's Hierarchy of Needs (which sorts human desires from basic survival like food and safety to higher goals like love and self-improvement) and Reiss's 16 Basic Desires (which looks at specific personal drives like curiosity, honor, or family).

The team asked the computer models to watch these stories frame-by-frame. At every step, the models had to guess the character's motivation based only on what they had seen so far. The tricky part? As the story continued, new information would appear that completely changed the reason for an earlier action. For example, a character might reach for a phone, and at first, it looks like they are just checking a message (a "Cognitive" need). But later, the story reveals they are looking at a picture of their family while feeling lonely at a party, which means the real reason was actually about "Love and Belonging." The computer had to be smart enough to say, "Wait, I was wrong before; now I know the real reason."

The results were a bit of a wake-up call for the tech world. The paper found that while these advanced AI models are decent at guessing motivations in short, simple stories, they really struggle when the story gets longer. Out of the 1,000 stories tested, the models could rarely keep their reasoning consistent from start to finish. In fact, when asked to get every single motivation right in a whole story, the best models only succeeded less than 1% of the time. The study suggests that these AI brains are like students who are great at memorizing facts but terrible at understanding the flow of a plot. They tend to get stuck on the first thing they see and fail to update their thinking when new evidence arrives. They often "hallucinate" reasons that sound plausible but aren't supported by the story, or they miss the deeper, more complex emotional reasons that require connecting dots across the entire narrative.

The researchers also compared the AI to actual humans. The humans, who were graduate students, were significantly better at the task, especially when the stories were long and complex. This gap shows that while AI is getting better at seeing and reading, it still lacks the "social brain" needed to understand how our motivations shift and evolve over time. The paper concludes that we are still far from having computers that can truly understand the messy, changing reasons behind human behavior in a story. It's a reminder that understanding why people do what they do is a lot harder than just knowing what they did.

Technical Summary: MultivationBench

Problem Statement

Multimodal Large Language Models (MLLMs) have demonstrated significant potential in social intelligence, yet their capacity for sequential motivation reasoning remains under-explored. Existing evaluation benchmarks predominantly rely on static text or isolated visual snapshots. These static settings fail to capture the cumulative nature of real-world behavioral drivers, where motivation is a latent construct inferred from evolving behavior and context over time. Current benchmarks often reward locally plausible guesses but do not assess whether a model can update its interpretation of a character's motivation as a narrative unfolds and new evidence accumulates. Consequently, there is a lack of rigorous evaluation tools to determine if MLLMs can perform sequential motivation reasoning based on accumulated multimodal context.

Methodology

Benchmark Construction: MultivationBench

To address this gap, the authors introduce MultivationBench, a large-scale benchmark designed to stress-test multimodal sequential motivation reasoning within story-driven visual narratives. The construction follows four core principles: psychological grounding, sequential human-centric dynamics, long-term consistency evaluation, and data integrity.

Data Sources: The dataset aggregates 1,000 unique visual narratives from three sources: SSID (social-media style), StoryReasoning, and MovieBench (cinematic narratives), covering diverse narrative styles and temporal complexities.
Psychological Frameworks: The benchmark grounds motivation labels in two complementary frameworks:
1. Maslow's Expanded Hierarchy of Needs (8 levels): Captures broad, universal categories of human needs (e.g., Physiological, Safety, Love and Belonging, Self-Actualization).
2. Reiss's 16 Basic Desires: Captures finer-grained, individual differences in motivational content (e.g., Family, Curiosity, Power, Status).
Task Formulation: For each visually grounded character behavior within a story, the benchmark formulates four tasks:
1. Maslow Definition: Classifying motivation based on standard theory labels.
2. Maslow Practical Motivation: Choosing context-specific options derived from Maslow's theory.
3. Reiss Definition: Classifying motivation based on Reiss's 16 desires.
4. Reiss Practical Motivation: Choosing context-specific options derived from Reiss's theory.
- All tasks are treated as multi-label prediction problems, as a single behavior may be supported by multiple valid motivations.
Pipeline: An AI-human pipeline is employed. High-performing MLLMs extract behavior chains and generate candidate motivations, which are then refined through automated cross-review and validation. Finally, human annotators (graduate students) verify visual grounding, psychological salience, and logical consistency, achieving high inter-annotator agreement (Fleiss's $\kappa$ ranging from 74.87% to 80.99%).

Experimental Setup

The authors evaluated eight state-of-the-art MLLMs (including closed-source models like Grok-4.1, Gemini-3, and o4-mini, and open-source models like Llama-4 and Phi families) in a zero-shot setting.

Settings: Evaluations were conducted under three modalities: Full Multimodal, Text-Only, and Image-Only.
Metrics:
- Exact Match (EM): Strict set-level accuracy requiring the predicted option set to exactly match the ground truth.
- Example-based F1: Measures instance-level partial agreement, rewarding overlap while penalizing false positives and negatives.
Consistency: A story-level consistency metric was introduced to measure the percentage of stories where a model answers every question correctly, testing psychological continuity across the entire narrative.

Key Results

Overall Performance

MultivationBench presents a significant challenge for all evaluated models. While models show varying strengths (e.g., Grok-4.1-Fast achieved the highest F1 of 55.0%, while Phi-4-multimodal-instruct achieved the highest EM of 39.2%), no model approaches human performance. Human annotators achieved an EM of 78.6% and F1 of 87.6% on Maslow Practical Motivation tasks, compared to the best model's 30.7% EM and 46.0% F1 on Reiss tasks.

Modality Analysis

Text vs. Multimodal: Text-only inputs preserved a relatively large portion of the F1 score but caused a sharp decline in EM accuracy. This suggests that while text narratives provide enough information for partial motivation recovery, visual cues are critical for refining and revising inferences to identify the exact ground-truth set.
Image-Only: Performance was consistently lower than multimodal settings, highlighting the necessity of combining visual and textual context.

Narrative Length and Consistency

Length Impact: Models generally improved from short (2–5 images) to medium (6–9 images) narratives but declined in performance on long narratives (10+ images). This indicates that current models struggle to maintain correct motivation reasoning as context accumulates beyond a moderate horizon.
Story-Level Consistency: The results reveal a critical disconnect: even the best-performing model achieved a full story-level consistency score of only 0.80%. This indicates that models rarely track character motivations correctly throughout an entire story, often failing to revise earlier interpretations when new evidence emerges.

Error Analysis

Over-Interpretation: Models exhibit a strong bias toward Over-Interpretation (False Positives), hallucinating motivations not supported by the ground truth. For instance, Nemotron-Nano-12B-v2-vl committed over 31,000 False Positives compared to ~7,000 False Negatives. This suggests models struggle to discard earlier, less relevant cues as new evidence unfolds.
Granularity: Performance drops significantly when moving from the coarse-grained Maslow taxonomy to the fine-grained Reiss taxonomy. Models perform better on labels inferable from single salient cues (e.g., Eating, Safety) but struggle with abstract motivations requiring accumulated context (e.g., Self-Actualization, Independence).

Significance and Claims

The paper claims that MultivationBench is the first human-annotated benchmark specifically designed to evaluate multimodal sequential motivation reasoning in story-driven visual narratives. Its significance lies in:

Exposing a Critical Gap: It reveals that current MLLMs possess strong static recognition capabilities but lack the dynamic reasoning required for human-like social understanding, specifically the ability to update motivation inferences over time.
Psychological Grounding: By utilizing Maslow and Reiss frameworks, it enables evaluation at both universal (coarse) and individual (fine) levels, moving beyond surface-level intent detection.
Benchmarking Consistency: It highlights that maintaining psychological continuity across a full narrative is a major unsolved challenge, with models failing to track motivations consistently even in short stories.

The authors conclude that while MLLMs can identify locally plausible motivations, they struggle with the sequential, cumulative reasoning necessary for deep social intelligence. The benchmark serves as a stress-test for these capabilities, indicating that future progress requires models that can effectively integrate and revise interpretations across long multimodal contexts.