GraphThinker: Reinforcing Video Reasoning with Event Graph Thinking

This paper introduces GraphThinker, a reinforcement finetuning method that constructs explicit event-based video scene graphs and incorporates visual attention rewards to enhance causal understanding and reduce hallucinations in video reasoning.

The Gist

🎬 The Problem: The "Daydreaming" Movie Critic

Imagine you have a very smart movie critic (an AI) who can watch any video and answer questions about it. This critic is great at describing what they see, but they have a bad habit: they daydream.

When you ask, "What happened first, the man jumping into the pool or the drone flying?" the critic might guess the wrong order just because it sounds logical in a story, even if the video shows the opposite. They might say, "Oh, people usually fly drones before jumping," ignoring the actual visual evidence. In the AI world, this is called hallucination. They are confident, but they are wrong because they aren't looking closely enough at the specific timeline of events.

🧠 The Solution: GraphThinker (The "Director's Cut" Editor)

The researchers created a new system called GraphThinker. Think of it as giving the movie critic a Director's Cut of the video before they answer your question. Instead of just watching the movie and guessing, the system forces the AI to break the video down into a structured "map" of events first.

Here is how it works, step-by-step:

1. The "Scene Graph" (The Movie Script)

Usually, AI watches a video and sees a blur of pixels. GraphThinker forces the AI to pause and write a script for the video.

• The Analogy: Imagine the video is a chaotic movie set. GraphThinker is the Script Supervisor.
• What it does: It breaks the video into small chunks (like 5-second clips). For each chunk, it writes a tiny story: "At 0:00, a man is wearing a red vest. At 0:05, he picks up a soap bottle."
• The Magic: It connects these tiny stories into a graph (a network of dots and lines). This graph explicitly shows: "Event A happened, then Event B happened." It creates a rigid timeline so the AI can't get confused about the order of things.

2. The "Self-Correction" Loop (The Editor)

The AI doesn't just write the script once; it checks its own work.

• The Analogy: Imagine the Script Supervisor writes a draft, then reads it again to see if it contradicts the actual footage.
• What it does: The AI generates a "coarse" (rough) summary and a "fine" (detailed) summary. It compares them. If the rough summary says "The man is swimming" but the detailed one says "The man is standing on the dock," the system catches the error and fixes the script before the AI tries to answer your question. This stops the AI from making things up.

3. The "Visual Attention" Reward (The Spotlight)

This is the most clever part. The researchers used a training method called Reinforcement Learning (like training a dog with treats).

• The Analogy: Imagine the AI is a student taking a test. If the student just memorizes the textbook (the text script) and ignores the actual video, they get a bad grade.
• The Reward: The system gives the AI a "treat" (a reward score) only if it proves it is looking at the video pixels while answering. It forces the AI to say, "I know the man jumped after the drone flew because I saw the drone at 5 seconds and the jump at 10 seconds in the video."
• The Result: The AI learns to stop daydreaming and start looking.

🏆 Why This Matters

Before GraphThinker, AI models were like actors who memorized lines but forgot the plot. They could talk fluently but got the timeline wrong.

GraphThinker changes the game by:

1. Forcing Structure: Making the AI build a map of events before speaking.
2. Checking Facts: Having the AI verify its own story against the video.
3. Rewarding Focus: Giving the AI a bonus for actually looking at the video, not just guessing based on common sense.

🚀 The Bottom Line

GraphThinker is like giving a movie critic a highlight reel and a timeline before they write their review. It stops them from making up facts and ensures they tell you exactly what happened and when it happened, based on what is actually on the screen. This makes AI much more reliable for things like medical video analysis, self-driving cars, or helping people with disabilities understand their surroundings.

Technical Summary

1. Problem Statement

Video reasoning requires understanding complex causal relationships and temporal dependencies between events within a video. Current Multimodal Large Language Models (MLLMs) struggle with this task due to two primary issues:

• Implicit Modeling: Existing models often rely on dense captions or video summaries that lack explicit structural representation of event relations. They infer relationships through token correlations rather than explicit causal structures.
• Hallucinations: Without explicit grounding in visual evidence and structured event logic, MLLMs frequently suffer from "temporal hallucinations" (incorrect ordering of events) and "action hallucinations" (inventing events that didn't happen). This leads to inconsistent reasoning and poor performance in tasks requiring precise temporal localization.

2. Methodology: GraphThinker

The authors propose GraphThinker, a reinforcement fine-tuning (RFT) framework designed to reduce hallucinations by integrating explicit Event-based Video Scene Graphs (EVSG) into the reasoning process. The method consists of two main stages:

A. Event-based Video Scene Graph (EVSG) Construction

Instead of relying on manual annotations, GraphThinker uses a self-generate and self-refine pipeline to construct structured graphs:

1. Multi-grained Dense Captioning: An MLLM generates dense captions at three levels of granularity (coarse, middle, and fine) for the video. This cross-level comparison helps detect and suppress inconsistencies early.
2. Graph Generation: The model extracts key object interactions from the captions to form event subgraphs. Each subgraph contains:
   • Intra-event relations: Triplets in the format <subject, relation, object> describing interactions within a specific time segment.
   • Temporal boundaries: Start and end timestamps for each event.
3. Graph Refinement: The initial graph is refined using the coarse and fine-grained captions as complementary evidence. The model verifies triplets to ensure they are visually grounded and temporally consistent, removing hallucinated relations and enforcing causal logic (e.g., prerequisite events must precede consequent ones).
4. Inter-event Linking: Subgraphs are connected via timestamp-based edges to model the global temporal sequence of the video.

B. Event Graph-based Reinforcement Fine-Tuning (GRPO)

The constructed EVSG is integrated into the Group Relative Policy Optimization (GRPO) framework to guide the MLLM's reasoning. The training objective is shaped by a composite reward function:

1. Accuracy Reward ($r_{acc}$): Combines temporal Intersection over Union (IoU) with semantic similarity to the ground truth.
2. Format Reward ($r_{form}$): Enforces a structured output format where reasoning is enclosed in <thought> tags and the final answer in <answer> tags, ensuring interpretability.
3. Visual Attention Reward ($r_{attn}$): A novel component that measures the model's attention distribution. It encourages the model to allocate higher attention scores to video tokens (visual evidence) relative to graph tokens (textual abstractions). This prevents the model from relying solely on the generated text graph and forces it to "look" at the video to verify the reasoning.

3. Key Contributions

• Explicit Event Modeling: The paper identifies the lack of explicit event-level relational structure in current MLLMs as a root cause of hallucinations. It proposes the EVSG to explicitly model both intra-event (semantic) and inter-event (temporal/causal) relations.
• Self-Supervised Graph Construction: Introduces a method to generate and refine EVSGs automatically without human annotations, using multi-grained captioning to ensure quality.
• Visual Attention Reward: Proposes a specific reward mechanism within RL that incentivizes the model to balance textual graph reasoning with direct visual evidence, effectively mitigating "visual drift."
• Unified Framework: Combines structured textual graphs with reinforcement learning to achieve visually grounded and temporally consistent reasoning.

4. Experimental Results

The method was evaluated on two benchmarks: RexTime (event causal reasoning and temporal grounding) and VidHalluc (video hallucination evaluation).

• RexTime Performance:
  • GraphThinker achieved state-of-the-art (SOTA) results among open-source models.
  • It significantly outperformed the baseline (Qwen2.5-VL) with improvements of +11.74% in mIoU and +8.86% in Accuracy@IoU≥0.5.
  • It surpassed strong baselines like TimeSearch and VITAL, demonstrating superior temporal consistency in full-video reasoning.
• VidHalluc Performance:
  • The model showed significant reductions in hallucinations across Action (ACH), Temporal Sequence (TSH), and Scene Transition (STH) dimensions.
  • Specifically, it improved TSH and STH scores by 7.83% and 7.81% respectively over the baseline, proving the effectiveness of EVSG in correcting event ordering and scene transitions.
• Ablation Studies:
  • Removing the EVSG or the Visual Attention reward led to performance drops, confirming that both the structural graph and the visual grounding mechanism are critical.
  • An optimal balance of event granularity (5, 10, 15 events) was found to be crucial for avoiding redundancy while capturing sufficient detail.

5. Significance

GraphThinker represents a significant shift in video reasoning from implicit token correlation to explicit structural reasoning. By forcing the model to construct a "thinking process" in the form of a scene graph and then validating that process against visual evidence via reinforcement learning, the method effectively bridges the gap between high-level semantic understanding and low-level visual grounding. This approach offers a scalable solution to the hallucination problem in MLLMs, making them more reliable for applications requiring precise temporal understanding, such as instructional video analysis, embodied AI, and assistive systems.