ENTER: Event Based Interpretable Reasoning for VideoQA

1. Problem Statement

Video Question Answering (VideoQA) requires models to recognize objects, actions, and events, and understand their temporal and causal evolution. While existing approaches have achieved high accuracy, they face a fundamental trade-off between performance and interpretability:

• Top-Down Approaches: Generate reasoning plans (often as code) based solely on the input question. While interpretable, they often ignore low-level visual context during the planning phase, making them brittle and prone to hallucinations when visual clues are missing.
• Bottom-Up Approaches: Process visual data directly to generate answers. While robust and context-aware, they function as "black boxes," lacking transparent reasoning paths.
• The Gap: Current systems struggle to combine the transparency of modular reasoning with the contextual robustness of visual data processing.

2. Methodology: ENTER

The authors propose ENTER (Event Based Interpretable Reasoning), a hybrid framework that bridges this gap by using Event Graphs as an intermediate, structured representation. The pipeline consists of four main stages:

A. Event Graph Generation

Instead of processing raw frames directly for reasoning, ENTER first converts the video into a structured graph:

1. Captioning: A Vision-Language Model (VLM) generates detailed, structured captions from the video, emphasizing temporal ("after," "meanwhile"), causal ("due to"), and hierarchical relationships.
2. Graph Construction: An LLM extracts events (nodes) and their relationships (edges) from the captions.
   • Nodes: Represent events with arguments (e.g., Actor, Object, Place).
   • Edges: Capture Temporal (sequence), Causal (cause-effect), and Hierarchical (containment) relationships.
   • Coreference: The system ensures consistent naming of entities across the graph to avoid confusion.

B. Code Generation and Execution

The Event Graph and the input question are fed into a Large Language Model (LLM) to generate Python code.

• API Design: The LLM uses a custom API (EventGraph, Event) to traverse the graph (e.g., find_node, get_temporal_after, get_caused_by).
• Execution: The generated code is executed by a Python interpreter. The logic explicitly models relationships between events, allowing the system to "reason" over the graph structure rather than just pattern matching.
• Output: The code execution produces the final answer.

C. Iterative Hierarchical Graph Update (Self-Correction)

To address information gaps (where the initial graph is insufficient), ENTER employs a cost-aware, three-stage iterative update mechanism:

1. Denser Graph Generation: If the caption exists but the graph missed details, the system reprocesses the existing captions to generate a denser graph.
2. Denser Caption Generation: If the captions lack necessary details, the system prompts the VLM to re-describe the video with a specific focus on the missing information.
3. Multimodal Information Incorporation: If text-based updates fail, the system retrieves relevant video clips (multimodal edges) using a text-to-video retrieval model (Gemini 1.5 Flash) and passes them to a VLM to answer the specific query.

This hierarchy ensures efficiency: the system only incurs the high cost of re-watching video or multimodal retrieval if cheaper text-based updates fail.

3. Key Contributions

1. Interpretable Event Graphs: Introduces a structured intermediate representation that explicitly captures temporal, causal, and hierarchical relationships, offering full transparency into the reasoning process via generated code.
2. Context-Aware Modular Reasoning: Unlike previous top-down methods, ENTER integrates visual context during the plan generation phase by grounding the reasoning in the Event Graph.
3. Robust Self-Correction: The hierarchical iterative update mechanism allows the system to detect and fix its own reasoning failures (e.g., missing nodes or edges) without human intervention, significantly improving robustness.
4. Superior Expressiveness: The paper demonstrates that Event Graphs are a superset of Scene Graphs, offering higher sufficiency (92% vs. 21%) for answering complex event-based questions.

4. Experimental Results

ENTER was evaluated on four major benchmarks: NExT-QA, IntentQA, NExT-GQA, and STAR.

• Performance:
  • NExT-QA: Achieved 75.1% accuracy, outperforming the best modular approach (MoReVQA) by a significant margin and matching/exceeding top bottom-up methods.
  • IntentQA: Achieved competitive performance (67.1%), comparable to state-of-the-art bottom-up models like VideoINSTA, while maintaining interpretability.
  • STAR: Surpassed all top-down and zero-shot approaches by over 20%, demonstrating superior handling of real-world situated reasoning.
  • NExT-GQA: Achieved SOTA results in grounded VideoQA, with significant improvements in IoU (Intersection over Union) metrics, proving the model can locate evidence in the video.
• Ablation Studies:
  • Removing the iterative update mechanisms dropped performance significantly (from 75.1% to ~63.8%).
  • The system remains robust across different reasoner LLMs (e.g., Llama 3 vs. GPT-4), showing a smaller performance gap compared to other methods when using open-source models.
• Interpretability: Human evaluation showed that in 86.67% of cases, the reasoning steps were deemed interpretable, and when the answer was correct, the reasoning was valid 88.89% of the time.

5. Significance

ENTER represents a paradigm shift in VideoQA by successfully unifying the transparency of modular reasoning with the robustness of data-driven visual processing.

• Trustworthiness: By generating executable code and visualizing the event graph, the system allows users to debug why a model failed (e.g., distinguishing between "insufficient knowledge," "missed information," or "inconsistent referencing").
• Efficiency: The hierarchical update strategy optimizes computational resources, only invoking expensive multimodal operations when necessary.
• Generalizability: The approach is not limited to specific datasets; it generalizes well to long-form videos and complex causal reasoning tasks, setting a new standard for explainable AI in video understanding.