Learning to Deny: Action Denial in Multimodal Large Language Models

Technical Summary: Learning to Deny: Action Denial in Multimodal Large Language Models

Problem Definition
Despite rapid advancements in video understanding, Multimodal Large Language Models (MLLMs) exhibit a critical blind spot: the inability to reliably deny an action when strong contextual cues are present but the defining motion is absent. Current models display an "agreeableness" or affirmative bias, often predicting that an action is occurring based on the presence of relevant objects, actors, or backgrounds, even when the specific temporal dynamics required for that action are missing. Existing benchmarks (e.g., Kinetics, UCF101) primarily consist of positive examples, reinforcing spurious correlations between context and action rather than teaching models to verify causal motion. Consequently, models fail to distinguish between a scene where an action could happen and one where it is happening.

Methodology
The authors propose a two-pronged approach involving a new benchmark and a causal reasoning framework:

1. UCF101-ActionDenial (UCF101-AD) Benchmark:

   • Construction: A dataset of 11,283 videos derived from UCF101 categories, featuring paired "Action-Presence" (positive) and "Action-Denial" (negative) clips.
   • Negative Design: Negative clips preserve the typical context (persons, objects, locations) of the target action but explicitly exclude the defining motion. They are categorized into two types:
     • Type 1 (Context Only): Salient objects and environment are present, but no interaction occurs (e.g., a person near a piano without playing).
     • Type 2 (Same Context, Different Motion): The setting and objects are present, but the actor performs a different, plausible action (e.g., dribbling a basketball without dunking).
   • Evaluation Protocol: Models are tested via a Multiple-Choice Question (MCQ) setup where they must select the correct caption from options including the target action, distractors, and a "None" option.

2. CausalAct Framework:

   • Causal Graph Formulation: The authors model action recognition as a Directed Acyclic Graph (DAG) where an action ($A$) is the outcome of a causal chain: Context (Persons $P$, Objects $O$, Location $L$) $\to$ Relations (Spatial $S$, Interaction $I$) $\to$ Motion ($M$) $\to$ Action ($A$).
   • Prompting (CausalAct-0): The graph structure is instantiated as a natural-language prompt, guiding the model to verify the causal chain (specifically the presence of motion $M$) before confirming an action.
   • Finetuning: To address cases where models fail to follow prompts, the authors introduce auxiliary finetuning tasks. Models are trained on graph-centric Visual Question Answering (VQA) tasks (probing node identities, edge relationships, and causal paths) derived from the graph structure. Crucially, these tasks do not expose the ground-truth action labels to prevent label leakage, forcing the model to learn the structure of causal reasoning.

Key Results

• Baseline Failure: Evaluation of 20 state-of-the-art MLLMs (including Ovis, Intern, Qwen, VideoLLaMA, and GPT-4o mini) reveals a severe performance gap. While models achieve high accuracy (>85-95%) on standard Action-Presence clips, their performance on Action-Denial clips collapses, with most models scoring below 50% (often in the 20-35% range). This indicates a strong reliance on contextual shortcuts rather than motion verification.
• Human vs. Model: Human evaluators achieved 86.6% accuracy on the denial task, confirming that the dataset is solvable and that the failure is specific to current model architectures.
• Correlation of Biases: The study finds a positive correlation between "shortcut learning" (failing to deny) and "sycophancy" (agreeing with a premise). Models that incorrectly select a distractor action in MCQs are also more likely to incorrectly answer "Yes" to binary questions about the action's presence.
• Reasoning Models: Counterintuitively, "thinking" models (those with explicit reasoning steps) often perform worse than standard models on this task, as their reasoning chains tend to over-index on visual context and hallucinate actions rather than justifying absence.
• CausalAct Efficacy:
  • Zero-Shot: Stronger language backbones (e.g., Ovis2.5-9B, Qwen2.5-VL-7B) show improved denial accuracy when guided by CausalAct prompts, though weaker models may degrade.
  • Finetuning: Finetuning smaller models on graph-based VQA tasks significantly improves Action-Denial accuracy, shifting behavior from contextual shortcutting to verifying the causal chain.
  • Generalization: The CausalAct-0 approach transfers to external datasets (K400, HMDB51, SSv2, Diving48), improving denial accuracy across diverse distributions.
  • Ablation: Performance drops when the causal graph structure is pruned or randomized after finetuning, confirming that models learn to utilize the specific causal dependencies rather than just following arbitrary text patterns. Furthermore, updating the vision encoder during finetuning is essential; freezing it leads to significant performance degradation.

Significance and Claims
The paper claims to provide the first systematic assessment of "action denial" in MLLMs, exposing a fundamental limitation where models equate context with action occurrence. The authors argue that robust video understanding requires the ability to causally reason about the absence of motion, not just its presence.

The significance of the work lies in:

1. Diagnosis: Identifying that current MLLMs lack a mechanism to verify defining motions, leading to false positives in critical applications like surveillance and autonomous systems.
2. Benchmarking: Introducing UCF101-AD as a necessary tool for diagnosing and improving causal reasoning, moving beyond standard positive-action recognition.
3. Methodological Insight: Demonstrating that denial is a learnable reasoning skill. By structuring scene understanding through causal graphs and training models to verify preconditions (interaction and motion), it is possible to mitigate the affirmative bias and reduce false positives without relying on negative action labels during training.

The authors maintain a modest stance, noting that while CausalAct improves performance, it does not fully solve the problem, and current "thinking" models still struggle with negative constraints. The work serves as a foundation for advancing causal reasoning in multimodal systems.