Increasing Computation Resolves Conflicts in Vision Language Models

1. Problem Statement

Vision-Language Models (VLMs) have achieved remarkable performance in perception and reasoning tasks. However, their ability to exhibit cognitive control—the ability to coordinate competing information sources to pursue specific goals in the presence of conflict—remains unexplored. In real-world scenarios (e.g., autonomous driving, medical diagnosis), models must prioritize task-relevant signals while suppressing misleading or conflicting cues (noise, adversarial attacks, or contradictory modalities).

The paper addresses two fundamental questions:

1. Do VLMs exhibit genuine susceptibility to cognitive conflict (i.e., do they suffer from interference similar to humans)?
2. Can increasing computational resources (model scale) enable VLMs to resolve these conflicts effectively, mirroring the human demand-resource relationship?

2. Methodology

A. Benchmark Construction

The authors constructed a comprehensive benchmark comprising 4,410 tasks across seven conflict paradigms, tested on 47 diverse VLMs (ranging from 1B to 110B parameters, including open-source and proprietary models).

The benchmark is divided into two categories:

1. Classic Conflict Paradigms (Synthetic):
   • Stroop Task: Identifying ink color while ignoring word meaning.
   • Flanker Task: Identifying a central target while ignoring surrounding distractors.
   • Squared Variants: These introduce a second, independent conflict (e.g., dual mappings) to increase difficulty and prevent performance saturation.
2. Realistic Photo-Realistic Paradigms:
   • Stroop-Object: Conflicts between overlaid text (OCR) and the actual visual object (e.g., an image of a car labeled "bus").
   • Stroop-Attribute: Conflicts between text and visual properties (e.g., shape, emotion, scene atmosphere).
   • Visual Context vs. Knowledge: Conflicts between immediate visual context and parametric knowledge (e.g., a giraffe in an ocean; the model must ignore the ocean and recall the giraffe's natural habitat).

Generation Pipeline: The authors used visual programming and AI-generated images (via Gemini-2.5-flash-image-preview) to create controlled stimuli. They ensured that congruent and incongruent pairs shared identical visual content, differing only in the textual manipulation, thereby isolating the conflict variable.

B. Experimental Design

• Models: 47 VLMs were evaluated, covering a wide spectrum of parameter sizes (1B to 110B+).
• Metrics: Accuracy on Congruent (task-relevant and irrelevant signals align) vs. Incongruent (signals conflict) trials.
• Key Hypothesis: The paper tests if the "Congruency Effect" (performance drop on incongruent trials) diminishes as model scale increases, and if larger models exhibit specific "below-chance" dips on high-conflict trials, indicating genuine processing rather than shortcut exploitation.

3. Key Contributions

1. First Unified Framework for VLM Cognitive Control: The paper introduces the first benchmark systematically adapting classic human cognitive control tasks (Stroop, Flanker) to multimodal VLMs, spanning from synthetic to photo-realistic scenarios.
2. Discovery of the "Below-Chance" Dip: The authors identify a critical signature of genuine conflict processing: on high-conflict (squared) incongruent trials, larger models drop below chance accuracy, while smaller models perform at chance. This mirrors human behavior at short processing times where interference is strong before resolution mechanisms engage.
3. Parameter Count as a Proxy for Cognitive Control: The study establishes that parameter count serves as a functional proxy for computational resources in conflict resolution. Larger models systematically resolve conflicts better than smaller ones.
4. Emergence of Human-Like Dynamics: The results demonstrate that human-like demand-resource relationships emerge naturally in large-scale neural networks trained via standard gradient descent, without specialized top-down control architectures.

4. Key Results

• Robust Congruency Effect: All 47 tested VLMs exhibited a significant performance drop on incongruent trials compared to congruent trials.
  • Non-Squared Tasks: Mean accuracy dropped from 0.852 (congruent) to 0.567 (incongruent).
  • Squared Tasks: The drop was even more pronounced, from 0.634 to 0.297.
• Scale-Dependent Resolution:
  • Small Models: Showed pronounced congruency effects and often failed to engage meaningfully with high-conflict tasks (performing at chance).
  • Large Models: Achieved comparable performance across congruent and incongruent conditions in standard tasks. Crucially, in high-conflict (squared) tasks, they showed above-chance performance on congruent trials but below-chance performance on incongruent trials.
• The "Dip" Phenomenon: The "below-chance" performance of large models on incongruent trials is a critical finding. It indicates that these models are actively processing the conflicting information (leading to interference) rather than relying on superficial shortcuts (which would result in monotonic improvement or chance-level performance). This pattern structurally corresponds to human temporal dynamics where conflict is initially overwhelming before resolution.
• Modality Bias: The study found a systematic bias toward text processing (OCR) over object recognition in "Stroop-Object" tasks, suggesting current VLMs rely heavily on textual priors. However, this bias did not eliminate the congruency effect.

5. Significance and Implications

• Mechanism of Cognitive Control: The findings challenge the notion that cognitive control requires specialized, hand-crafted architectural modules (like those in traditional cognitive architectures such as ACT-R). Instead, adaptive flexibility and conflict resolution emerge naturally from the optimization dynamics of scaling up standard deep learning models.
• Safety and Robustness: The ability to resolve conflicting signals is vital for safety-critical AI applications. The paper suggests that scaling models may naturally improve their robustness against noisy sensors and adversarial inputs.
• Limitations: While larger models show improved conflict resolution, they do not fully recover to the performance levels of low-conflict tasks even at the largest scales tested, indicating a ceiling effect in zero-shot vision question answering.
• Future Directions: The paper calls for mechanistic interpretability studies to identify how specific attention heads or layers implement conflict resolution and for testing these capabilities in "in-the-wild" uncontrolled environments.

In summary, the paper provides strong evidence that increasing computation resolves conflicts in VLMs, demonstrating that the fundamental mechanisms of human-like cognitive control are an emergent property of scaling large-scale neural networks.