Technical Summary: GridVQA-X

Problem Statement

The rapid advancement of Large Vision-Language Models (LVLMs) has outpaced the development of reliable Multimodal Explainable AI (MxAI) evaluation protocols. While current MxAI methods claim to attribute decisions to cross-modal interactions, existing benchmarks lack the ground truth necessary to distinguish between genuine cross-modal reasoning (e.g., spatial composition) and shallow cross-modal shortcuts (e.g., Bag-of-Words attribute matching).

Current evaluation relies on natural datasets where feature distributions are noisy and causally undefined, or on human annotations that can be spuriously correlated with visual elements. Consequently, it remains unknown whether MxAI methods faithfully capture synergistic interactions or merely hallucinate reasoning on models acting as simple feature detectors. There is a critical gap in rigorously assessing whether explainers can differentiate between a model using true spatial reasoning and one exploiting statistical shortcuts.

Methodology: The GridVQA-X Framework

To address this, the authors introduce GridVQA-X, a diagnostic framework utilizing a closed-world synthesis logic to generate mathematically guaranteed explanations. The framework operates on a four-stage pipeline:

1. Dataset Generation with Ground-Truth Masks

The authors generate two synthetic datasets, $D_{pure}$ and $D_{spur}$, parameterized by a 4-tuple: Depth ($D$), Question Type ($Q$), Form ($F$), and Density ($\rho$).

• $D_{pure}$ (Pure): Systematically removes all enumerated heuristic shortcuts. It enforces Spatial Independence by populating "confuser regions" with adversarial distractors that match target attributes but violate spatial constraints. This mathematically guarantees that the only valid causal explanation for a query is the specific intersection of target and anchor objects.
• $D_{spur}$ (Spurious): Intentionally preserves the most pervasive cross-modal shortcut (Case-1: Bag-of-Words alignment). The generator ensures that no target-matching objects exist outside the valid spatial region, making the shortcut perfectly predictive ($P(Y_{ans} | \text{Target Attrs}) = 1.0$).

2. Paired Model Training

Two reference models, $M_{pure}$ and $M_{spur}$, are trained on identical architectures (MDETR) but on the respective datasets.

• $M_{pure}$: Learns robust spatial-relational reasoning to achieve high accuracy on $D_{pure}$.
• $M_{spur}$: Structurally forced to rely on cross-modal shortcuts. When evaluated on $D_{pure}$, its performance collapses (dropping to ~49% overall and ~8% on multi-hop queries), confirming it operates via a unimodal shortcut rather than true reasoning.

3. Attribution Generation

Post-hoc MxAI methods are applied to both models. The framework evaluates whether the explainers produce distinct reasoning pathways for $M_{pure}$ (true reasoning) versus $M_{spur}$ (shortcut reliance).

4. Evaluation Metrics

The authors adapt existing metrics and introduce new ones:

• Relevance Mass Accuracy (RMA): Measures the relevance mass assigned to ground-truth masks versus distractor masks. A faithful explainer should assign near-zero mass to distractors for $M_{pure}$ but high mass to them for $M_{spur}$.
• Additive Fallacy Check: For global methods, the synergy score ($S_{score}$) must monotonically increase with query complexity (Depth) for $M_{pure}$ while remaining invariant for $M_{spur}$.

Key Contributions

1. GridVQA-X Datasets: The introduction of $D_{pure}$ and $D_{spur}$, which provide mathematically guaranteed unique ground-truth features and a taxonomy isolating diverse degrees of real-world interactions. These datasets explicitly prove the absence (in $D_{pure}$) or presence (in $D_{spur}$) of known spurious correlations.
2. Reference Models ($M_{pure}$ and $M_{spur}$): The release of two models trained via explanation-guided dynamics. Their verifiable reasoning mechanisms (true spatial reasoning vs. cross-modal shortcut) allow for the evaluation of explainers with zero ambiguity regarding expected ground-truth attributions.
3. Novel Evaluation Protocol: The adaptation of MxAI metrics to the multimodal domain and the introduction of the Additive Fallacy check to assess cross-model interaction scalars on compositional complexity.

Results and Analysis

The evaluation of state-of-the-art MxAI methods (including DIME, MultiSHAP, MultiViz, EMAP, and InterSHAP) reveals significant failures:

• Failure to Distinguish Models: Most local explainers fail to differentiate between $M_{pure}$ and $M_{spur}$.
  • MultiViz exhibits "absolute model blindness," yielding statistically identical Relevance Mass Accuracy (RMA) for both models despite their distinct causal pathways.
  • DIME suffers from "accidental faithfulness," where its diffuse heatmaps accidentally intersect with true targets in $M_{spur}$, masking the shortcut.
  • MultiSHAP structurally prefers the spurious model ($M_{spur}$), yielding higher RMA for it than for the faithful model, likely because game-theoretic marginals align better with independent feature detectors (shortcuts) than entangled intersections.
• Inability to Capture Spatial Constraints: Local methods often default to shallow feature detection. MultiSHAP highlights all objects matching target attributes (including adversarial distractors), failing to apply binary spatial constraints.
• Hallucinated Synergy: Global methods fail the Additive Fallacy check. EMAP and InterSHAP hallucinate high cross-modal synergy for the shortcut model ($M_{spur}$) and, conversely, show a decrease in estimated interaction as complexity scales for the faithful model ($M_{pure}$), proving their estimators are misaligned with the causal graph.

Significance and Claims

The paper claims that GridVQA-X exposes a critical blindspot in current MxAI research: state-of-the-art explainers fundamentally fail to capture true cross-modal synergy.

The authors argue that current methods create an "illusion of interpretability" that masks biased model behavior. By providing a strictly controlled environment with mathematically verifiable ground truth, GridVQA-X offers a definitive, zero-ambiguity testbed. The framework aims to shift the field away from superficial plausibility metrics toward the development of explainers capable of verifiably diagnosing relational cross-modal grounding. The authors do not claim to have solved the problem of explainability but rather to have provided the necessary diagnostic tool to identify where current methods fail.