🤖 AI

Visual Credit Audit for Multimodal Spatial Reasoning

This paper introduces Visual Credit Audit (VCA), a training- and label-free framework that decomposes multimodal spatial reasoning performance into correctness, visual support, and relation-specific response, revealing that a significant portion of correct benchmark answers are actually uncredited by the visual evidence.

Original authors: Feixiang Liu, Qiang Qiu, Lanbo Sun, Nan Wei, Huawei Shen, Xueqi Cheng

Published 2026-07-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Feixiang Liu, Qiang Qiu, Lanbo Sun, Nan Wei, Huawei Shen, Xueqi Cheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are taking a test where a teacher shows you a picture of a cat sitting under a TV and asks, "Is the TV above the cat?" You answer "Yes," and you get a gold star. But what if you got that gold star not because you actually looked at the picture, but because you guessed right based on the words alone? Maybe you've heard that phrase a million times, or maybe you just know that TVs usually go on walls and cats on floors. In the world of Artificial Intelligence, specifically with "Multimodal Large Language Models" (AI that can see and read), this is a tricky problem. These models are like super-smart students who have read every book in the library, but sometimes they cheat by guessing the answer from the question's wording rather than actually analyzing the image. Scientists want to know: When the AI says "Yes," is it really seeing the TV above the cat, or is it just a lucky guess based on text?

This is exactly the puzzle tackled in a new paper called "Visual Credit Assignment for Multimodal Spatial Reasoning." The researchers built a special audit tool called Visual Credit Audit (VCA). Think of VCA as a strict detective who doesn't just check if the answer is right, but investigates why it was right. They ask two main questions: First, did the picture actually give the AI extra help to make that decision compared to just reading the text? Second, if you swapped the relationship in the picture (like putting the cat above the TV), would the AI change its mind? The paper finds that even when AI models get the right answer, they are often "cheating" by relying on text clues rather than visual evidence. In fact, for some models, up to 26.25% of their correct answers on spatial tests were actually uncredited—they got the right answer, but the image didn't actually help them get it.

The Detective's Toolkit: How VCA Works

To understand how the researchers caught these "cheaters," imagine a game of "Spot the Difference" played with a very stubborn robot.

The Setup: The "No-Image" Controls
The researchers set up a clever experiment. They took a question like "Is the TV above the cat?" and showed it to the AI in three different ways:

The Original: The question plus the real picture.
The Text-Only: The question with no picture at all (just a blank space).
The Blank Canvas: The question with a gray, empty box where the picture should be.

If the AI answers "Yes" in all three scenarios with the same confidence, the researchers say, "Aha! The picture didn't do any work." The AI is just guessing based on the text. But if the AI is much more confident when the picture is there, then the picture gets "credit" for the answer. This is the first part of the audit: Image Dependence.

The Twist: The "Relation Reversal"
But wait, there's a second trap. What if the AI is sensitive to the picture, but only in a weird way? Maybe it sees the TV and the cat, but it doesn't actually understand which one is on top. To catch this, the researchers used a "factorial" test. They created scenarios where the text said "Is the TV above the cat?" but the picture showed the TV below the cat.

If the AI is truly smart, it should say "No" when the picture contradicts the text. If it still says "Yes" because it's ignoring the picture, it fails the second test: Relation Consistency.

The Big Reveal: "Correct but Uncredited"

The researchers ran this audit on four different AI models (Qwen, InternVL, LLaVA, and Ministral) using two different spatial reasoning benchmarks. The results were eye-opening.

They found that a significant chunk of the AI's "correct" answers were actually Correct-but-Uncredited (C-U).

On the GSR-COCO dataset, the Qwen model got 90.91% of the answers right. But when they applied the audit, only 78.18% of those answers were actually supported by the image. That means 12.73% of the time, the model got the right answer, but the image didn't help it at all.
For the LLaVA model on the same dataset, the gap was even wider: 26.25% of its correct answers were uncredited.

The paper explicitly rules out the idea that these models are just "bad" at spatial reasoning. In fact, the models are sensitive to visual changes. When the researchers swapped the positions of objects in the pictures (a "relation reversal"), 81.57% to 100.00% of the models did change their answers in the correct direction. This proves the models can see the difference, but in the standard test, they often didn't need to look to get the right answer. They were relying on shortcuts.

Why This Matters (Without the Jargon)

Think of it like a student taking a math test.

Accuracy is just the grade: "Did they get the right number?"
VCA is the teacher looking at the student's scratch paper: "Did they actually do the math, or did they just guess the answer because they knew the teacher likes the number 42?"

The paper shows that current AI benchmarks are like a test where the teacher accidentally gives away the answers in the question phrasing. The models are getting high scores, but they aren't necessarily learning to "see" better.

The researchers also tested what happens if you swap the picture with a completely different, unrelated image (like swapping the cat/TV photo with a photo of a beach). When they did this, the models' "credit" score dropped by 21.25 to 47.80 points. This confirms that the original picture was indeed doing the heavy lifting for those specific correct answers, and without it, the models would have stumbled.

The Takeaway

The paper doesn't say AI is broken or that it can't see. Instead, it offers a new way to measure success. It suggests that we shouldn't just count how many times an AI gets the right answer. We need to count how many times the AI needed the picture to get that answer.

By separating "correctness" from "visual support," the researchers found that a lot of what we think is "smart vision" is actually just "smart guessing." They propose that future tests for AI should include these "credit audits" to ensure that when an AI says "I see the cat," it really means it's looking at the cat, and not just reciting a phrase it memorized. It's a call to stop celebrating the grade and start checking the homework.

Technical Summary: Visual Credit Assignment for Multimodal Spatial Reasoning

Problem Statement

Multimodal Large Language Models (MLLMs) are increasingly evaluated on spatial reasoning benchmarks (e.g., VSR, GSR-COCO) using closed yes/no questions. However, standard accuracy metrics conflate two distinct phenomena: a model answering correctly due to genuine visual evidence versus answering correctly due to strong text priors, dataset biases, or instruction-following habits. A model may answer "yes" to "Is the TV above the cat?" correctly, yet the image might provide no more support for this decision than a text-only prompt or a blank image would. Existing debiasing and counterfactual methods often focus on reducing language priors or improving overall accuracy but do not explicitly audit whether a specific, fixed spatial decision receives incremental support from the image compared to no-image controls, while simultaneously maintaining relation consistency.

Methodology: Visual Credit Audit (VCA)

The paper proposes Visual Credit Audit (VCA), a training-free, decision-level audit framework that separates benchmark success into three components: correctness, additional image support, and relation-consistent response. VCA operates on a fixed binary interface (e.g., Yes/No) and freezes the MLLM, analyzing log-likelihood margins under matched contexts.

Two-Axis Audit Framework

VCA identifies two distinct estimands:

Relative Image Dependence (Support Axis):
- Goal: Determine if the original image provides more support for the model's declared decision than text-only ( $t$ ) or blank ( $b$ ) controls.
- Metric: The VCA dependence gap ( $G^{\text{img}}_i$ ) is defined as the difference between the prediction-aligned support of the original image ( $r_{i,o}$ ) and the maximum support of the no-image controls ( $\max(r_{i,t}, r_{i,b})$ ).
- Definition: A decision is "credited" ( $D_i=1$ ) if $G^{\text{img}}_i > 0$ . Crucially, this does not require the no-image controls to favor the opposite answer, only that the image offers more support than the controls.
- Decomposition: This leads to a four-cell decomposition of accuracy:
  - D-CC (Dependence-Credited Correct): Correct decisions with image support advantage.
  - C-U (Correct-but-Uncredited): Correct decisions where the image offers no advantage over text/blank controls.
  - Support-Advantaged Errors: Incorrect decisions where the image actually provided stronger support than controls (a category often hidden in gold-only metrics).
  - Uncredited Errors: Incorrect decisions with no image support advantage.
Relation Consistency (Interaction Axis):
- Goal: Determine if the model distinguishes the true spatial relation from false alternatives on fixed pixels.
- Metric: Semantic Relation Contrast (SRC) compares the margin of the true relation query ( $q_{r+}$ ) against the maximum margin of false alternatives ( $q_{r-}$ ).
- PairCredit: Measures if the image-specific separation ( $G_{\text{int}}$ ) exceeds the separation found in no-image controls.
- Joint: A conjunction requiring both absolute relation consistency and paired dependence.

Diagnostic Interventions

To validate these estimands and address the limitations of null controls (which cannot identify relation responsiveness), the authors employ:

Matched Image Permutation: Replacing the original image with a deterministic, same-split unrelated image to calibrate the D-CC metric.
Controlled Evidence-Source Factorial: A $3 \times 3$ design varying visual evidence (relation-supporting, relation-opposite, neutral) and textual evidence (supporting, opposing, neutral) to isolate whether uncredited decisions still respond to visual relation changes.
Human-Audited Edits: A set of 108 geometry-compatible natural image edits to verify correspondence between controlled interventions and natural image changes.

Key Contributions

Label-Free Decision-Level Audit: Introduced a method that assigns credit to declared decisions without relying on ground-truth labels for the audit itself. This recovers gold-aligned gain on correct items but uniquely retains "support-advantaged errors," providing a complete decomposition of accuracy.
Separation of Marginal Support and Relation Response: Established that null controls (text/blank) cannot identify whether a model responds to relation reversals. The paper demonstrates that a decision can be "uncredited" (no marginal image advantage) yet still be "relation-responsive" (changing correctly when the visual relation is reversed).
Comprehensive Empirical Analysis: Applied VCA across four open MLLMs (Qwen, InternVL, LLaVA, Ministral) and two benchmarks (VSR, GSR-COCO), utilizing 1,800 factorial inputs per model and 108 independent edit audits.

Results

Prevalence of Uncredited Correctness: Across models and datasets, 12.73% to 26.25% of correct decisions are "uncredited" (C-U), meaning the image provided no additional support over text-only or blank controls.
Image Specificity: Matched image permutation reduces D-CC by 21.25 to 47.80 points across all runs, with 95% confidence intervals strictly above zero. This confirms that the original image's advantage is specific to its content, not just the presence of any image.
Relation Responsiveness in Uncredited Cases: Among correct-but-uncredited decisions, 81.57% to 100.00% of models still respond correctly to relation reversals in the factorial setup. Furthermore, 32.11% of these uncredited decisions actually flip their binary answer when the relation is reversed, proving they are not purely text-driven.
Natural Correspondence: On human-audited natural image edits, 25.00% of source-correct but uncredited decisions successfully flip with visibility, confirming that responsive uncredited cases exist in natural images, not just procedural renders.
Decomposition: The framework reveals that aggregate accuracy masks significant variance in how models utilize visual evidence. For instance, on GSR-COCO, Qwen achieves 90.91% accuracy, but only 78.18% of examples are both correct and image-supported.

Significance and Claims

The paper claims that VCA provides a necessary decomposition of benchmark success that accuracy alone cannot offer.

Beyond Accuracy: It argues that "correctness" and "visual support" are distinct dimensions. A model can be accurate but rely on priors (C-U), or be inaccurate but correctly utilizing visual evidence (support-advantaged errors).
Methodological Rigor: By separating the "marginal support" estimand from the "relation responsiveness" estimand, VCA avoids the conflation found in existing null-control methods. It demonstrates that null controls are insufficient for identifying relation-specific visual responses.
Practical Utility: VCA serves as a post-hoc companion to accuracy, offering a "layered report" that helps diagnose whether improvements in spatial reasoning stem from genuine visual evidence or stronger answer priors. It does not claim to replace accuracy but to refine the interpretation of benchmark success.
Limitations: The authors modestly note that VCA measures relative support under prespecified controls, not internal causal pathways or pixel-level attribution. It is designed for fixed forced-choice interfaces and does not directly address unconstrained open-ended generation.