🤖 machine learning

How Do VLMs Behave When Blind or Misled? Behavioral Evaluation of VLMs on Scientific Figures

This paper introduces SciFigBench, a comprehensive benchmark and the Admittance-Resistance-Inductance (A-R-I) framework to evaluate the behavioral reliability of Vision-Language Models under uncertainty in scientific figure understanding, revealing that high perception and reasoning accuracy do not guarantee a model's ability to acknowledge missing evidence or resist misleading context.

Original authors: Paul Osemudiame Oamen, Owusu-Banahene Osei, Ananya Mukherjee, Christian Greisinger, Steffen Eger, Pius Onobhayedo, Wei Zhao

Published 2026-08-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Paul Osemudiame Oamen, Owusu-Banahene Osei, Ananya Mukherjee, Christian Greisinger, Steffen Eger, Pius Onobhayedo, Wei Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a super-smart assistant to help you read a complex map. You want someone who can not only see the mountains and rivers but also tell you exactly how high the peaks are and which path is the shortest. In the world of artificial intelligence, these assistants are called Vision-Language Models (VLMs). They are like digital detectives that look at pictures (like charts, graphs, or diagrams) and talk about what they see. For a long time, scientists have been testing these AI detectives by asking them simple questions: "What color is this bar?" or "How many items are in this list?" If the AI gets the answer right, we assume it is smart and reliable.

But here is the tricky part: what happens when the map is smudged, or when someone hands the AI a fake map and asks, "Where is the treasure?" A truly reliable detective shouldn't just guess the location of the treasure because the fake map says it's there; they should admit, "I can't see that part," or "That map looks wrong." This paper dives into a specific corner of AI research called "behavioral reliability." It asks a crucial question: Do these AI models know when they are blind, or do they just make up answers to look helpful? The researchers wanted to see if an AI that is great at describing a clear picture would also be honest when the picture is blurry or misleading.

The Great "Blind Spot" Test

The researchers built a special playground called SCIFIGBENCH to test this. Imagine a giant library of 250 scientific charts—like bar graphs and line plots—from real research papers. But instead of just asking the AI to describe them normally, the team decided to play some tricks. They took these charts and:

Blurred specific labels so they were impossible to read.
Rotated the images so they were tilted.
Added fake captions that told lies about what the chart showed.
Asked questions about things that didn't exist in the chart at all.

They wanted to see how eight different top-tier AI models (including big names like GPT-5.2 and Gemini 3.1 Pro) would react to these traps. To measure this, they created a new scoring system called A-R-I, which stands for Admittance, Resistance, and Inductance. Think of it like a three-part test for honesty and smarts:

Admittance: If the AI can't see something (because it's blurred), does it admit, "I can't see that"? Or does it just guess?
Resistance: If someone tells the AI a lie (like a fake caption), does it say, "No, that doesn't match the picture"? Or does it just agree to be nice?
Inductance: If a piece of the chart is hidden but the rest of the picture gives a clue (like a pattern), can the AI figure out the missing piece without making it up?

The Shocking Results: The "Confident Fabricator"

The results were a bit like a plot twist in a mystery movie. The researchers found that being good at describing a clear picture doesn't mean you are good at handling a tricky one.

The Star Performer (GPT-5.2):
This model was the best at describing clear charts, scoring a 91.6 out of 100 on a quality scale called MQM. It was a superstar at seeing what was right there. But when the researchers blurred a label so it was unreadable, this model acted like a confident fabricator. In 96% of the cases where it couldn't see the answer, it just made one up anyway! It would say, "The label says 'Customer Support'," even though that word wasn't there and it was clearly blurred. It was so eager to give an answer that it forgot to be honest.

The Honest Detective (Gemini 3.1 Pro):
This model was almost as good at describing clear charts (scoring 90.2), but it behaved very differently when things got tricky. When faced with a blurred label, it admitted uncertainty 71% of the time. It was willing to say, "I can't read that," instead of guessing. It also had the highest Resistance score of 0.91, meaning it was excellent at ignoring fake captions and false questions. It didn't get tricked by lies.

The Others:
The other models in the test, like Llama 4 and various Qwen models, generally struggled more. Some, like Phi-4 Multimodal, were so eager to please that they followed fake captions almost 100% of the time, even when the picture clearly showed something else.

Why This Matters

The paper shows that we can't just look at how well an AI describes a picture to know if it's safe to use in real life. If you are using an AI to help with scientific research, you don't want a model that confidently invents data just because it wants to give you an answer.

The authors conclude that high accuracy on clean tests does not guarantee reliable behavior. A model can be a great artist at describing a clear painting but a terrible detective when the painting is smudged or when someone tries to trick it. For AI to be truly useful in serious fields like science, it needs to learn the art of saying, "I don't know," when it's actually blind. This new benchmark, SCIFIGBENCH, gives us a way to test for that honesty, ensuring that our AI assistants are not just smart, but also trustworthy.

Technical Summary: SCIFIGBENCH – Behavioral Evaluation of VLMs on Scientific Figures

Problem Statement

Existing benchmarks for Vision-Language Models (VLMs) primarily focus on perception (how well a model describes an image) and reasoning (how accurately it answers questions based on visual evidence). However, these benchmarks largely ignore behavioral reliability under uncertainty. In scientific workflows, models often encounter degraded visual evidence (e.g., blurred labels, low-resolution PDFs), misleading context (e.g., incorrect figure captions), or false premises. Current metrics fail to capture how models behave when evidence is missing or contradictory, specifically whether they acknowledge uncertainty, resist misleading inputs, or infer cautiously from partial information. This gap poses a significant risk for deploying VLMs in scientific research, where "confident fabrication" can corrupt downstream analysis.

Methodology: SCIFIGBENCH and the A-R-I Framework

To address this, the authors introduce SCIFIGBENCH, a diagnostic benchmark designed to jointly evaluate perception, reasoning, and behavioral reliability.

1. Dataset Construction

Source: 250 high-quality scientific figures (bar charts, line plots, pie charts) from 187 arXiv papers (2023–2025).
Annotations: Expert descriptions generated by trained annotators with 94% agreement (inter-annotator reliability measured by Krippendorff's $\alpha$ = 0.91) and 1,000 reasoning questions covering counting, computation, comparison, and pattern analysis.
Stress-Testing: The base dataset is expanded via image transformations and adversarial probes to create over 34,000 evaluation setups. These include:
- Visual Transforms: Rotation, noise, low contrast, and in-paper context (figures embedded in full PDF pages).
- Selective Blur: Targeted blurring of specific elements to test Admittance (unrecoverable elements) and Inductance (inferable elements).
- Resistance Probes:
  - Caption Bias: Modifying captions with 2–3 plausible but false claims.
  - False Premises: Questions embedding non-existent elements (Inexist), wrong numerical anchors (Contra), or unanswerable queries (Unanswerable).

2. The A-R-I Framework

The paper proposes the Admittance–Resistance–Inductance (A-R-I) framework to decompose model behavior under uncertainty:

Admittance: Measures epistemic honesty. Does the model acknowledge when visual evidence is altered or unrecoverable (e.g., blurred text) rather than fabricating an answer?
Resistance: Measures robustness to misleading context. Does the model reject false premises, non-existent elements, or misleading captions that conflict with visual evidence?
Inductance: Measures bounded inference. When evidence is degraded but recoverable from surrounding context (e.g., inferring a blurred label from axis patterns), does the model infer correctly rather than abstain or hallucinate?

3. Evaluation Protocol

Models: Eight VLMs were evaluated, including commercial leaders (GPT-5.2, Gemini 3.1 Pro) and open-weight models (Llama 4, Qwen3-VL, Gemma 3, Phi-4).
Metrics:
- Perception: MQM (Multidimensional Quality Metrics) score for description quality.
- Reasoning: Accuracy on capability questions.
- Behavior: Resistance scores (0–1 scale) and percentages for Admittance (acknowledging uncertainty) and Inductance (correct inference).
Judging: Automated evaluation using GPT-4o as a judge, validated against human annotators (Krippendorff's $\alpha$ = 0.91) and cross-judge ablation with Mistral Large 3.

Key Results

1. Perception vs. Behavior Disconnect

High perception and reasoning accuracy do not guarantee behavioral reliability.

GPT-5.2 achieved the highest baseline description quality (MQM 91.6) and strong reasoning (78.4%). However, it exhibited a "confident fabricator" profile: when asked about selectively blurred (unrecoverable) elements, it admitted uncertainty in only 8% of active probes, while fabricating answers in 96% of such cases.
Gemini 3.1 Pro, with comparable perception (MQM 90.2) and reasoning (81.0%), demonstrated superior behavioral reliability. It admitted uncertainty in 71% of cases involving unrecoverable elements and achieved the highest Resistance score (0.91).

2. Behavioral Divergence

Resistance: Models with similar overall capabilities showed stark differences in resisting false premises. Phi-4 Multimodal had a resistance score of 0.21, while Gemini 3.1 Pro scored 0.91.
Caption Bias: Resistance to misleading captions varied significantly. Phi-4 followed modified captions 95% of the time, whereas Gemini and GPT-5.2 maintained high resistance (~0.89).
Inductance: When elements were blurred but inferable, top models (Gemini 66%, GPT-5.2 59%) could correctly infer values, distinguishing this from fabrication on unrecoverable data.

3. Failure Modes

Presupposition Embedding: The "Inexist" probes (asking about non-existent elements using definite articles) were the most effective deception vector, causing even strong models to fail more often than on direct false-value questions.
Active vs. Passive: Models often admitted uncertainty in passive descriptions but switched to definitive (and often fabricated) answers when directly questioned, suggesting a "must-answer" bias driven by helpfulness training.

Significance and Claims

The paper claims that SCIFIGBENCH reveals a critical third dimension of VLM competence: behavioral reliability. The authors argue that:

Accuracy is Insufficient: Selecting models for scientific workflows based solely on perception/reasoning benchmarks risks embedding "confident fabrication" into research pipelines.
Behavior is Separable: The A-R-I dimensions are empirically separable from quality metrics; a model can be a top-tier describer yet a poor estimator of its own uncertainty.
Framework Utility: The A-R-I framework provides a structured way to diagnose specific failure modes (silent fabrication, context manipulation, failure to reason from partial evidence) that are invisible to conventional aggregate accuracy metrics.

The authors conclude that evaluating VLMs for scientific deployment requires explicit testing of how models handle degraded evidence and misleading context, not just how well they perform on clean, standard tasks.