⚡ electrical engineering

Vision-Language Models vs Human: Perceptual Image Quality Assessment

This paper benchmarks six Vision-Language Models against human psychophysical data for perceptual image quality assessment, revealing that while these models show strong attribute-dependent alignment and increased reliability with clear stimulus differences, their high self-consistency does not necessarily correlate with human agreement.

Original authors: Imran Mehmood, Imad Ali Shah, Ming Ronnier Luo, Brian Deegan

Published 2026-03-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Imran Mehmood, Imad Ali Shah, Ming Ronnier Luo, Brian Deegan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to decide which of two new soups tastes better. Traditionally, you'd have to ask 20 different people to taste them, record their opinions, and do the math. This is accurate, but it's slow, expensive, and hard to scale.

Now, imagine you have a team of super-smart AI robots (called Vision-Language Models or VLMs) that can "see" the soup and tell you which one looks more appetizing. The big question this paper asks is: Can these AI robots replace the human taste testers?

The researchers put six different AI models to the test against real human data to see how well they judge image quality. Here is what they found, explained simply:

1. The Test: A "Taste Test" for Pictures

The researchers used a dataset of 70 images created by taking 10 beautiful high-definition photos and running them through 7 different "filters" (called Tone Mapping Operators). These filters change how the image looks, specifically tweaking:

Contrast: How dark the darks are and how bright the brights are (like the "pop" of the image).
Colorfulness: How vibrant and rich the colors are.
Overall Preference: Which image just looks "better" overall.

They asked both humans and AI models to look at pairs of these images and pick a winner for each category.

2. The Results: The AI Team Has Different Specialties

Just like a sports team where one player is great at defense but bad at offense, the AI models had very different strengths and weaknesses.

The "Color Connoisseurs" (Claude, Intern, Qwen): These models were amazing at judging colorfulness. They agreed with humans almost perfectly (93% match). If you wanted to know which image had the most vibrant colors, these AIs were your go-to.
The "Contrast Critics" (Qwen, Gemini, GPT): These models were better at judging contrast. They could tell when an image looked too flat or too harsh.
The "All-Rounder" (GPT): This model was the most balanced. It wasn't the absolute best at just one thing, but it did a great job at Overall Preference, matching human opinion better than anyone else (86% match).

The Catch: Being consistent doesn't mean being right.
One model (Claude) was incredibly consistent—it gave the exact same answer every time you asked it. But, it was consistently wrong compared to humans when judging contrast. It was like a robot that stubbornly insists a gray sky is blue every single time. Meanwhile, another model (GPT) changed its mind a little bit more often, but those changes actually made it align better with how humans think.

3. The "Blind Spot" Problem

The researchers found something interesting about how hard the images were to tell apart.

Easy Mode: When two images were very different (e.g., one was bright and colorful, the other was dark and dull), the AI models were very reliable. They could easily spot the winner.
Hard Mode: When the images were very similar (subtle differences), the AI models started to get confused and disagreed with humans.
The Metaphor: Think of it like a music critic. If you play a heavy metal song and a lullaby, the critic will easily say which is which. But if you play two very similar jazz songs, the critic might struggle to explain the difference, and their opinion might drift away from what the average listener feels.

4. How the AI "Thinks" (The Recipe)

The researchers analyzed how the AI combined "contrast" and "color" to make a final decision on "Overall Preference."

Humans tend to care a lot about color. If an image is colorful, we usually like it more, even if the contrast isn't perfect.
Most AIs copied this human behavior. They gave "color" a high weight in their decision-making recipe.
One AI (Grok) was weird. It cared almost exclusively about contrast and ignored color, which made it less like a human.

The Bottom Line: Are AI Taste Testers Ready?

Not quite yet.

The Good News: AI is fantastic for rapid screening. If you have 1,000 images and need to quickly filter out the 900 that look terrible, AI is perfect. It's fast, cheap, and good at spotting obvious differences.
The Bad News: AI cannot yet replace human experts for the final decision. If you need to know the exact subtle difference between two high-quality images, you still need real humans. The AI is too inconsistent on the fine details and sometimes gets stuck in its own "biases."

In summary: These AI models are like enthusiastic interns. They are great at doing the heavy lifting and spotting the big differences, but they still need a human supervisor to double-check the subtle nuances before you serve the final dish.

1. Problem Statement

Perceptual Image Quality Assessment (IQA) is critical for computational imaging but traditionally relies on psychophysical experiments involving human observers. While reliable, these experiments are expensive, time-consuming, and lack scalability.

The Gap: Existing automated metrics often fail to correlate with human perception in complex scenes.
The Question: Can Vision-Language Models (VLMs) approximate human perceptual judgments across specific quality attributes (contrast, colorfulness, and overall preference) to serve as a scalable alternative for rapid screening?
The Challenge: Perceptual attributes are subjective, context-dependent, and influenced by subtle interactions, making it unclear if VLMs can capture these nuances without controlled experimental bias.

2. Methodology

The authors conducted a systematic benchmark comparing six VLMs against established human psychophysical data.

A. Dataset & Ground Truth

Source: Data from a psychophysical experiment by Mehmood et al. [15] involving 10 HDR images rendered by 7 Tone Mapping Operators (TMOs), creating 70 distinct images.
Attributes Evaluated: Contrast, Colorfulness, and Overall Preference.
Human Protocol: 20 observers performed pairwise comparisons in a controlled environment (dark room, calibrated display). Data was converted to standardized z-scores using Thurstone's law of comparative judgment.

B. VLM Evaluation Protocol

Models Tested: Six models (4 proprietary, 2 open-weight):
- Proprietary: Anthropic Claude Opus 4.6, Google Gemini 3.1 Pro, OpenAI GPT-5.2, xAI Grok-4.1.
- Open-weight: OpenGVLab InternVL-3.5-38B, Alibaba Qwen3-VL-32B-Instruct.
Prompting: A forced-choice paradigm was used. Models were presented with image pairs (labeled A and B) and asked to select the superior image for each attribute with a rationale.
Consistency Check: Each image pair was evaluated three times per model to measure stochasticity and internal consistency.
Data Processing: Responses were parsed, and preference scores were converted to z-scores to match the human data format.

C. Analysis Framework

The study analyzed performance across three axes:

Intra-Model Variability: Consistency of a single model across repeated runs.
Cross-Model Variability: Agreement between different VLMs.
Human-VLM Alignment: Correlation between VLM rankings and human psychophysical rankings (Spearman's $\rho$ ).
Attribute Weighting: Regression analysis to determine how models weight contrast vs. colorfulness when judging overall preference.
Perceptual Separability: Analyzing how agreement changes based on the difficulty (separability) of the scene.

3. Key Contributions

First Systematic Benchmark: Provides the first direct comparison of six state-of-the-art VLMs against established psychophysical IQ data.
Attribute-Dependent Variability: Demonstrates that VLM performance is highly specific to the attribute being judged; no single model excels at all dimensions.
Trade-off Discovery: Identifies a counterintuitive trade-off where the most internally consistent models are not necessarily the most human-aligned, suggesting that some response variability may actually reflect human-like sensitivity to context.
Separability Insight: Shows that VLM reliability is strongly correlated with perceptual separability; they perform well when image differences are obvious but struggle with subtle distinctions.

4. Key Results

A. Intra-Model Consistency

Claude showed the highest internal consistency (lowest variability, e.g., <2% for contrast).
Intern showed the highest variability (up to 17.78% for contrast), indicating high stochasticity.
Observation: High consistency does not guarantee human alignment.

B. Human-VLM Alignment (Spearman's $\rho$ )

Performance varied significantly by attribute:

Colorfulness: Claude and Qwen achieved the highest correlation ( $\rho \approx 0.93$ ). Intern also performed well ( $\rho \approx 0.92$ ).
Contrast: Qwen ( $\rho = 0.86$ ) and Gemini ( $\rho = 0.79$ ) led. Intern performed poorly ( $\rho = 0.18$ ).
Overall Preference: GPT achieved the highest alignment ( $\rho = 0.86$ ) and correctly identified the Top-1 rendering model.
Top-1 Accuracy: Only GPT, Gemini, and Grok correctly predicted the Top-1 model for Overall Preference.

C. Attribute Weighting Analysis

Human Behavior: Humans weigh colorfulness more heavily than contrast when forming an overall preference.
VLM Behavior:
- Qwen, Gemini, Intern: Mimic human behavior by weighting colorfulness heavily.
- GPT: Adopts a more balanced strategy but leans slightly toward contrast.
- Grok: Heavily relies on contrast, ignoring colorfulness, diverging from human weighting.

D. Perceptual Separability

Trend: Agreement between VLMs and humans increases as the perceptual separability of the scene increases (i.e., when differences between images are clear).
Failure Mode: In scenes with low separability (subtle differences), agreement drops significantly, and some models (like Intern) even show negative correlation, suggesting they rely on different cues than humans in ambiguous scenarios.

5. Significance and Implications

Rapid Screening: VLMs are viable for rapid hypothesis testing and screening large datasets where image differences are distinct. They can accelerate algorithm development by filtering out poor candidates before costly human studies.
Not a Replacement: VLMs cannot yet replace psychophysical experiments for rigorous, fine-grained perceptual evaluation, particularly for subtle quality differences.
Evaluation Strategy: Researchers must use attribute-aware evaluation. A single aggregate score is insufficient; models must be benchmarked separately for contrast, colorfulness, and overall preference.
Model Selection:
- Use GPT for holistic preference tasks.
- Use Claude/Qwen for colorfulness assessment.
- Use Qwen/Gemini for contrast assessment.
Future Work: The paper suggests that ensemble methods or fine-tuning VLMs on psychophysical data could bridge the gap between model stability and human alignment.

Conclusion

The study concludes that while VLMs show promise as a scalable tool for perceptual IQA, their performance is attribute-dependent and context-sensitive. The most consistent models are not always the most human-aligned, and reliability is contingent on the clarity of perceptual differences in the stimulus.