💬 NLP

Do Vision-Language Models Understand Visual Persuasiveness? A Diagnosis via Visual Persuasive Factors

This paper diagnoses the limitations of Vision-Language Models in assessing visual persuasiveness, revealing their tendency to over-predict persuasion due to a recall-oriented bias and a failure to properly align object identification with semantic context, while demonstrating that performance can be improved through targeted Visual Persuasive Factors (VPFs) interventions.

Original authors: Gyuwon Park, Hyounghun Kim

Published 2026-09-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gyuwon Park, Hyounghun Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world, images are rarely just decorations. From public health warnings on cigarette packs to political campaign posters, visual elements are engineered to shape how we think, feel, and act. This process, known as visual persuasion, relies on a complex interplay between what we see and the message we read. A bright, cheerful photo might make a safety tip feel encouraging, while a dark, cluttered one could make the same tip feel threatening. For decades, psychologists have studied how humans process these cues, understanding that our brains weigh colors, the placement of objects, and the presence of people to decide if a message is convincing. Now, as artificial intelligence systems become capable of seeing and reading images simultaneously, a critical question has emerged: do these machines understand persuasion the way humans do, or are they simply guessing based on surface-level patterns?

A team of researchers at POSTECH in South Korea set out to answer this by putting modern vision-language models to a rigorous test. These are advanced computer systems that can look at an image and read a text message, then attempt to understand how well the picture supports the text. The researchers started with a carefully curated collection of 562 image-and-message pairs. These were not random internet finds; they were selected because human judges had already looked at them and agreed almost perfectly on whether each pair was highly persuasive or not persuasive. This created a clear "ground truth" against which the computer models could be measured. The goal was to see if the machines could replicate human judgment or if they were missing something fundamental about how visual persuasion works.

The results revealed a striking and consistent flaw in how these artificial intelligence systems operate. When asked to judge persuasiveness, the models exhibited a strong bias toward saying "yes." They were incredibly good at catching the images that humans found persuasive, but they were also far too eager to label non-persuasive images as convincing. In technical terms, they achieved high "recall" but suffered from a flood of false positives. Essentially, the machines were over-predicting persuasion, treating almost any image that contained a plausible visual element as a successful argument. They seemed to lack the human ability to discern when a visual cue was merely present versus when it was actually doing the work of persuasion.

To understand why this was happening, the researchers broke down the visual world into specific, measurable components they called Visual Persuasive Factors. These factors ranged from the immediate sensory impact of an image, such as its colorfulness and brightness, to its composition, like whether the main subject was placed in the center or at the intersection of imaginary grid lines. They also looked at semantic elements, such as whether a key object, a human figure, or a piece of text was visible. When the researchers compared how humans and machines weighed these factors, a clear divergence appeared. Humans used these cues with nuance, understanding that a bright image might be persuasive for a positive message but not for a negative one. The machines, however, often treated the mere presence of a cue as a guarantee of success. For instance, if an image contained a human face or a specific object mentioned in the text, the models were likely to declare it persuasive, even if the overall context suggested otherwise. They were mistaking the ingredients of a recipe for the finished dish.

The study then explored whether giving the machines more explicit instructions could fix this problem. The researchers tried two main approaches. First, they fed the models detailed knowledge about the visual factors, telling them, for example, that the presence of a person should be treated as a supporting clue rather than a deciding factor. Second, they asked the models to think step-by-step, breaking down their reasoning before making a final judgment. The findings here were nuanced. Providing the models with the right context—specifically, framing these visual cues as something to be interpreted rather than a fixed rule—did help reduce the number of errors. However, simply asking the models to reason through the problem or pointing out the presence of an object was not enough. In some cases, forcing the models to reason with explicit cues actually made their bias worse, causing them to double down on incorrect judgments.

The core issue, the researchers discovered, lay in a specific bottleneck in the machine's reasoning process. When the team analyzed the step-by-step explanations the models generated, they found that the machines were generally good at identifying objects and describing the text. They could also explain how an object might relate to a message. The failure occurred in the final step: connecting that evidence to the ultimate goal of communication. The models struggled to decide whether the visual evidence they had found actually supported the intended message or if it was just a coincidence. They could see the parts, but they could not reliably synthesize them into a coherent judgment of intent.

This research suggests that while artificial intelligence has made tremendous strides in recognizing what is in an image, it still lacks a deep, contextual understanding of why that image matters. The machines are currently excellent at spotting the ingredients of persuasion but poor at tasting the final product. They tend to assume that if the right visual elements are present, the message must be persuasive, ignoring the subtle human ability to weigh context, tone, and intent. The study concludes that for these systems to truly understand visual persuasion, they need to move beyond simply identifying objects and learning to connect those observations to the communicative purpose behind them. Until they can make that leap, they will remain prone to seeing persuasion where there is none, mistaking the presence of a cue for the power of a message.

Technical Summary: Do Vision–Language Models Understand Visual Persuasiveness?

Problem Statement

Visual persuasion involves using images to shape cognition, emotion, and behavior, where effects depend on the interplay between visual attributes and semantic context. While Vision–Language Models (VLMs) have advanced in multimodal understanding, it remains unclear whether they genuinely comprehend visual persuasiveness. Specifically, it is unknown if VLMs can accurately assess whether an image supports an intended message, which visual factors drive this judgment, and how their reasoning aligns with human consensus. Existing work provides limited insight into the specific visual factors VLMs utilize versus those used by humans, particularly regarding why certain images are persuasive and how models might misinterpret visual cues.

Methodology

The authors propose a framework to diagnose VLM understanding of visual persuasiveness through the lens of Visual Persuasive Factors (VPFs), a taxonomy informed by cognitive psychology.

1. Dataset and Evaluation Setting

The study utilizes the Personalized Visual Persuasion (PVP) dataset. To isolate the effect of visual factors and minimize subjectivity, the authors filter for a high-consensus subset of 562 image-message pairs where human raters achieved "almost-perfect" agreement (Landis and Koch, 1977) on binary persuasiveness labels (High vs. Low). This subset covers 20 distinct topics and includes both DALL·E-generated and web-retrieved images.

2. Visual Persuasive Factors (VPFs)

VPFs are defined as a three-level decomposition of persuasive content:

Perceptual Level: Captures immediate visual impressions, including colorfulness, color entropy, representative color congruence (alignment between image color and message affective framing), and brightness (and its alignment with persuasion strategy polarity).
Compositional Level: Describes spatial organization, measuring Center Bias (concentration of visual attention near the image center) and Rule of Thirds (RoT) alignment for both saliency maps and Message-Relevant Regions (MRR) derived from open-vocabulary segmentation.
Semantic Level: Captures the presence of key objects (nouns from the message), humans, and text.

3. Experimental Design

Baseline Evaluation: VLMs (GPT-5, GPT-5-mini, Gemma3, Qwen2.5-VL) are evaluated on the binary classification task (persuasive vs. not persuasive) against human ground truth.
Factor-Level Analysis: The authors compare the association of VPFs with human labels versus VLM predictions using logistic regression and effect size metrics (Cohen's $d$ and Odds Ratios).
Intervention Studies: To address identified biases, the authors test:
- Knowledge Injection: Providing explicit VPF definitions or key object contexts to the model.
- Reasoning Scaffolds: Using Chain-of-Thought (CoT) prompting, Knowledge-Conditioned Chains, and Key Object-Aware CoT.
Rationale Ablation: Model-generated rationales are parsed into three functional steps—(S1) Object Identification, (S2) Semantic Message Support, and (S3) Final Judgment Connection. The authors perform step-wise ablations to determine which reasoning step contributes most to errors.

Key Contributions

VPF Taxonomy: The introduction of Visual Persuasive Factors, a structured, cognitive-psychology-informed framework for quantifying visual cues that shape persuasive judgments.
Diagnostic Framework: A systematic comparison showing that while VPFs effectively distinguish human judgments, VLMs only partially reproduce these patterns.
Identification of Bottlenecks: The discovery that VLMs suffer from a recall-oriented bias (over-predicting persuasiveness) and a failure in persuasive integration—specifically, the inability to correctly connect visual evidence (objects, text) to communicative intent.

Key Results

1. Recall-Oriented Bias

VLMs exhibit a pronounced recall-oriented bias. They achieve near-perfect recall (identifying almost all truly persuasive images) but suffer from inflated false positives, predicting "High" persuasiveness far too often. This suggests models struggle to distinguish between images that merely contain plausible cues and those that genuinely support the message.

2. Partial Reproduction of Human Patterns

Alignment: VLMs capture some human-aligned VPF patterns (e.g., representative color congruence and key object presence are associated with predicted High labels).
Misalignment: VLMs fail to weight other factors correctly. For instance, text presence is associated with human "Low" persuasiveness in some contexts but is treated as neutral or "High" by VLMs. Similarly, models show inconsistent weighting for human presence and brightness strategies compared to humans.

3. Overgeneralization of Cues

False positives arise because VLMs overgeneralize persuasion-relevant cues. When ground-truth "Low" images contain plausible cues (e.g., a key object or human), models often mistake their mere presence as sufficient evidence for persuasiveness, failing to evaluate whether the cue actually supports the specific message.

4. Intervention Limitations

Knowledge Injection: Explicitly framing VPFs as "context-dependent auxiliary cues" (rather than fixed rules) helps reduce false positives and improve F1 scores. However, simply providing key object context without framing leads to performance degradation.
Reasoning Scaffolds: Standard Chain-of-Thought prompting is insufficient. In some cases (e.g., Key Object-Aware CoT), explicit reasoning amplifies the recall-oriented bias, suggesting models struggle to integrate cues into a final judgment even when prompted to reason step-by-step.

5. Rationale Analysis

Step-wise ablation reveals that the primary bottleneck lies in S3 (Final Judgment Connection). Removing S1 (Object ID) or S2 (Semantic Alignment) often leaves accuracy unchanged or even improves it, but removing S3 directly hurts performance. Crucially, ablating the combination of S1 and S2 (the evidence generation) often improves accuracy, indicating that the models' intermediate reasoning steps (linking objects to messages) are flawed and lead to overconfident, incorrect conclusions.

Significance and Claims

The paper claims that VLMs do not yet possess a robust notion of visual persuasive understanding. The central limitation is not the identification of visual elements but the integration of visual evidence with communicative intent.

Diagnostic Value: The VPF framework provides a structured way to diagnose why models fail, moving beyond simple accuracy metrics to understand specific cognitive misalignments.
Intervention Insight: The study demonstrates that simply adding more visual cues or reasoning steps does not solve the problem; the framing of these cues as context-dependent rather than deterministic is critical.
Future Direction: The authors posit that closing the gap requires models to reliably connect object-level and semantic evidence to the specific communicative goal of the image-message pair, rather than treating the presence of persuasive cues as a proxy for persuasiveness itself.

The authors maintain a modest scope, acknowledging that VPFs are a behavioral diagnostic framework for high-agreement scenarios and do not yet model individual personality traits or complex cross-factor interactions, which remain areas for future work.