💬 NLP

Images Amplify Misinformation Sharing in Vision-Language Models

This study reveals that vision-language models exhibit human-like biases by significantly increasing their propensity to reshare false news when images are present, with this vulnerability further amplified by specific persona traits and varying across different model architectures.

Original authors: Alice Plebe, Timothy Douglas, Diana Riazi, R. Maria del Rio-Chanona

Published 2026-04-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alice Plebe, Timothy Douglas, Diana Riazi, R. Maria del Rio-Chanona

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a group of very smart, super-fast robots (called Vision-Language Models, or VLMs) that read news and decide what to share with their friends. These robots can read text and look at pictures.

The researchers in this paper wanted to answer a simple question: Does adding a picture to a news story make these robots more likely to share it, even if the story is fake?

Think of it like this: If you tell a friend a boring rumor, they might ignore it. But if you show them a dramatic photo of the rumor happening, they are much more likely to believe it and tell everyone else. This paper asks if robots do the exact same thing.

Here is what they found, broken down simply:

1. The "Magic Picture" Effect

The researchers tested four different types of robots. They gave them news stories that were either True or False.

The Experiment: They showed the robots the stories in two ways: just text, and text with a picture.
The Result: When a picture was added, the robots shared the stories more often.
The Catch: The robots were much more likely to share fake news when a picture was attached. It was like the picture acted as a "trust booster" that tricked the robots into thinking a lie was true.
- For fake news, sharing went up by about 14.5%.
- For real news, sharing only went up by 5.3%.

Analogy: Imagine a magician. If they tell you a trick is real, you might doubt them. But if they show you a flashy picture of the trick, you are much more likely to believe it and tell your friends about it. The robots fell for the flashy picture, especially when the trick was a lie.

2. Not All Robots Are the Same

The researchers tested four different robot models (like different brands of smartphones).

Some robots (like GPT-4o-mini and Qwen2-VL) were very easily fooled by the pictures. They shared fake news with pictures almost 20% more often than without them.
One robot (Claude-3-Haiku) was much tougher. It didn't get fooled as easily by the pictures. It was the most "skeptical" of the group.

3. The "Personality" Test

The researchers also gave the robots different "personalities" to see if that changed their behavior. They pretended the robots were different types of people:

The "Dark" Personalities: They gave some robots traits like being manipulative, selfish, or reckless (called the "Dark Triad").
- Result: These "dark" personalities shared fake news much more often. They didn't care if it was true; they just wanted to share it.
The "Political" Personalities: They gave some robots a specific political identity (like being a Republican or a Democrat).
- Result: The robots with a Republican identity became less careful about checking if news was true or false. They shared both real and fake news at about the same rate, whereas other robots were better at spotting the difference.

Analogy: Imagine a party. If you ask a shy, cautious person to share a rumor, they might check if it's true first. But if you ask a bold, reckless person, they might shout the rumor out immediately, even if it's made up. The robots acted exactly like these different types of people.

4. What Didn't Matter

The Topic: Whether the news was about politics, health, or technology didn't change the robots' behavior much.
People in the Photo: It didn't matter if the picture showed a human face or just an object; the robots reacted the same way.

The Big Takeaway

The main conclusion is that these AI robots are not just cold calculators. They have a human-like flaw: they trust pictures too much.

Just like humans, these robots are more likely to believe and share a story if it has a picture attached to it, even if the story is a lie. This is especially dangerous when the robot is given a "reckless" personality or when the news is fake.

The researchers warn that as these robots start helping to curate news for us (deciding what we see on our screens), we need to be careful. If they are programmed to share things that look good (because of pictures) rather than things that are true, they could accidentally spread a lot of fake news.

In short: Pictures make robots more likely to share news, but they make them share fake news even more than real news. And if you give the robot a "bad attitude," it shares the fake stuff even faster.

1. Problem Statement

As Vision-Language Models (VLMs) become integral to information curation and social media recommendation systems, there is a critical gap in understanding how they process multimodal misinformation. While human studies confirm that the presence of images significantly increases the perceived credibility and shareability of false information (the "truthiness" effect), it remains unknown whether VLMs exhibit similar biases. Furthermore, it is unclear how persona conditioning (simulating specific personality traits or demographics) interacts with visual cues to influence a model's decision to reshare content. Current safety mechanisms often cause VLMs to refuse engaging with controversial or conspiratorial topics, masking their true behavioral tendencies under specific conditions.

2. Methodology

The authors designed a controlled experimental framework to evaluate four state-of-the-art VLMs: GPT-4o-mini, Claude-3-Haiku, LLaVA-1.6, and Qwen2-VL.

A. Dataset Construction

Source: A novel multimodal dataset was curated from PolitiFact, containing 500 fact-checked political news items (250 true, 250 false).
Criteria: Items were selected to be post-training-cutoff (to prevent memorization), paired with original images, and annotated with ground-truth veracity labels.
Attributes: Each item was tagged with topics (e.g., politics, economy) and image content type (people vs. no people).

B. Prompting Strategy (Jailbreaking-Inspired)

To overcome safety refusals when simulating antisocial traits or discussing controversial news, the authors developed a third-person prompting strategy:

Framing: Instead of asking the model directly ("Would you share this?"), the prompt asks the model to simulate a hypothetical user with specific traits ("A user with profile X is reading this...").
Chain-of-Thought (CoT): The model is instructed to reason step-by-step: identify the central message, analyze supporting image elements, evaluate alignment with the persona, and finally decide on resharing.
Output: The model rates its likelihood to reshare on a 5-point Likert scale (1 = Strongly Disagree to 5 = Strongly Agree).

C. Experimental Variables

Modality: Text-only vs. Text + Image.
Persona Conditioning:
- Personality Traits: Big Five (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) and Dark Triad (Narcissism, Machiavellianism, Psychopathy).
- Demographics: Sex, Age, Race/Ethnicity, and Political Affiliation (Democrat/Republican).
Models: Two proprietary (GPT-4o-mini, Claude-3-Haiku) and two open-weight (LLaVA-1.6, Qwen2-VL).

D. Statistical Analysis

Likert scores were mapped to binary decisions (Share/Not Share).
Wilcoxon signed-rank tests compared text-only vs. image conditions.
Linear Mixed-Effects Models analyzed interactions between modality, veracity, and persona.
Fleiss' $\kappa$ measured inter-completion agreement.

3. Key Contributions

First Systematic Study of Visual Bias in VLMs: The paper provides the first empirical evidence that VLMs replicate human-like "truthiness" biases, where images increase the propensity to reshare misinformation.
Novel Prompting Technique: The development of a third-person, jailbreaking-inspired prompt strategy that successfully bypasses safety refusals to elicit persona-consistent behaviors, particularly for antisocial traits.
Multimodal Dataset: Creation of a high-quality, fact-checked, image-text paired dataset specifically designed to test VLMs on recent political news, avoiding training data contamination.
Persona-Modality Interaction Analysis: A comprehensive analysis of how personality traits (especially the Dark Triad) and political alignment modulate the visual amplification of misinformation.

4. Key Results

A. The "Visual Amplification" Effect (RQ1)

Overall Impact: The presence of an image increased the resharing rate by 14.5% for false news and 5.3% for true news.
Disproportionate Risk: The effect is significantly stronger for misinformation. VLMs are not just more likely to share with images; they are disproportionately more likely to share false content when images are present.
Architectural Validation: A control experiment using "blank images" (all-zero pixels) confirmed that the effect is due to the semantic content of the image, not merely the activation of a multimodal processing pathway.

B. Model Variability (RQ2)

Sensitivity Differences:
- Most Affected: GPT-4o-mini and Qwen2-VL showed the strongest amplification effects (approx. 20% increase for false news).
- Most Robust: Claude-3-Haiku demonstrated the greatest resistance to visual misinformation, likely due to its conservative alignment strategy (Constitutional AI).
Response Patterns: Qwen2-VL exhibited a strong affirmative bias (often selecting "Strongly Agree"), while LLaVA-1.6 showed low internal consistency and indecision.

C. Persona Conditioning (RQ3)

Dark Triad Traits: Traits such as Psychopathy and Narcissism consistently amplified the resharing of false news across all models. This mirrors human psychological findings where these traits correlate with higher misinformation susceptibility.
Political Affiliation: Republican-aligned profiles showed reduced sensitivity to veracity, sharing true and false news at similar rates. Democratic profiles maintained a higher distinction between true and false content.
Demographics: Political affiliation and race had significant effects on specific models, whereas age and sex had negligible impacts.

D. Content Attributes (RQ4)

Veracity: The strongest predictor of sharing behavior was the actual truthfulness of the news (models generally preferred true news).
Topic & Image Content: News topic had a marginal effect. Whether an image depicted people or not had no measurable effect on resharing decisions.

5. Significance and Implications

Systemic Risk: As VLMs are increasingly deployed in content recommendation and curation pipelines, their susceptibility to visual misinformation poses a threat to public discourse. They may systematically amplify visually rich falsehoods, especially when personalized to users with specific personality profiles.
Alignment Limitations: The study highlights that safety alignment (e.g., in Claude-3-Haiku) can mitigate visual bias but may not fully suppress persona-driven biases. Conversely, models with strong reasoning capabilities (GPT-4o) may still be highly susceptible to visual manipulation.
Future Directions: The authors call for:
- Evaluation frameworks that explicitly test multimodal inputs and persona contexts.
- Architectural separation of fact-checking mechanisms from persona conditioning in AI systems.
- Stricter verification thresholds for image-text content compared to text-only content.

This research fundamentally shifts the understanding of AI misinformation from a purely textual phenomenon to a multimodal vulnerability, demonstrating that VLMs are not immune to the same cognitive biases that plague human information consumption.