Visual Persuasion: What Influences Decisions of Vision-Language Models?

This paper introduces a framework that treats VLM decision-making as a latent visual utility to be inferred through revealed preference, utilizing systematic image perturbations and visual prompt optimization to identify consistent visual themes driving agent choices and proactively surface safety vulnerabilities in image-based AI systems.

The Gist

Imagine you are walking through a giant, invisible art gallery where the paintings aren't made by humans, but by AI agents. These agents are like super-fast, super-smart shoppers, recruiters, and real estate agents who make millions of decisions every second based on what they "see" in images.

This paper, titled "Visual Persuasion," asks a scary but fascinating question: Can we trick these AI agents into making different choices just by changing the "lighting" or "background" of a picture, without changing the actual object?

Here is the story of how they found out, explained simply.

1. The Setup: The "Magic Mirror"

The researchers started with a simple wooden chair.

• The Original: A boring photo of the chair on a white background.
• The AI's Job: They asked an AI agent, "Which chair would you buy?" The agent looked at the boring chair and said, "Meh, not really."

Then, they used a Magic Mirror (an AI image generator) to change the chair's surroundings. They didn't change the chair itself; they just put it in a beautiful, sun-drenched Mediterranean setting with a pool and olive trees.

• The Result: Suddenly, the AI agent loved the chair! It was now 2 to 3 times more likely to be "chosen."

The Analogy: Think of it like a job interview. If you wear a t-shirt and sit in a messy room, you might not get the job. If you wear a sharp suit and sit in a sleek office, you get the job—even if you are the exact same person with the exact same skills. The AI agents are surprisingly sensitive to the "outfit" and "room" of the image.

2. The Experiment: Teaching the AI to "Paint"

The researchers didn't just guess what looked good. They built a feedback loop that acted like a relentless art teacher.

1. The Artist: An AI generates a new version of the image (e.g., "Add a sunset").
2. The Judge: Another AI looks at the new image and the old one and picks a winner.
3. The Critic: The Judge tells the Artist why it won (e.g., "The sunset made it look warmer and more inviting").
4. The Loop: The Artist uses that feedback to make the next image even better.

They did this over and over again. It's like a game of "Hot and Cold." The AI keeps tweaking the image until it finds the perfect "recipe" that makes the decision-maker say, "Yes, I want this one!"

They found three different ways to play this game (called CVPO, VFD, and VTG), but the "Competitive" one (CVPO) was the best at finding the winning formula.

3. The Discovery: The "Hidden Cheat Codes"

After running this experiment on thousands of images (houses, people, products, hotels), they discovered something huge: AI agents have very specific, predictable "visual cravings."

They used a special tool to read the AI's mind and found the "cheat codes" that worked every time:

• For Hotels: The AI loved images with plants, warm golden lighting, and people in the background. It made the hotel feel "lived-in" and luxurious.
• For Houses: The AI preferred houses shown at sunset (golden hour) with manicured lawns and no power lines in the way.
• For Job Candidates: The AI wanted to see people in business suits, smiling, sitting in an office, not a messy bedroom.
• For Products: The AI wanted products shown in a lifestyle setting (e.g., a coffee maker on a nice counter with a cup of coffee next to it), not just floating in a white void.

The Metaphor: It's like discovering that a specific type of fish always bites on a red worm, regardless of the water temperature. The researchers found the "red worm" for AI decision-making.

4. The Human Test: Do We Fall for It Too?

The researchers then asked real humans to look at the same pictures.

• The Result: Humans also preferred the "optimized" images!
• The Catch: While humans liked the pretty pictures, the AI agents were even more easily swayed than humans. The AI's preference was much stronger and more consistent.

This suggests that if someone knows how to "game" the AI's visual preferences, they could manipulate the AI into picking a worse product, a less qualified candidate, or a more expensive house, simply by making the image look slightly better.

5. The Solution: The "Neutralizer"

The researchers tried to fix this by creating a "Neutralizer." Before the AI makes a choice, they force the two images to be stripped of their fancy backgrounds and lighting, making them look as similar as possible.

• Did it work? It helped a little, but not completely. The AI still had a slight preference for the "pretty" version. It's like trying to ignore a delicious smell while eating; it's hard to do perfectly.

Why Does This Matter?

This paper is a wake-up call.

• The Risk: If companies know these "visual cheat codes," they could manipulate AI agents to favor their products unfairly. Imagine a real estate agent AI being tricked into recommending a house just because the photo has a sunset, even if the house is a dump.
• The Benefit: Now that we know how these agents think, we can build better safety checks. We can teach AI to look past the "pretty packaging" and focus on the real facts.

In a nutshell: The world is full of AI agents making decisions based on pictures. This paper shows that these agents are easily "persuaded" by lighting, backgrounds, and styling, just like humans, but often even more so. We need to understand these tricks to make sure the AI is making fair choices, not just pretty ones.

Technical Summary

1. Problem Statement

Vision-Language Models (VLMs) are increasingly deployed as autonomous agents making consequential decisions based on visual inputs (e.g., selecting products, hiring candidates, or choosing real estate). While current evaluations focus on accuracy (object recognition, instruction following), they largely ignore behavioral preferences.

The core problem is that VLMs may possess latent, systematic visual sensitivities that differ from human rationality. These agents can be manipulated by superficial but plausible changes to image presentation (lighting, background, context) without altering the semantic identity of the object. If these sensitivities are not understood, they can be exploited at scale to bias agent decisions, potentially shifting visual culture and market outcomes toward manipulated preferences rather than intrinsic value.

2. Methodology

The authors propose a framework to treat an agent's decision function as a latent visual utility landscape that can be explored and mapped through Visual Prompt Optimization (VPO). Instead of adversarial perturbations (imperceptible to humans), the method uses naturalistic image edits to maximize selection probability.

A. Visual Prompt Optimization (VPO)

The process iteratively modifies an image $x_0$ using a text-to-image editing model guided by a prompt $p$. The goal is to find a prompt that maximizes the utility $U_\tau(x(p))$ defined by a VLM evaluator, subject to identity constraints (the core object/scene must remain recognizable).

Three specific optimization algorithms were developed and tested:

1. VisualTextGrad (VTG): Adapts the TextGrad algorithm. An LLM critic scores the current image, generates structured feedback, and computes a "gradient" direction to update the text prompt.
2. VisualFeedbackDescent (VFD): Based on Feedback Descent. A proposer model generates candidate prompts based on history and feedback. These are evaluated via pairwise comparison against the incumbent.
3. Competitive Visual Prompt Optimization (CVPO): A novel method framing optimization as a competitive selection process. Two candidate prompts compete in a panel of judges (VLMs). The loser is refined based on judge feedback to generate new challengers. This continues until an equilibrium is reached.

B. Experimental Setup

• Datasets: Four realistic agentic tasks: Product Purchasing (Amazon Berkeley Objects), House Searching (price estimation dataset), Candidate Hiring (StyleGAN-Human synthetic portraits), and Hotel Scouting (aesthetic properties dataset).
• Scale: 100 images per dataset, optimized over multiple rounds.
• Evaluators: 9 frontier VLMs (including GPT-4o, GPT-5, Gemini 3, Claude 3.5/4.5, Llama 4, Qwen-VL).
• Human Validation: Online experiments with $N=154$ participants to compare human vs. agent preferences.

C. Interpretability & Mitigation

• Auto-Interpretability: A "Matryoshka" summarization pipeline. The system extracts visual differences between original and optimized images, clusters them, and recursively summarizes them into high-level themes (e.g., "Biophilic integrations," "Twilight lighting").
• Mitigation: A strategy called Image Normalization, where a model is instructed to align the visual contexts of two competing images (removing irrelevant differences) before the VLM makes a decision, to test if vulnerabilities can be neutralized.

3. Key Contributions

1. Empirical Evidence of Visual Sensitivity: Demonstrated that VLMs exhibit strong, systematic biases toward specific visual presentations (e.g., warm lighting, lush landscaping, professional attire) even when semantic content is held constant.
2. Visual Prompt Optimization Framework: Introduced CVPO, VFD, and VTG as methods to systematically exploit these sensitivities, shifting choice probabilities significantly.
3. Benchmarking: Created a benchmark of 9 frontier VLMs across 4 tasks, showing that optimized edits can double or triple the probability of an image being selected.
4. Human vs. Agent Alignment: Found that while optimized images also shift human choices, the specific visual themes that maximize agent selection are not always identical to those that maximize human selection, highlighting a "machine fluency" gap.
5. Auto-Interpretability Pipeline: Developed a method to hierarchically surface the specific visual themes (e.g., "Golden hour lighting," "Removal of visual clutter") that drive agent decisions.
6. Mitigation Analysis: Showed that image normalization can partially mitigate these biases but does not fully eliminate them, suggesting inherent fragility in current VLM agents.

4. Key Results

• Effectiveness of Optimization:
  • Zero-shot edits (single-step improvements) already increased selection probability by 0.2–0.4 relative to originals.
  • Iterative optimization (CVPO/VFD) yielded further gains of +0.1–0.3.
  • CVPO was the most effective strategy, outperforming VFD and VTG in head-to-head comparisons across most models (e.g., on Qwen-VL, CVPO achieved 77.1% choice probability vs. 13.1% for VTG).
• Model Sensitivity:
  • All tested models were susceptible, but the magnitude varied. Anthropic models (Claude) showed slightly more resistance to certain optimizations compared to others, but were still significantly influenced.
  • Table 1 in the paper shows that for many models, the "Final" optimized image is chosen with >70% probability against the original.
• Interpretability Findings:
  • Hotels: Preferences for "Biophilic integrations" (plants), "Luxury furniture," and "Warm ambient lighting."
  • Houses: Preferences for "Twilight lighting," "Lush landscaping," and "Removal of visual clutter" (power lines, cars).
  • People: Strong preference for "Formal business attire," "Corporate backgrounds," and "Professional expressions."
  • Products: Preference for "Lifestyle environments" (contextual props) and "Human subject integration."
• Human Correlation: Humans also preferred optimized images, but the effect size was generally smaller than for VLMs, and the specific winning strategies were not perfectly aligned (e.g., VTG performed better for humans in some tasks where it failed for agents).
• Mitigation: Image normalization (3 passes) reduced the advantage of optimized images but did not eliminate it, and increased decision inconsistency (order effects), suggesting the bias is deeply embedded.

5. Significance and Implications

• Safety & Governance: The work reveals that VLM agents are vulnerable to "visual persuasion" that does not require adversarial attacks. This poses risks in high-stakes domains like hiring, lending, and real estate, where agents could be manipulated to favor specific entities based on aesthetic manipulation rather than merit.
• Methodological Shift: Argues that behavioral systems require behavioral tests (preference elicitation) rather than just accuracy benchmarks.
• Machine Fluency: Highlights a new form of inequality where actors with "machine fluency" (the ability to craft prompts that exploit agent biases) can gain unfair advantages over those who do not.
• Future Directions: Suggests that robust AI agents require visual normalization, explicit checks for irrelevant cues, and training to recognize these manipulated visual patterns, similar to deepfake detection.

In conclusion, the paper establishes that VLMs have a distinct, exploitable "visual utility function" that can be reverse-engineered through iterative optimization, necessitating new frameworks for auditing and governing image-based AI agents.