🤖 AI

Persona Conditioning of Brand Recommendations in Retrieval-Augmented Commercial Chat: A Prominence-Stratified Cross-Provider Audit

This study audits how AI assistants' brand recommendations for commercial queries vary significantly across different buyer personas, revealing that while category leaders remain consistent, mid-market brand suggestions shift dramatically based on the user's context, with the effect being most pronounced in models that rely more heavily on training-data priors than retrieval evidence.

Original authors: Will Jack, Noah Lehman, Keller Maloney, Sarah Xu

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Will Jack, Noah Lehman, Keller Maloney, Sarah Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you walk into a very smart, well-read travel agent's office. You ask the same question: "What's the best hotel in Paris?"

If you tell the agent, "I'm a solo backpacker with a tiny budget," they might recommend a cozy, cheap hostel. If you tell them, "I'm a CEO of a massive tech company looking to host a board meeting," they might recommend a luxury palace.

This paper asks: How much does the answer actually change just because you changed who you said you were?

The researchers ran a massive experiment where they asked AI chatbots (like ChatGPT and Claude) the same business questions (e.g., "What's the best CRM software?") but pretended to be 10 different types of people—from a solo startup founder to a big corporate VP. They wanted to see if the AI's list of recommended brands would stay the same or completely swap out depending on the "persona" of the asker.

Here is what they found, broken down simply:

1. The "Who You Are" Effect is Real and Big

The study found that changing the persona significantly changes the list of brands the AI recommends.

The Analogy: Imagine the AI's list of recommendations is a playlist. When you ask as a "solo founder," the playlist has 10 songs. When you ask as a "corporate VP," about 20% to 30% of those songs are completely different.
The Numbers: The researchers measured this using a "similarity score." When the same person asked the same question twice, the answers were about 50% similar. But when two different people asked the same question, the answers dropped to only about 22–35% similar. That's a huge shift.

2. The "Famous vs. Mid-Tier" Rule

The most interesting part is which brands change and which ones stay the same. The researchers sorted brands into five levels of fame (from "Superstars" to "Local Shops").

The Superstars (L1): These are the most famous brands (like Salesforce or HubSpot). They are immune to the persona. Whether you ask as a kid or a CEO, the AI almost always recommends these top brands. They are the "safe bet" that never changes.
The Local Shops (L5): These are regional brands. The AI only recommends them if your persona matches their location (e.g., a UK persona gets UK brands). This is a mechanical change based on geography.
The "Mid-Market" Sweet Spot (L3): This is where the magic happens. These are the solid, middle-tier companies. This is where the AI changes its mind the most. Depending on who you pretend to be, the AI might swap out 75% of its recommendations for these brands.
- Analogy: Think of the Superstars as the "Main Street" everyone walks on. The Mid-Market brands are the "Side Streets." If you tell the AI you are a tourist, it points you to the main shops. If you tell it you are a local, it points you to the side streets. The AI is most flexible with the side streets.

3. The "Stronger" AI is Actually More Flexible

There was a surprising twist regarding the different AI models tested (OpenAI vs. Anthropic).

The Expectation: Usually, we think "stronger" or "smarter" AI models are more rigid and deterministic (they give the same answer every time).
The Reality: The "stronger" Anthropic model actually changed its recommendations more based on the persona than the OpenAI models did.
The Theory: The authors suggest this might be because the Anthropic model relies more on its internal "memory" (what it learned during training) rather than looking up fresh facts. Since its "memory" contains lots of associations between "types of people" and "brands," it gets very good at tailoring its answer to the specific persona. The OpenAI models, which rely more on looking up current documents, are a bit more stubborn and stick to the same list regardless of who is asking.

4. Why This Matters for Brands

The paper concludes that if you are a brand trying to get noticed by AI, you can't just have one generic "Best Product" page.

The Problem: If you only optimize for a generic "best software" search, you might miss the specific lists generated for different types of buyers.
The Insight: The AI is already acting like a highly personalized salesperson. It doesn't just give a generic list; it curates a list based on who it thinks is asking. If you are a mid-market brand, your visibility in the AI's answer depends entirely on whether the AI thinks the user matches your ideal customer profile.

Summary

The paper proves that AI chatbots are not neutral list-makers. They are highly sensitive to the "persona" of the user. While the most famous brands stay the same no matter what, the middle-tier brands get swapped in and out like a deck of cards depending on whether the AI thinks it's talking to a startup founder, a big corporation, or a small business owner. The "stronger" AI models are actually better at this personalization, making the "who you are" factor even more critical in the future of how people find products online.

Technical Summary: Persona Conditioning of Brand Recommendations in Retrieval-Augmented Commercial Chat

Problem Statement
The paper addresses a critical gap in understanding how Large Language Models (LLMs) handle commercial recommendations in retrieval-augmented chat interfaces. While existing literature has established that persona conditioning affects factual accuracy and closed-form tasks, the impact on open-ended commercial recommendation sets remains unquantified. The core question is empirical: to what extent does the recommendation set shift when only the user persona (e.g., "solo founder" vs. "enterprise VP") varies, holding the prompt string, temperature, and system template constant? The authors posit that if the effect is large, AI commercial recommendation is effectively de facto persona-segmented, rendering single "best [category]" brand strategies insufficient.

Methodology
The study employs a cross-provider audit design sampling 2,000 runs across a controlled design space:

Design Space: 10 hand-curated personas (spanning industry, company size, role, and geography) $\times$ 8 commercially framed prompts $\times$ 3 model configurations.
Model Cells:
- OpenAI gpt-5.4-mini (low reasoning effort)
- OpenAI gpt-5.4-mini (high reasoning effort)
- Anthropic claude-sonnet-4-6 (low reasoning effort)
- Note: The Anthropic cell used only 4 of the 8 prompts used by OpenAI cells due to load constraints.
Execution: Each leaf (persona $\times$ prompt $\times$ cell) was run $N=10$ times.
Consensus Extraction: Brand mentions were extracted by two LLM judges (claude-haiku-4-5 and gpt-5-mini). A brand was counted as recommended only if both judges agreed (intersection mode) on a specific run. The consensus set for a leaf is the union of brands across its 10 reruns.
Metrics:
- Within-Persona Jaccard: Stability of the recommendation set for a single persona across reruns.
- Cross-Persona Jaccard: Similarity between recommendation sets of two different personas (holding prompt and cell fixed).
- Persona-Shift Effect ( $\Delta$ ): Calculated as $Cross\text{-}Persona\ Jaccard - Within\text{-}Persona\ Jaccard$ . A negative $\Delta$ indicates a larger persona effect (more swapping of brands when the persona changes).
- Prominence Stratification: Effects were decomposed by brand prominence levels (L1: Category Leaders to L5: Regional Players).
Statistical Rigor: Effect sizes are reported with prompt-clustered 95% bootstrap confidence intervals (1,000 iterations). The study anchors results against a within-cell rerun-stability baseline (0.50–0.61) established in prior work (Jack [3]) to ensure the persona effect exceeds nondeterministic noise.

Key Results

Significant Persona Shift: Across all three model cells, changing the persona significantly alters the recommendation set. The effect size $\Delta$ $Δ$ ranges from −0.12 to −0.20. All clustered 95% confidence intervals exclude zero, confirming the signal is statistically distinguishable from noise.
- OpenAI gpt-5.4-mini / low: $\Delta = -0.12$
- OpenAI gpt-5.4-mini / high: $\Delta = -0.16$
- Anthropic sonnet-4-6 / low: $\Delta = -0.20$
Prominence-Stratified Interaction: The persona effect is not uniform; it is sharply stratified by brand prominence:
- L1 (Category Leaders): Highly persona-resistant (swap rates 0.20–0.29). Models recommend the same top brands (e.g., Salesforce, HubSpot) regardless of persona.
- L2 (Established Challengers): Mixed signal. OpenAI cells show low swap rates (0.05–0.13), while Anthropic shows higher rates (0.23).
- L3 (Mid-Market): The inflection point where the persona effect peaks. Swap rates reach 0.75 on OpenAI high-effort and 0.67 on low-effort. This indicates mid-market brands are highly sensitive to buyer context.
- L4 (Long-Tail): Undersampled due to low surface rates; no point estimates reported.
- L5 (Regional): Moderate effect (0.27–0.33), largely driven by mechanical geographic gating (e.g., UK personas seeing UK-specific brands).
Provider Heterogeneity: The Anthropic sonnet-4-6 / low cell exhibits a larger point-estimate effect ( $\Delta = -0.20$ ) than the OpenAI configurations. While the confidence intervals for sonnet vs. gpt-5.4-mini / high overlap, the sonnet vs. low contrast is just distinguishable. The authors note this contradicts the prior that "stronger" models are more deterministic, suggesting instead that models relying more on training-data priors (Anthropic, per Jack [2]) may be more responsive to persona context than retrieval-driven models.
Persona Corpus Heterogeneity: "Sharp" personas (e.g., specific roles like "US e-commerce operator") yield high within-persona stability (Jaccard $\ge$ 0.55), while "broad" personas (e.g., "EU finance manager") yield lower stability (Jaccard $\le$ 0.39), indicating that persona specificity dictates recommendation set concentration.

Significance and Claims
The paper claims that AI commercial recommendation is not persona-agnostic. The magnitude of the persona shift ( $\Delta \approx 0.12\text{--}0.20$ ) is comparable to the difference in recommendation sets between different AI providers (Jack [2]) and exceeds the noise floor of model reruns.

Strategic Implication: A single "best [category]" positioning page is an insufficient strategy for brands. Because the recommendation set changes materially based on who is asking, brand optimization must mirror this segmentation, particularly for L3 mid-market brands where the persona effect is maximal.
Mechanism Hypothesis: The authors propose that the larger persona effect in the Anthropic cell is consistent with a mechanism where persona attributes act more strongly on priors-driven generation (training data associations) than on retrieval-driven generation. Retrieval tends to anchor recommendations to authoritative brands regardless of context, whereas priors allow the model to integrate user-archetype attributes more fluidly.
Limitations: The authors explicitly state they do not claim causal evidence for the priors-vs-retrieval mechanism, as they lack within-run joint measurement of retrieval attribution and persona movement. They also note that L4 results are inconclusive due to undersampling and that the study is limited to a single day of measurement and English-language personas.

In conclusion, the paper establishes that persona conditioning is a substantial, measurable variable in commercial chat, with effects concentrated in the mid-market tier, necessitating a shift in how brands approach AI-driven discoverability and positioning.