🤖 AI

Paraphrase Brittleness in Production Retrieval-Augmented Commercial Recommendation: Reproducibility Below the Rerun-Stability Baseline

This paper demonstrates that commercial AI recommendation systems exhibit severe paraphrase brittleness, where minor rewordings of the same buyer intent drastically alter brand visibility, rendering current prompt-based tracking metrics structurally unstable and unreliable for measuring AI-driven brand performance.

Original authors: Will Jack, Noah Lehman, Keller Maloney, Sarah Xu

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Will Jack, Noah Lehman, Keller Maloney, Sarah Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Fragile Glass" of AI Recommendations

Imagine you are a brand manager trying to see if your company is famous. You hire a detective (an AI monitoring tool) to ask a specific question to a giant, all-knowing librarian (the AI model) every week: "What is the best CRM software?"

The detective counts how many times your brand is mentioned in the librarian's answer. If the number goes up, you're happy. If it goes down, you panic.

This paper argues that this entire system is built on shaky ground. The "detective" is measuring the wrong thing because the librarian is incredibly sensitive to exactly how the question is asked.

1. The "Same Question, Different Answer" Problem

The researchers tested what happens when you ask the librarian the same question but change just a few words.

Question A: "What is the best CRM?"
Question B: "What is the top CRM?"
Question C: "What is the best CRM for a SaaS startup?"

The Finding: Even though a human would understand these are asking about the same thing, the AI gives completely different lists of software for each one.

When you ask the exact same question twice, the AI gives a similar list about 50–60% of the time (like getting the same weather forecast for two days in a row).
When you ask a slightly reworded question (like "best" vs. "top"), the lists only overlap about 29% of the time.
When you add a specific detail (like "for a startup"), the overlap drops to a tiny 13%.

The Metaphor: Imagine you ask a chef, "What's the best pizza?" They give you a list of 10 places. If you ask, "What's the top pizza?" the chef gives you a list of 10 different places. If you ask, "What's the best pizza for a vegetarian?" the list changes again. The chef isn't being random; they are just hyper-sensitive to the specific words you use.

2. The "Magic Mirror" Illusion

Commercial tools currently act like a magic mirror that only shows you what happens when you stand in one specific spot. They assume that if you ask "Best CRM," that question represents all the ways people might ask about CRMs.

The Paper's Reality Check: The mirror is broken. The specific string of words you type is the main driver of the answer, not the actual intent behind it.

If a brand's "visibility" score drops, it might not be because the AI suddenly dislikes them. It might just be because the tracking tool happened to ask a slightly different version of the question that week.
The "noise" (randomness caused by word choice) is so loud that it drowns out the actual "signal" (whether the brand is actually doing better or worse).

3. The "Think Harder" Myth

There is a popular idea in AI that if you tell the model to "think harder" or use more brainpower (reasoning effort), it will become more consistent and accurate.

The Paper's Finding: This doesn't work here.

Analogy: Imagine asking a student to solve a math problem. If you tell them to "show their work" and think twice, they get the right answer every time.
But here: Asking the AI to "think harder" about which CRM is best doesn't help. Because there is no single "correct" answer (many CRMs are good), the AI just spends more time justifying the first list it pulled up. It doesn't make the list more stable; it just makes the explanation longer. The list of brands still changes wildly based on the wording.

4. Why This Matters for Business

The paper concludes that the current way companies track their "AI visibility" is structurally unstable.

The Current Method: Counting mentions on a single, fixed question is like trying to measure the temperature of a room by sticking a thermometer in one specific drafty corner. If the wind blows (the wording changes), the reading jumps, but the room's actual temperature hasn't changed.
The Problem: You can't tell if a brand is growing or shrinking because the measurement tool itself is too fragile.
The Solution (according to the paper): You can't just ask more questions to fix this, because there are too many ways to ask a question (the "natural phrasing space" is too huge). Instead, companies need to stop trying to measure "mentions on a specific question" and start measuring things that happen after the AI talks, like actual clicks or sales, because those happen regardless of how the customer phrased their question.

Summary in One Sentence

Asking an AI a question is like asking a picky chef for a recommendation: changing just one word in your order will get you a totally different list of dishes, making it impossible to track your "popularity" based on a single, fixed question.

Technical Summary: Brittleness in Production Retrieval-Augmented Commercial Recommendation

Problem Statement

The paper addresses a critical fragility in the emerging field of AI Engine Optimization (AEO) and Generative Engine Optimization (GEO). Commercial monitoring tools currently track brand visibility in AI assistants by issuing a small, fixed set of prompts (e.g., "best CRM") and counting brand mentions in the model's response. These tools operate on the assumption that a specific prompt string acts as a stable label for an underlying buyer intent, such that week-over-week changes in brand mention rates reflect genuine shifts in the model's recommendation distribution or bounded sampling noise.

The authors challenge this assumption, positing that the prompt string itself, rather than the underlying intent, is the dominant input determining which brands surface. They argue that the "paraphrase brittleness"—the variance introduced by natural variations in how buyers phrase the same intent—creates a signal-to-noise ratio where the measurement artifact (prompt choice) overwhelms the actual signal (model behavior). This makes single-prompt tracking structurally unstable as a unit of measurement.

Methodology

The study employs a large-scale empirical audit across approximately 12,000 runs on OpenAI and Anthropic models to quantify two distinct metrics:

Rerun-Stability Baseline: The authors measured the Jaccard similarity between recommendation sets generated by the same prompt run twice within a single day ( $N=30$ repeats). This establishes the floor for "real" effects versus run-to-run sampling noise.
Paraphrase Brittleness: The authors measured the Jaccard similarity between recommendation sets generated by different natural-language paraphrases of the same underlying buying intent.

Experimental Design:

Corpus: ~20 base commercial prompts spanning B2B SaaS, professional services, and consumer products.
Paraphrase Axes: Five axes were tested:
- Cosmetic rewordings: Synonym swaps (e.g., "best" vs. "top") and structural rewrites (e.g., question vs. imperative).
- Constraint-adding rewordings: Modifier substitution (e.g., "AI CRM"), region/language binding, and specificity ladders (broad to narrow intent).
Models: Four production model cells (OpenAI gpt-5.4-mini and Anthropic claude-sonnet-4-6) tested at both low and high single-sample reasoning effort settings.
Evaluation: Brand mentions were extracted using a cross-judge consensus protocol (intersection of two LLM judges) to ensure conservative, high-precision extraction. Jaccard similarity was calculated on the consensus recommendation slots.

Key Results

1. The Stability Gap

The study reveals a massive discrepancy between rerun stability and paraphrase robustness:

Rerun Baseline: Same-prompt reruns yield a Jaccard similarity of 0.50–0.61.
Cosmetic Paraphrases: Natural rewordings of the same intent drop the Jaccard similarity to 0.288 (95% CI [0.215, 0.361]).
Constraint-Adding Paraphrases: Adding modifiers or narrowing intent drops the Jaccard similarity further to 0.135 (95% CI [0.098, 0.175]).

Even the upper bound of the cosmetic paraphrase confidence interval (0.361) is 14 percentage points below the lower bound of the rerun baseline (0.50). This indicates that the variation caused by a buyer simply rephrasing a question is significantly larger than the variation caused by the model's inherent sampling noise.

2. Reasoning Effort Does Not Mitigate Brittleness

Contrary to expectations from inference-time scaling literature (e.g., Guo et al., Wang et al.), increasing single-sample reasoning effort from "low" to "high" did not meaningfully reduce variance:

Rerun Stability: The effect of increased effort was bounded between -0.015 and +0.005 Jaccard.
Paraphrase Robustness: The effect was bounded by ±0.05.
The authors conclude that the "more reasoning reduces variance" prior, which holds for single-correct-answer tasks (math/competition), does not transfer to multi-plausible-answer commercial recommendation tasks. In commercial contexts, the model elaborates on whichever brands its retrieval system surfaced based on the prompt's surface form, rather than converging on a single correct answer.

3. The Role of Retrieval

The paper identifies the Retrieval-Augmented Generation (RAG) mechanism as the amplifier of this brittleness. Small changes in the query surface form shift the retrieved document pool, which in turn reshapes the candidate brands available for recommendation. The retrieved-domain Jaccard stability (0.40–0.74) correlates with recommendation-slot stability, suggesting that upstream retrieval variance drives downstream recommendation variance.

Significance and Claims

The paper claims that the current methodology for tracking "AI visibility" is fundamentally flawed.

Structural Instability: Prompt-by-prompt mention tracking is not a stable unit of measurement. The dominant source of variance in these metrics is the specific paraphrase chosen by the tracker, not the model's behavior toward the brand.
Unidentifiable Trends: Week-over-week movements in brand visibility metrics cannot be attributed to model updates or content changes because the "prompt-choice artifact" is larger than the signal.
Inefficacy of Current Solutions: Simply increasing the number of prompts in a tracking set or using higher-reasoning models does not solve the problem. The natural phrasing space is too large to be fully sampled by commercial budgets, and reasoning effort does not collapse the distribution of plausible answers.
Methodological Shift Required: The authors argue that the industry must move away from single-prompt mention counting. They suggest that meaningful improvement likely requires a different unit of measurement entirely (e.g., downstream outcome attribution) rather than a larger prompt set.

The paper concludes that the prompt string is not a benign label for an intent but the active determinant of the AI's answer, rendering the current AEO/GEO framing of "marketing to AI" as a simple discoverability metric on fixed queries conceptually erroneous.