Technical Summary: Prominence-Stratified Failure Modes in Retrieval-Augmented Commercial Recommendation

Problem Statement

AI assistants (e.g., ChatGPT, Claude) function as recommendation engines rather than search engines, nominating specific brands in response to commercial queries rather than returning link lists. The prevailing industry assumption, framed as Generative Engine Optimization (GEO) or Answer Engine Optimization (AEO), treats brand visibility as a singular "discoverability" problem analogous to Search Engine Optimization (SEO).

This paper challenges that monolithic view. It posits that the failure modes of brand recommendation are not uniform but are strictly stratified by a brand's prominence (its awareness footprint within a sector). The core research question is whether the optimal marketing intervention depends on where a brand sits on a prominence ladder, and whether the "one-size-fits-all" approach to GEO misallocates effort by ignoring these distinct failure mechanisms.

Methodology

Data Collection and Scope

The authors conducted a large-scale audit comprising approximately 37,000 production runs across four distinct model configurations (cells) and 215 commercially-framed prompts spanning 19 sectors (B2B SaaS, professional services, consumer products, etc.).

• Model Ladder: The audit utilized four production cells: OpenAI gpt-5.4-mini (low/high effort) and Anthropic claude-sonnet-4-6 (low/high effort).
• Reference Catalog: A curated list of 533 brands was constructed, stratified into five prominence tiers (L1–L5) based on external authority lists (e.g., Gartner Magic Quadrant, G2, Crunchbase, Y Combinator batches, regional registries).
  • L1: Category leaders (e.g., Salesforce, HubSpot).
  • L2: Established challengers (e.g., Pipedrive, Gusto).
  • L3: Mid-market brands.
  • L4: Long-tail specialists.
  • L5: Regional players.
• Retrieval Conditions: The study compared native web-search tools against external retrieval substrates (Exa for neural retrieval, Brave for keyword search) to isolate retrieval-source bias.

Measurement Framework

The audit classifies brand performance using a four-stage funnel adapted from consumer decision journeys:

1. Stage 1 (S1): No retrieval, no mention (Brand is absent from queries, retrieved domains, snippets, and completion text).
2. Stage 2 (S2): Retrieval, no mention (Brand appears in retrieval layers but is excluded from the generated answer).
3. Stage 3 (S3): Mention, not recommended (Brand is mentioned in the text but not endorsed as a final recommendation).
4. Stage 4 (S4): Recommended (Brand is consensus-recommended by dual LLM judges).

Extraction Protocol: Brand mentions were extracted using a conservative consensus approach requiring two parallel LLM judges (claude-haiku-4-5 and gpt-5-mini) to agree on a mention. Statistical significance was assessed using Wilson 95% confidence intervals for descriptive metrics and prompt-clustered bootstrap CIs for inferential comparisons to account for within-prompt clustering.

Key Contributions

1. Prominence-Stratified Discoverability Map: The paper maps aggregate and per-query surface rates across the five prominence levels, revealing that long-tail reachability is not a smooth gradient but a bimodal distribution where L1–L2 are universally reachable, while L4–L5 face catastrophic invisibility.
2. Five-Mode Taxonomy of Recommendation Failure: The authors identify that each prominence tier corresponds to a distinct dominant failure mode:
   • L1: Failure at S2/S3 (Compellingness/Positioning).
   • L2: Mixed signals, with persona-mediated substitution observed on Anthropic models.
   • L3: Hybrid failure across all stages (S1–S3) simultaneously.
   • L4: Failure at S1 (Discoverability).
   • L5: Failure at S1 plus geographic gating.
3. Per-Prominence Prescription Framework: The study derives tier-specific optimization strategies, arguing that interventions effective for L1 (differentiation) are ineffective for L4 (which requires retrieval-side seeding), and vice versa.

Key Results

Surface Rates and Visibility

• L1 Brands: Achieve a 77% per-query surface rate and 100% aggregate surface rate. They are almost always retrieved.
• L2 Brands: Achieve a 60% per-query surface rate and 100% aggregate surface rate.
• L3 Brands: Surface rates drop significantly to 23% per-query and 88% aggregate.
• L4 & L5 Brands: Face "catastrophic invisibility." Only 8% of L4 and 3% of L5 brands appear in relevant per-query searches. In aggregate, 48–52% of L4–L5 brands never surface in any of the 37,000 runs.

Conversion Rates (Stage 4)

Even when brands are retrieved, conversion to a recommendation is not guaranteed:

• L1: Converts at 25–41%. High visibility does not guarantee recommendation; differentiation is the bottleneck.
• L2: Exhibits the highest conversion rates (37–52%) across all tiers, though they are susceptible to persona-mediated substitution on Anthropic models.
• L3: Conversion drops to 34–40%, with the highest sensitivity to persona shifts (0.67–0.75 swap rate).
• L4–L5: Conversion rates decline sharply in the long tail, with L5 on sonnet-4.6/high dropping to 12.8%.

Retrieval and Persona Dynamics

• Retrieval Source Bias: Cross-provider agreement (OpenAI vs. Anthropic) is high for L1–L2 (0.63 Jaccard) but drops sharply at L3 (0.40). Conversely, within-provider agreement between native search and external neural retrieval (Exa) declines monotonically with prominence (0.83 for L1 down to 0.50 for L5), indicating that long-tail brands rely heavily on non-native retrieval systems.
• Persona Effects: L3 brands are most volatile regarding user personas. L1 brands are relatively persona-resistant.
• Model Class & Generation:
  • OpenAI: Moving from mini to non-mini (e.g., gpt-5.4-mini to gpt-5.4) increases brand density, particularly for L2–L5.
  • Anthropic: No statistically significant density increase was observed when moving from sonnet to opus.
  • Generations: Upgrades (e.g., 5.4 to 5.5) yield small, monotonic lifts in surface rates that decay to zero at the long tail (L5).
• Multi-Turn Interaction: Buyer-led iterative dialogue (4-turn conversations) does not rescue S1 invisibility for L4–L5 brands. Follow-up turns tend to drill down into brands already named in Turn 1 rather than expanding the retrieval pool.

Significance and Claims

The paper claims that the dominant failure mode of AI commercial recommendation is not a single discoverability problem but a prominence-dependent phenomenon.

• Rejection of Uniform Optimization: The authors argue that "marketing to AI" cannot be reduced to a single set of content interventions (e.g., adding citations or statistics). The "right" investment depends entirely on the brand's tier.
  • L1 Strategy: Focus on differentiation and positioning (S2/S3) because visibility is already saturated.
  • L4/L5 Strategy: Focus on retrieval-side seeding (S1) via authority lists and canonical content on platforms accessible to neural retrieval (e.g., Medium, GitHub), as content differentiation is moot if the brand is never retrieved.
• RAG Limitations: The study provides direct evidence that Retrieval-Augmented Generation (RAG) does not solve the long-tail discoverability gap. In fact, reliance on a small set of authority sources may narrow the gap further, leaving 48–52% of long-tail brands completely invisible.
• Provider Specificity: The "density mechanism" (where larger models surface more brands) is specific to OpenAI's architecture at the tested generations and does not replicate on Anthropic's sonnet vs. opus comparison.

The paper concludes that practitioner advice in the AEO/GEO space that ignores prominence stratification systematically misallocates effort, applying L1-focused differentiation tactics to brands that need L4-focused retrieval visibility, and vice versa.