Technical Summary: Divergent Recommendations, Convergent Diagnoses

Problem Statement

As brand audiences fragment across multiple AI providers (e.g., ChatGPT, Claude, Gemini), marketing teams face a strategic uncertainty: should they deploy a single optimization playbook across all providers, or must they tailor distinct strategies for each? The industry premise, driven by per-provider dashboards, assumes that recommendation sets differ significantly across providers, implying that interventions (content, authority, positioning) are provider-specific. However, the theoretical "algorithmic monoculture" hypothesis suggests that providers sharing training data and architectural lineage should exhibit correlated exclusions (failures to retrieve) even if their specific inclusions (recommendations) diverge. This paper empirically tests whether the recommendation sets diverge while the diagnosis of joint failures converges, and whether this convergence holds across brand prominence levels.

Methodology

The study employs a cross-provider audit design comparing OpenAI (gpt-5.4, gpt-5.4-mini) and Anthropic (claude-sonnet-4-6, claude-opus-4-6) models.

• Data & Prompts: The analysis covers 215 commercially framed prompts across four independent measurement batches, evaluated against a shadow corpus of 533 reference brands stratified by five prominence levels (L1: Category Leaders to L5: Regional Players).
• Consensus Extraction: Brand mentions in retrieval layers (search queries, snippets, domains) and completion text were extracted using a cross-judge consensus protocol (two LLM judges) to ensure robustness against single-judge instability.
• Funnel-Stage Classification: For each (prompt × brand × provider) cell, brands were classified into four stages:
  • S1 (Discoverability): Brand absent from all retrieval layers and completion.
  • S2 (Compellingness): Brand present in retrieval but absent from completion.
  • S3 (Positioning): Brand present in completion but not recommended.
  • S4 (Success): Brand consensus-recommended.
• Joint-Failure Restriction: The core analysis focuses on 7,763 "joint-failure" cells where neither provider recommended the brand (i.e., both failed at S1, S2, or S3).
• Mechanistic Signals:
  • Pure-Priors Share: The proportion of recommendations made without any observable retrieval-layer evidence (indicating reliance on parametric memory).
  • Batch-Reconstruction (OpenAI only): Tracing whether brand-disambiguating queries appeared in the first search batch to measure retrieval-anchoring.
• Statistical Analysis: Agreement metrics (Jaccard similarity, raw agreement, Cohen's $\kappa$) were calculated with prompt-clustered bootstrap 95% confidence intervals (1,000 iterations) to account for the non-independence of (prompt × brand) observations.

Key Contributions

1. Quantified Cross-Provider Recommendation Divergence: The study establishes that cross-provider recommendation sets are significantly less similar than within-provider rerun stability. The cross-provider Jaccard similarity is 0.35 (95% CI: [0.323, 0.391]), falling well below the within-cell rerun baseline of 0.50–0.61.
2. Mechanistic Asymmetry in Recommendation Routes: The providers reach their picks via measurably different generative routes.
   • Anthropic: Flagship models recommend brands based on parametric priors (without retrieval evidence) in 43–52% of cases.
   • OpenAI: Models rely heavily on retrieval, with pure-prior recommendations ranging from 8–29%. Batch reconstruction confirms that OpenAI issues brand-disambiguating queries in the first search batch 60–71% of the time.
3. Failure-Mode Convergence with a Prominence Gradient: While picks diverge, the diagnosis of why a brand was not recommended converges overwhelmingly, but this convergence is strictly stratified by brand prominence.
   • Overall Agreement: On joint failures, both providers diagnose the same failure stage 95.1% of the time (95% CI: [94.3%, 95.7%]).
   • Monotone Gradient: Agreement rises as brand prominence falls:
     • L1 (Category Leaders): 81.2% agreement.
     • L5 (Regional Players): 99.6% agreement.
   • Driver: The convergence is driven by S1 (Discoverability) failures. For long-tail brands (L4–L5), both providers fail to discover the brand at the same rate. For leaders (L1), disagreement persists in post-retrieval stages (S2/S3).

Results

• Divergent Picks, Convergent Diagnoses: The providers select different brands (low Jaccard) but agree on why they fail to recommend a brand when they both fail.
• The Role of Prominence: The 95.1% pooled agreement figure is heavily influenced by the dominance of S1 failures in the long tail. When restricting analysis to cases where at least one provider fails at S2 or S3 (post-retrieval), diagonal agreement drops to 14.9%. This indicates that while providers agree on "missing" brands (S1), they do not agree on why a retrieved brand is not recommended (S2 vs. S3) for high-prominence entities.
• Cross-Cell Portability: A 3,000-run extension testing within-provider variations (e.g., gpt-5.4-mini vs. gpt-5.4, sonnet vs. opus) found that same-stage agreement remains high (94.9%–98.3%). This suggests the diagnostic convergence is a property of the joint-failure population rather than a specific artifact of cross-provider comparison.
• Mechanistic Evidence: The "pure-priors" signal confirms that Anthropic relies more on internal knowledge for recommendations, while OpenAI is more retrieval-anchored. Despite these different routes, the exclusion of long-tail brands is correlated, supporting the algorithmic monoculture hypothesis for failures but not for successes.

Significance and Claims

The paper claims to provide the first concrete commercial-recommendation evidence for the "algorithmic monoculture" hypothesis, specifically demonstrating correlated exclusions (convergent failures) alongside uncorrelated inclusions (divergent picks).

The primary practical implication is a scoping of marketing-to-AI interventions:

• For Long-Tail Brands (L4–L5): A single, provider-agnostic intervention targeting S1 (discoverability)—such as authority-list seeding or canonical content creation—is expected to lift visibility across both providers simultaneously. The diagnosis of "why" the brand is missing is effectively the same for both.
• For Category Leaders (L1–L2): Provider-specific strategies remain necessary. Since providers diverge on post-retrieval failure modes (S2/S3) for high-prominence brands, interventions regarding positioning and content optimization must be tailored to the specific provider's retrieval and generation logic.

The authors conclude that the "diagnose once, advise once" strategy is a strong claim for long-tail discoverability but a weaker, prominence-dependent claim for post-retrieval failure attribution. The study refines, rather than rejects, the monoculture hypothesis by showing that shared substrate leads to shared failures, while divergent post-training leads to divergent success routes.