📈 economics

Agent-Facing Information Design in LLM Tool Registries

This paper reveals that legal puffery, rather than fabricated claims, is the primary driver of bias in LLM tool selection, demonstrating that current disclosure methods fail and proposing a registry-layer design that separates structured capability descriptions from marketing copy to ensure agent accountability.

Original authors: Haochuan Kevin Wang

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haochuan Kevin Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Wild West" of AI Tool Stores

Imagine a massive digital marketplace where AI assistants (the "agents") go to pick tools to do jobs for you. Maybe you ask your AI to "find the best weather forecast," and it has to choose between five different weather apps.

Right now, this marketplace is like a Wild West flea market with no rules.

The Problem: The tool owners write their own descriptions. They can use fancy words, superlatives ("The #1 Best Tool!"), and big claims to trick the AI into picking them.
The Missing Piece: Unlike human shopping, where we have ratings, verified reviews, or "Sponsored" labels that actually work, this AI marketplace has no measurement system. There is no way to check if a tool is actually good or if the description is just a lie.

The researchers ran over 17,700 experiments to see how easily these AI agents can be swayed by marketing fluff.

Key Findings: What They Discovered

1. The "Hype" Wins Every Time

The researchers found that if two tools do the exact same job, but one has a boring description and the other has a "hyped" description (using words like "fastest," "most accurate," or "trusted by millions"), the AI picks the hyped one 100% of the time.

The Analogy: Imagine two identical cars parked side-by-side. One has a plain sign saying "Car." The other has a giant neon sign saying "The World's Fastest, Most Reliable Car!" The AI doesn't care that the cars are identical; it blindly drives toward the neon sign.
The Result: Just changing the words (copywriting) captures all the traffic. The AI is essentially "clickbait-prone."

2. Lying Doesn't Help (But Hype Does)

The team tested if making up fake numbers (like "Used by 12 million developers" when it's not true) made the AI pick the tool more often than just using legal exaggeration (puffery).

The Finding: No. The AI was just as likely to pick the tool with legal exaggeration as the one with fake lies.
The Takeaway: The "lie" didn't add any extra power. The problem isn't that the AI is being tricked by lies; it's that the AI is reacting to the style of the language (superlatives) rather than the facts. This means regulators can't just ban "fake ads" to fix the problem, because the "legal" hype is already doing 100% of the damage.

3. Warning Labels Don't Work

The researchers tried to "fix" the problem by adding warnings, similar to how we see "Sponsored" labels on Google or star ratings on Amazon.

The Result: It failed completely.
- For most AIs: The warning was ignored. The AI still picked the hyped tool. It's like putting a "Caution: This ad is paid for" sticker on a neon sign, and the AI still runs toward the light.
- For one AI (Claude): It overreacted. When it saw a "Sponsored" label, it actually avoided the tool, even if it was the best one.
The Conclusion: You can't fix this by just telling the AI "be careful." The system is broken, not the AI's brain.

4. The "Prisoner's Dilemma" (The Race to the Bottom)

Because the hyped tool wins every time, every tool owner is forced to write hyped descriptions just to survive.

The Analogy: Imagine a job interview where everyone is qualified. If one candidate wears a suit and says "I'm the best," they get the job. So, everyone else starts wearing a suit and saying "I'm the best." Eventually, everyone is wearing a suit and shouting, so the interviewer can't tell who is actually good.
The Result: The marketplace becomes useless. The descriptions stop telling you anything about quality because everyone is just shouting the same marketing slogans.

The Proposed Solution: The "Menu" vs. The "Flyer"

The paper suggests a complete redesign of how these tool stores work. They propose splitting the information into two separate buckets:

The "Selection Menu" (For the AI):
- This is what the AI sees before it picks a tool.
- Rule: No marketing words allowed. No "best," no "fastest," no "trusted by."
- Format: Just cold, hard facts in a structured list (e.g., "Input: Text. Output: 10 links. Speed: 200ms").
- Why: The researchers found that AIs actually prefer this boring, structured format over flowery prose. It helps them pick the right tool without being distracted by hype.
The "Marketing Flyer" (For You, the Human):
- This is what the human sees after the AI has already picked the tool.
- Rule: The tool owner can write all the fancy marketing copy they want here.
- Why: The AI has already done its job based on the facts. Now, you (the human) can read the "Sponsored" pitch to decide if you like the tool.

Summary in One Sentence

AI agents are currently being fooled by marketing hype because tool registries are unregulated advertising platforms; the solution isn't to teach the AI to ignore lies, but to force the registries to show the AI a boring, fact-only menu while saving the fancy sales pitch for the human user.

Technical Summary: Agent-Facing Information Design in LLM Tool Registries

1. Problem Definition

The paper identifies a critical gap in the emerging ecosystem of Large Language Model (LLM) tool registries (e.g., PulseMCP, Composio). These registries function as de facto advertising platforms where tool providers compete for agent visibility through free-text descriptions. Unlike the digital advertising industry, which spent two decades building measurement infrastructure (viewability standards, quality scores), the agent tool ecosystem lacks any accountability mechanism.

The core problem is information asymmetry. In the absence of verified quality signals, agents rely on description richness as a proxy for capability. This creates a market for "generative engine optimization" (GEO), where providers strategically frame descriptions to manipulate agent selection. The paper investigates whether standard regulatory mechanisms (disclosure labels, warnings) effective in human-facing advertising transfer to agentic settings, and proposes a structural redesign for registries to restore market efficiency.

2. Methodology

The study employs a systematic experimental framework combining 17,700+ trials across five LLMs (DeepSeek-V3, o4-mini, GPT-5.4-mini, GPT-5.4-nano, Claude Sonnet 4.6, and GPT-4o) and ten synthetic task domains (e.g., web search, e-commerce, code execution).

Experimental Design

Task: A two-alternative forced-choice task where an agent must invoke exactly one of two functionally identical tools differing only in description framing.
Metric: Selection Bias Coefficient (SBC), defined as $P(\text{select optimized}) - 0.5$ . An SBC of 0 indicates chance selection; +0.5 indicates perfect selection of the optimized tool.
Framing Levels (L0–L4): Descriptions range from neutral functional specs (L0) to commercially optimized versions incorporating social proof, superlatives, authority framing, and outcome framing (L4).
Disclosure Interventions: Four conditions tested to mitigate bias:
1. Baseline (no intervention).
2. [SPONSORED] labels appended to descriptions.
3. Star ratings (e.g., "3/5 stars - commercial partner").
4. System-prompt warnings (general transparency mandates).
Ecological Validity: Experiment C utilized verbatim descriptions from live MCP registries (e.g., Brave Search vs. DuckDuckGo) to validate synthetic findings.
Legal Boundary Analysis: Experiment E4 isolated "legal puffery" (FTC-permissible superlatives) from "fabricated claims" to determine the source of selection bias.

3. Key Contributions

A. Registry Design Prescription (Primary Contribution)

The paper proposes a structural separation of descriptions into two distinct layers:

Selection-Facing Description: Structured, registry-controlled, and stripped of all evaluative language. This layer is used exclusively for agent selection.
Marketing-Facing Description: Provider-authored, containing all persuasive copy, displayed only post-selection to the human user.
This architecture, grounded in Proposition 3, achieves "first-best welfare" (selecting tools based on function rather than copy) independent of the specific LLM model.

B. The Agent Attention Quality Score (AAQS)

The authors introduce AAQS to distinguish genuine capability from description-driven selection:
$AAQS_i(q) = P(\text{select}_i | q) \times C_i(q)$
Where $C_i(q)$ is a capability match score. A low AAQS relative to selection probability indicates the tool is "coasting on description optimization" (the agent equivalent of clickbait).

C. Strategic Equilibrium Analysis

Using a game-theoretic model (Propositions 1–3), the paper formalizes the incentive structure as a Prisoner's Dilemma. Without intervention, the Nash Equilibrium is for all providers to optimize descriptions, rendering them uninformative and wasting resources on a "description arms race." The model proves that normalization is necessary (not merely sufficient) because disclosure mechanisms fail to break this equilibrium.

4. Empirical Results

4.1. Optimization Effectiveness

Total Traffic Capture: In consumer domains (web search, e-commerce), four of five models selected the optimized tool with $P=1.0$ (SBC = +0.456) when no disclosure was present.
Dominant Feature: Superlatives (e.g., "fastest," "most accurate") are the single most powerful feature (SBC = +0.35), outperforming social proof (+0.12) and authority framing (+0.21).
Dose-Response: The selection advantage is front-loaded. The jump from neutral (L0) to a single trust signal (L1) captures the majority of the achievable lift (+0.33). Additional rhetorical layers yield diminishing or negative returns.

4.2. Failure of Disclosure Mechanisms

Disclosure interventions failed structurally for distinct reasons:

Ceiling Inertia: For models already at $P \approx 1.0$ , labels like [SPONSORED] had negligible effect (e.g., DeepSeek showed only a -3 percentage point drop).
Architectural Blindness: System-prompt warnings produced zero measurable effect for four of five models; the warning did not propagate to the function-call selection logic.
Overcorrection: Claude Sonnet treated [SPONSORED] labels as disqualifiers, penalizing labeled tools below chance levels (SBC $\to$ -0.155).

4.3. Legal Puffery vs. Fabrication

Legal Puffery Captures 100% of the Effect: FTC-permissible puffery (subjective superlatives, benefit framing) alone produced the full optimization effect (SBC = +0.33).
Fabricated Claims Add Zero: Adding fabricated statistics or endorsements provided zero incremental bias. Consequently, FTC enforcement of deceptive advertising rules against fabricated claims would have near-zero impact on agent selection, as the active mechanism is constitutionally protected puffery.

4.4. Ecological Evidence

Preliminary tests with live registry descriptions (e.g., Brave Search vs. DuckDuckGo) confirmed the synthetic patterns, with SBC values ranging from +0.30 to +0.46, suggesting the phenomenon is not an artifact of synthetic data.

5. Significance and Claims

The paper argues that the problem is not agent "irrationality" but a flawed information environment. In a registry without verified quality signals, description richness is the only available proxy for capability, making agents rationally responsive to commercial framing.

Regulatory Implication: Current regulatory approaches (disclosure) are structurally ineffective against agentic selection. The solution requires mechanism design (registry normalization) rather than better agent alignment.
Economic Impact: The optimized provider captures ~83% of traffic (vs. 50% fair share), creating a massive, unmonetized arbitrage opportunity for providers and a loss of welfare for users.
Design Shift: The paper concludes that registries must evolve from "advertising platforms" to "structured data repositories" for selection, reserving marketing copy for post-selection human consumption. This shift eliminates bias, reduces social waste (the arms race), and aligns with agent preferences for structured data (agents preferred structured formats over neutral prose in sanity checks).

The authors release their benchmark and analysis code to support future work in agent-facing information design, emphasizing that without structural intervention, the tool selection market will remain unaccountable and inefficient.