Agentic AI and Hallucinations

1. Problem Statement

The paper addresses a critical economic challenge in the emerging market of Agentic AI: how to ensure high-quality, hallucination-free outputs in a competitive environment where AI agents act as intermediaries between upstream Large Language Model (LLM) providers and end-users.

• The Core Conflict: Upstream models have varying baseline hallucination rates ($h_0$) and costs. Downstream agents can reduce these risks through costly, non-contractible verification efforts (e.g., retrieval-augmentation, human-in-the-loop).
• The Moral Hazard: Since verification effort is unobservable to users, agents have an incentive to shirk (exert zero effort) to save costs, especially if users cannot distinguish between a "good" answer and a "hallucinated" one until after the fact.
• User Heterogeneity: Users differ significantly in their tolerance for errors. High-stakes users (e.g., in law or medicine) suffer massive disutility from hallucinations, while low-stakes users (e.g., entertainment) treat them as minor nuisances.
• Research Question: Can competitive markets and relational contracts alone induce agents to exert sufficient verification effort to satisfy high-criticality users, and how does market composition affect equilibrium prices and quality?

2. Methodology

The authors construct a tractable discrete-time, infinitely-repeated game model to analyze the interaction between competitive agents and heterogeneous users.

Key Model Components:

• Agents: A continuum of long-lived, competitive firms. They choose:
  1. An upstream model $m$ (with wholesale cost $k_m$ and baseline hallucination rate $h_0(m)$).
  2. A retail price $p$.
  3. A verification effort level $e$ (unobservable, costly).
• Users: A unit mass of users with private types $\theta \in \{H, L\}$ (High vs. Low criticality).
  • Utility: $v(\theta)$ for a correct answer; disutility $\alpha(\theta)$ for a hallucination.
  • Assumption: $v(H) > v(L)$ and crucially $\alpha(H) \gg \alpha(L)$.
  • Population: A share $\mu$ are High-type users.
• Dynamics:
  • Hallucination Mechanism: The probability of a hallucination is $h(m,e) = h_0(m) \exp(-\beta e)$, where $\beta$ is verification efficacy.
  • Termination: If a hallucination occurs, the user observes it, the relationship terminates, and the user exits the market.
  • Renewal: New uninformed users enter the market to replace those who exited, keeping the population constant.
• Incentive Mechanism:
  • Agents offer long-term relational contracts.
  • Enforcement: The threat of losing future rents (the "shadow of the future") disciplines the agent. If an agent shirks and a hallucination occurs, they lose the stream of future profits from that user.
  • Cost of Effort: Convex cost function $c(e)$ with $c(0)=0, c'>0, c''>0$.

Equilibrium Concept:

The authors analyze Reputational Equilibria where agents exert positive effort because the present value of future relational rents ($V_C$) exceeds the immediate cost of shirking. The price is determined by the Incentive Compatibility (IC) constraint binding, ensuring the agent is indifferent between exerting effort and shirking.

3. Key Contributions

1. Relational Contracts in Agentic AI: The paper formalizes how relational incentives (reputation) solve the moral hazard problem in AI verification, acting as a substitute for explicit, unverifiable contracts.
2. Endogenous Market Discipline: It demonstrates that market composition ($\mu$) acts as a disciplinary device. The presence of high-criticality users forces the entire market toward higher verification standards, even for low-criticality users.
3. Two-Layer Quality Choice: The model explicitly analyzes the trade-off between upstream selection (choosing a better, more expensive LLM) and downstream verification (expending effort).
4. Welfare Implications: It shows that without future interactions ($\delta=0$), the market collapses to zero effort (the "Spot Market" outcome), highlighting the necessity of long-term relationships for AI reliability.

4. Main Results

A. The Spot Market Benchmark ($\delta = 0$)

• If agents are myopic (no future value), the IC constraint forces effort $e^* = 0$.
• Agents simply pass through the wholesale cost of the model ($p = k_m$).
• Quality is determined solely by the baseline model selection, which maximizes the weighted average utility of all users. This leads to suboptimal outcomes for high-stakes users.

B. The Reputational Equilibrium ($\delta > 0$)

• Existence: A unique reputational equilibrium exists if the discount factor $\delta$ is sufficiently high (agents are patient enough).
• Effort Level: The equilibrium verification effort $e^*$ is strictly increasing in the share of high-type users ($\mu$).
  • Intuition: As the proportion of high-stakes users grows, the potential loss of future rents from a hallucination increases (since these users are more sensitive to errors). This tightens the IC constraint, requiring higher effort to maintain the relationship.
• Pricing: Since effort is costly, the equilibrium price $p$ increases with $\mu$. High-criticality sectors (law, medicine) endogenously lead to more expensive AI services due to the required verification intensity.
• Binding Constraints:
  • If the hallucination rate is high, the participation constraint for High-types ($V_H \ge 0$) binds.
  • If the hallucination rate is low, the constraint for Low-types ($V_L \ge 0$) binds.
  • The transition point depends on the sensitivity ratio $\kappa = \frac{\alpha(H)-\alpha(L)}{v(H)-v(L)}$.

C. Numerical Findings

• Patience ($\delta$): Higher patience relaxes the IC constraint, allowing for higher effort levels across all market compositions.
• Technology ($\beta$): More effective verification technology (higher $\beta$) shifts the effort curve upward, allowing agents to achieve quality targets with less effort or at lower market shares of high-types.
• Model Switching: There is a threshold for $\mu$ (approx. 0.26 in the example).
  • For $\mu < 0.26$: Agents choose the cheaper, riskier upstream model (Model A) and compensate with high verification effort.
  • For $\mu > 0.26$: Agents switch to the expensive, cleaner upstream model (Model B) because the cost of verifying the riskier model becomes too high relative to the wholesale savings.

5. Significance and Implications

• Sector-Specific Regulation: The findings suggest that "one-size-fits-all" AI regulations may be inefficient. Sectors with high $\mu$ (law, medicine) naturally evolve toward higher verification standards and prices due to market forces, whereas entertainment sectors may not.
• Role of Agents: AI agents are not just passive resellers; they are active quality controllers. Their profitability depends on maintaining a reputation that disciplines their verification behavior.
• Market Failure Risk: If the discount factor is too low (e.g., in a market with high churn or short-term contracts), the market will fail to provide safe AI for high-stakes applications, regardless of the availability of better models.
• Policy Insight: To ensure safety in critical domains, policies might need to enforce minimum contract durations or transparency mechanisms that enhance the "shadow of the future," thereby strengthening the reputational incentive to verify.

In summary, the paper provides a rigorous economic framework showing that reputation is the primary mechanism ensuring AI safety in competitive markets, and that the mix of user types fundamentally dictates the equilibrium level of verification effort and pricing.