Technical Summary: Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI

1. Problem Statement

As AI agents become increasingly autonomous, they will operate in shared environments requiring cooperation (e.g., resource allocation, collaborative projects). A central challenge in AI safety is ensuring these agents behave beneficially when interacting with other parties. While mechanism design—the theory of designing rules to align individual incentives with collective objectives—has been applied to multi-agent systems, it remains an open question whether it is sufficient to maximize social welfare in complex, open-ended environments.

The paper argues that relying solely on mechanism design is insufficient due to the fundamental limitations of contract incompleteness. In real-world scenarios, it is impossible to specify and enforce rules for every relevant future contingency. When contracts cannot distinguish between all relevant future states, a "cooperation gap" emerges where self-interested agents fail to achieve socially optimal outcomes, regardless of how well-designed the mechanism is.

2. Methodology

Theoretical Framework

The authors formalize the limitations of mechanism design using incomplete contract theory (Grossman, Hart, and Moore).

• Incontractible Cells: They define a partition of future states where a mechanism cannot efficiently distinguish between states that require different welfare-maximizing actions.
• Strict Incontractibility: A property where, within a set of states (a cell), no single action profile is first-best for all states, and the welfare-maximizing action varies across the cell.
• The Cooperation Gap: They prove that in environments exhibiting strict incontractibility within a social dilemma structure, no admissible mechanism can eliminate the welfare loss caused by self-interested play. The expected welfare loss is strictly positive ($\Delta(M) > 0$).
• Prosocial Agents: They model agents with $\lambda$-cooperative preferences, where an agent's utility function includes a weighted term for total social welfare: $V_i^\lambda = u_i + \lambda W$.
  • $\lambda = 0$: Self-interested agent.
  • $\lambda > 0$: Prosocial agent.
  • $\lambda \to \infty$: Team game (maximizing total welfare).

The theory demonstrates that while mechanisms cannot close the gap in incontractible regions, sufficiently prosocial agents ($\lambda$ large enough) can internalize the welfare term, effectively selecting the socially optimal action even when the mechanism cannot distinguish the states.

Experimental Setup

The theoretical claims are validated empirically using Large Language Models (LLMs) in two benchmark environments:

1. TableGames (GT-HarmBench): Extends the Prisoner's Dilemma and Stag Hunt to a 4x4 moral hazard setting.
   • Agents choose a base action (Cooperate/Defect) and an unobservable effort level (High/Low).
   • The unobservable effort creates an incontractible cell, as the mechanism can only observe the outcome, not the effort invested.
2. GovSimContract: A multi-agent commons governance simulation where agents fish from a shared, regenerating pond.
   • Includes deterministic (known regeneration rate) and stochastic (unknown, random regeneration rate) regimes.
   • Stochastic regeneration introduces incontractibility regarding future resource availability.

Conditions Tested:

• Contract Regimes: No Contract, Natural Language (NL) Contract (unenforced), and Code Contract (enforced via Python code).
• Agent Types: Selfish ($\lambda=0$) vs. Prosocial ($\lambda \to \infty$).
• Models: GPT-4o, GPT-5.4, GPT-5.4 Mini, Gemma-4-31B, and Grok-4.1-Fast.

3. Key Contributions

1. Formalization of Contractual Incompleteness: The paper offers a formalization of contractual incompleteness as a partition over future states, highlighting its relevance to algorithmic mechanism design where complete contractibility is often assumed.
2. Proof of the Cooperation Gap: The authors prove that under incontractible cells with a social dilemma structure, no realistic mechanism can eliminate the welfare loss from self-interested play (Theorem 6). This loss is inherent to the problem structure, not a deficiency of a specific mechanism.
3. Prosociality as a Solution: They demonstrate that $\lambda$-cooperative agents can close this gap. Prosocial agents internalize the cost of defection on others, achieving socially superior outcomes that are also individually beneficial in strict social dilemmas (Theorem 9).
4. Empirical Validation: The paper provides evidence that the theoretical cooperation gap manifests in practice across frontier LLMs. It shows that while contracts incentivize cooperation, prosociality delivers additional gains that contracts alone cannot reach.
5. Contracting Ability Analysis: The study tests LLMs' ability to coordinate on contracts, finding that while they improve over baselines, current models still fall short of the theoretically optimal contract, particularly in complex (4x4) or stochastic environments.

4. Results

• The Gap Persists: In 4x4 TableGames (where effort is incontractible), even the strongest "Code Contracts" failed to close the cooperation gap. The gap was approximately 60–68 percentage points in welfare efficiency compared to the social optimum.
• Prosociality Outperforms Contracts: In incontractible regimes (4x4 games and stochastic GovSim), prosocial agents achieved significantly higher collective gains, survival rates, and resource sustainability than selfish agents, regardless of the contract regime.
• Substitution vs. Complementarity:
  • In contractible environments (e.g., 2x2 games, deterministic GovSim), contracts and prosociality act as substitutes; strong contracts capture most available welfare.
  • In incontractible environments, they act as complements; contracts cannot close the gap, and prosociality is required to recover the lost welfare.
• Model Behavior:
  • Most frontier models (GPT-5.4, GPT-5.4 Mini, Gemma-4, Grok-4.1) exhibited near-zero implicit prosociality ($\hat{\lambda} \approx 0$) when prompted to be self-interested.
  • GPT-4o was an outlier, showing a non-zero estimated prosociality ($\hat{\lambda} \ge 4.10$), though it still struggled with complex contract drafting.
  • LLMs often drafted suboptimal contracts (e.g., overly prudential caps in deterministic GovSim) or failed to renegotiate inefficient contracts.

5. Significance and Implications

The paper concludes that designing adequate mechanisms is not sufficient to enable cooperative interactions at scale. When environments are complex enough that contracts are inevitably incomplete, agents' intrinsic values become a requirement for cooperative behavior, not an optional add-on.

• AI Safety: The findings provide formal motivation for building intrinsically prosocial AI agents. Without such intrinsic alignment, welfare gains cannot be realized in incomplete contract settings.
• Design Parameters: Prosociality is presented as a measurable design parameter ($\lambda$). System designers can compute the minimum prosociality required for a given interaction structure rather than treating cooperativeness as a binary choice.
• Verification: The work suggests a need for verification methods to certify an agent's cooperativeness to counterparties, as prosocial agents are exploitable by selfish ones without such certification.

The authors modestly note limitations, such as the vulnerability of simple prosocial checks to selfish agents and the current reliance on prompt-based implementation of prosociality, suggesting future work in mechanistic interpretability and deeper model integration.