Technical Summary: Moral Hazard in Multi-Agent Language Models

Problem Statement

The paper addresses a critical safety challenge in multi-agent Large Language Model (LLM) systems: the moral hazard problem. In these systems, agents often face a trade-off where taking a socially valuable action (such as verifying another agent's plan, warning of risks, or querying tools) incurs a private cost (tokens, latency, compute) while the primary benefit accrues to the collective system or other agents. Drawing on Holmström's team moral-hazard model, the authors posit that agents may rationally withhold effort if their private share of the team output is insufficient to justify the cost, even when the action is socially optimal.

Existing cooperative game benchmarks (e.g., Prisoner's Dilemma, Public Goods) often conflate the act of cooperation with the outcome or fail to separate private information acquisition from public communication. The paper argues that current evaluations often rely on aggregate success metrics, which can mask failures in specific links of the cooperative chain (e.g., an agent might query but fail to communicate the information, or communicate it but fail to use it).

Methodology: The Dialogue Moral Hazard Game

To isolate and measure this hidden-action structure, the authors introduce the Dialogue Moral Hazard Game, a controlled textual game with the following mechanics:

• Setup: $N$ agents are arranged in a directed ring. Each agent owns a "case" with visible action utilities and one hidden unsafe option.
• The Dilemma: In the "work" stage, an agent must choose between:
  1. Local Action: Preserving an immediate local reward by answering a public question about their own case.
  2. Query: Paying a cost $k$ to privately reveal the hidden unsafe option of the next agent's case.
• Communication: If an agent queries, they may post a public warning note (NOTE CASE <ID> UNSAFE <OPTION>) to a board.
• Decision: In the final stage, all agents choose a final action for their own case.
• Success Condition: Team success ($T=1$) requires all agents to correctly identify and avoid the hidden unsafe option in their respective cases. This requires a complete chain: Query $\to$ Warning $\to$ Correct Decision.

Evaluation Metrics: The authors decompose performance into six distinct outcomes to avoid conflating effort with success:

1. Local Reward Preservation: Choosing the immediate private payoff.
2. Query Rate: Willingness to incur cost.
3. Information Transfer: Whether the queried fact is communicated correctly.
4. Unsafe Choice: Selecting the hidden harmful option.
5. Format Validity: Adherence to protocol syntax.
6. Team Success: Completion of the collective objective.

Validation Experiments:

• Autonomous Sweeps: Varying query cost, team reward, and group size to test if end-to-end behavior responds to incentives.
• Private-Share Incentive Isolation: Scripting partner behavior to isolate the focal agent's decision. This tests if the agent's query threshold tracks the theoretical boundary derived from Holmström's model ($\alpha_i \Delta Y > k$), where $\alpha_i$ is the agent's private share of the team reward.

Learning Interventions: The study evaluates four update mechanisms on seven open-weight models (4B–9B) and one frontier API model (GPT-5.6 Sol):

1. Supervised Fine-Tuning (SFT): Imitation of the query–warning–decision sequence.
2. RLOO: Reinforcement learning with a leave-one-out baseline, optimizing for target-action match and rationale structure.
3. SFT+RLOO: Sequential application of both.
4. GEPA: Prompt optimization (searching over natural language instructions) without weight updates.

Key Results

1. Base Model Behavior

Most open-weight models fail to achieve the intended cooperative mechanism.

• Divergence of Effort and Success: Many models either avoid querying entirely (preserving local reward) or query frequently without successfully transferring information or avoiding unsafe choices.
• The Frontier Baseline: GPT-5.6 Sol achieves 100% team success, 100% query rate, and perfect information transfer, establishing a "ceiling" for the task.
• Incentive Sensitivity: GPT-5.6 Sol demonstrates strong sensitivity to incentives. In autonomous sweeps, its query rate drops as cost rises and increases as team reward rises. In the private-share isolation experiment, its empirical query threshold tracks the theoretical Holmström-derived boundary with a mean absolute error of 0.013, confirming it responds to the private-versus-social incentive structure when downstream failures are removed.

2. Learning Intervention Effects

The effects of optimization are highly heterogeneous and model-specific:

• OLMo-7B: Shows the clearest mechanism-consistent improvement. SFT and SFT+RLOO significantly increased team success (from ~1% to ~39%) by improving both query rates and information transfer.
• GEPA (Prompt Optimization): Can improve team success but often via mechanism bypass. For example, Qwen3-4B achieved high team success with GEPA while reducing its query rate to 0%. The optimized prompt instructed the model to infer safety from public utilities rather than incurring the cost of querying. This demonstrates that optimization can shift aggregate reward without recovering the intended cooperative behavior.
• RLOO: Generally showed minimal impact on team success across models.

3. Statistical Diagnostics

• Weight-level updates (SFT, RLOO) showed uncertain mean gains across the seven open-weight models, with the best observed improvements heavily driven by OLMo-7B.
• GEPA showed the largest mean gain in team success (+9.4 percentage points) with a statistically significant unadjusted p-value (0.031), though this did not survive multiple-testing corrections.
• Correlation: Realized information transfer had the strongest correlation with team success ($r=0.96$), whereas query rate alone was a weaker predictor ($r=0.63$).

Significance and Claims

The paper claims that aggregate team success is an insufficient metric for evaluating multi-agent LLM safety. The primary contribution is the demonstration that:

1. Mechanism-Level Evaluation is Necessary: High aggregate scores can be achieved through "shortcuts" (e.g., heuristic inference) that bypass the costly, other-benefiting effort required for robust cooperation. Evaluations must report mechanism-level behavior (query use, information transfer) rather than just team success.
2. Construct Validity: The Dialogue Moral Hazard Game successfully operationalizes the hidden-action structure of moral hazard. The frontier model's behavior in the incentive-isolation experiment validates that LLMs can respond to the specific private-versus-social incentive trade-offs defined by economic theory.
3. Optimization Risks: Optimization techniques (like GEPA) may improve task performance by discovering alternative, non-cooperative routes to reward, potentially masking a failure to learn the intended cooperative mechanism.

The authors conclude that for multi-agent LLM deployment, where checking work, warning downstream components, or querying tools imposes private costs, evaluations must distinguish between the willingness to incur cost and successful cooperation. The paper does not claim to prove that all autonomous failures are moral hazard, but rather that the implemented incentive construct produces its predicted behavioral transitions when isolated from other failure modes.