Technical Summary: Information and Contract Design for Repeated Interactions between Agents with Misaligned Incentives

Problem Statement

This paper investigates the dynamics of repeated interactions between two agents with misaligned incentives and information asymmetry: a Sender (S) who possesses private, vital information about the environment but cannot act, and a Receiver (R) who must make sequential decisions but lacks full environmental observability.

The core challenge lies in the strategic dimension of information sharing. While the Sender can influence the Receiver's actions through signals, their reward functions are not necessarily aligned. The Sender aims to maximize its own cumulative reward, which depends on the Receiver's actions, potentially at the expense of the Receiver's utility. The paper explores how the Sender learns optimal communication strategies under these conditions and how the introduction of linear contracts (where the Sender charges a fraction of the Receiver's reward for information) alters these dynamics.

The study addresses four specific research questions (RQs):

1. Can the Sender learn optimal signalling policies under varying degrees of incentive misalignment, and can the Receiver learn to leverage these signals?
2. How does the quality of the Receiver's private observations affect learned strategies and outcomes?
3. Can the Sender learn to use contracts effectively to price information?
4. How does the use of contracts impact the Receiver's accumulated reward?

Methodology

Theoretical Model

The authors formalize the interaction as a Markov Game involving two interrelated decision problems:

• The Receiver: Modeled as a Partially Observable Markov Decision Process (POMDP) where the state space is projected through an observation function $O$. The Receiver's effective observation space includes both local environmental observations and signals ($\Sigma$) from the Sender. The Receiver seeks a policy $\pi_R$ to maximize expected discounted rewards.
• The Sender: The Sender has full observability of the environment but no direct agency. Its "actions" are signals. The Sender commits to a signalling policy defined by a probability $p \in [0, 1]$.
  • With probability $p$, the Sender recommends the action optimal for the Receiver ($\pi'_R$).
  • With probability $1-p$, the Sender recommends the action optimal for itself ($\pi'_S$).
  • The Sender chooses $p$ at the start of an episode to maximize its own expected return, subject to the Receiver's obedience constraint (the expected utility of following the signal must exceed the Receiver's independent utility).

Linear Contracts: The model is extended to include a "take it or leave it" linear contract. The Sender proposes a tuple $\langle p, c \rangle$, where $c \in [0, 1]$ is the fraction of the Receiver's reward transferred to the Sender. If accepted, the reward structures are modified: $Re'_S = Re_S + cRe_R$ and $Re'_R = (1-c)Re_R$.

Experimental Setup

The authors evaluate the model in two distinct environments using Multi-Agent Reinforcement Learning (MARL) and Multi-Armed Bandits (MAB):

1. Recommendation Letter Scenario: A classic Bayesian Persuasion setup adapted for learning.

   • Context: A professor (Sender) knows a student's quality (Strong/Weak); a recruiter (Receiver) decides to hire.
   • Learning Mechanism: The Sender uses a multi-armed bandit to select signalling policies ($\langle p_1, p_2 \rangle$). The Receiver uses a contextual bandit to decide whether to hire based on the signal.
   • Extension: Contracts are introduced as a third dimension to the Sender's action space.

2. Gridworld Environment: A 10x10 grid where the Receiver (a robot) collects apples and diamonds.

   • Asymmetry: The Sender sees the full grid; the Receiver has limited visibility controlled by a parameter $v$ (Moore neighborhood radius).
   • Misalignment: Controlled by an angle $\theta$ between the reward vectors of the two agents. The Receiver only values apples ($\langle 1, 0 \rangle$), while the Sender's preference shifts toward diamonds as $\theta$ increases (up to $180^\circ$).
   • Learning Mechanism:
     • Sender: Uses a discounted $\epsilon$-greedy bandit to select strategies $\langle p, c \rangle$.
     • Receiver: Uses Tabular Q-learning to decide whether to accept a contract and PPO (Proximal Policy Optimization) to decide whether to follow the Sender's action recommendation.

Key Results

1. Learning Optimal Signalling (RQ1 & RQ2)

• Convergence: In both environments, the Sender successfully learns signalling policies that maximize its cumulative reward, and the Receiver learns to best-respond to these signals.
• Impact of Observability: The Sender's strategy is highly sensitive to the Receiver's observation quality ($v$).
  • Low Visibility: When the Receiver has poor observations, the Sender tends to adopt strategies with low $p$ (prioritizing its own interests), as the Receiver is heavily dependent on the signal.
  • High Visibility: When the Receiver has better observations, the Sender is forced to increase $p$ (aligning more with the Receiver's interests) to maintain the Receiver's obedience, as the Receiver can rely more on its own data.
• Misalignment: As reward misalignment ($\theta$) increases, the Sender's influence shifts the Receiver's behavior toward the Sender's preferred outcomes (e.g., collecting diamonds), provided the Receiver's observability is low.

2. Impact of Contracts (RQ3 & RQ4)

• Sender's Advantage: When allowed to price information via contracts, the Sender learns to extract significant surplus, increasing its cumulative rewards compared to the non-contract signalling setting.
• Receiver's Cost: The introduction of contracts generally reduces the Receiver's accumulated reward. The Sender effectively monetizes its information advantage.
• Strategic Pricing:
  • Aligned Incentives ($\theta < 90^\circ$): The Sender offers useful information ($p \approx 1$) but charges a high price ($c \ge 0.8$), extracting more than half of the Receiver's surplus.
  • Misaligned Incentives ($\theta > 90^\circ$): The Sender shifts to self-serving signals ($p = 0$). Consequently, the Receiver learns that the information has low value, and the contract price drops to $c = 0$.
• Observability Interaction: In high-visibility settings, the Receiver is less reliant on the Sender, which limits the Sender's ability to extract surplus unless incentives are well-aligned.

Significance and Contributions

The paper claims to bridge three distinct fields: Bayesian Persuasion, Algorithmic Contract Theory, and Multi-Agent Reinforcement Learning.

1. Dynamic Bayesian Persuasion with Learning: Unlike static Bayesian Persuasion literature, this work models sequential decision-making where both agents learn. It demonstrates that optimal signalling is not static but evolves based on the Receiver's learning and observation capabilities.
2. Integration of Contracts: The study introduces linear contracts into the persuasion framework, showing how financial incentives can be used to align agents or, conversely, how they can be exploited by the informed party to extract surplus. This contrasts with information market models where the seller's utility is purely revenue-driven; here, the Sender's utility remains tied to the operational outcome (the Receiver's action).
3. Fairness and Incentive Design: The results highlight a tension between efficiency and fairness. While contracts allow the Sender to improve its rewards, they often come at the cost of "fairness," as the Sender extracts a large portion of the Receiver's surplus. This raises critical questions for the design of multi-agent systems, particularly in contexts like online marketplaces and human-AI collaboration, where information asymmetry is prevalent.

The authors conclude that while structured information and payments can enable coordination even with conflicting interests, the resulting dynamics often favor the information-rich agent, necessitating further research into mechanisms that promote balanced outcomes.