Technical Summary: Intention Inference Under Execution Noise

Problem Statement

In social dilemmas such as the Iterated Prisoner's Dilemma (IPD), the stability of cooperation relies on agents reliably observing each other's choices. However, under execution noise, intended actions are stochastically corrupted before enactment (e.g., a cooperative intent is executed as a defection). This creates an attribution problem: an observed defection may stem from hostile intent or merely from an action error.

Standard Markov Decision Process (MDP) formulations treat executed actions as the state space. This structural choice precludes the distinction between intent and execution, forcing agents to treat all observed defections as deliberate. Consequently, standard agents exhibit systematic over-retaliation, destabilizing cooperation. While engineered forgiveness strategies (e.g., Generous Tit-for-Tat) offer practical fixes, they lack a principled account of how much forgiveness is warranted by the evidence and cannot adapt to specific opponents. Reinforcement learning approaches often fail to model latent intentions, leading to the same structural inability to hypothesize that an opponent "intended to cooperate but executed incorrectly."

Methodology

The authors propose a Partially Observable Markov Decision Process (POMDP) formulation solved within the Active Inference (AIF) framework.

1. The Latent Intention POMDP

The core innovation is separating aleatoric uncertainty (execution noise) from epistemic uncertainty (uncertainty about intent).

• States ($S$): Encode joint intended actions (latent states) prior to execution (e.g., $CC$, $CD$, $DC$, $DD$).
• Observations ($O$): Correspond to executed joint actions, which are noisy versions of the states.
• Observation Model ($A$): A fixed binary symmetric channel with noise parameter $\epsilon$, mapping intended states to observed actions.
• Transition Model ($B$): Encodes the opponent's reactive policy (how their next intention depends on the previous joint outcome). This matrix is unknown and learned online.
• Preferences ($C$): Log-preferences over observations derived from the payoff structure (e.g., $T > R > P > S$).

2. Active Inference Solution

The agent minimizes Expected Free Energy (G), which decomposes into two components:

• Pragmatic Value: Drives the agent toward preferred, high-payoff observations (exploitation).
• Epistemic Value: Drives information-seeking actions to resolve uncertainty. This is further split into:
  • State Information Gain: Reducing uncertainty about the opponent's current latent intention.
  • Parameter Information Gain: Reducing uncertainty about the opponent's transition model (reactive policy).

3. Online Learning

The agent learns the opponent's reactive policy ($B$ matrix) via Dirichlet count accumulation. Belief updates over hidden states provide a "clean" training signal for the Dirichlet counts, decoupling the learning of the opponent's strategy from the noise of execution. This creates a coupled system where accurate intention estimates improve policy learning, and accurate policy learning improves intention estimates.

Key Contributions

1. POMDP Formulation: A novel separation of aleatoric execution noise and epistemic uncertainty about intent, with preferences consistent with payoff structures.
2. Online Learning Mechanism: Coupling state inference with model learning via Dirichlet posteriors, allowing the agent to actively learn the opponent's reactive policy through noise.
3. Critical Noise Threshold Analysis: Derivation of a theoretical threshold governing the collapse of cooperation. The authors identify that cooperation fails not because agents cannot distinguish noise from hostility (discriminability), but because the probability of cooperative intent persisting across the planning horizon drops below a critical point.
4. Empirical Validation: Demonstration that the value of intention inference is context-dependent, showing advantages against reciprocating opponents but fragility under mutual inference at high noise.

Results

Performance Against Different Opponents

• Self-Play (POMDP vs. POMDP):
  • At low noise ($\epsilon \le 0.10$), the POMDP maintains near-perfect cooperation, outperforming the MDP baseline.
  • At a critical threshold ($\epsilon \in [0.10, 0.15]$), the POMDP undergoes a sharp, catastrophic collapse in cooperation. This is driven by correlated belief dynamics: when one agent retaliates due to noise, the other correctly infers hostile intent and retaliates, creating a self-reinforcing spiral.
  • The MDP baseline degrades gradually without a sharp threshold, as it lacks the latent intention layer to trigger this specific belief-driven cascade.
• vs. Tit-for-Tat (TFT):
  • The POMDP maintains a consistent advantage across all noise levels. Because TFT's policy is stationary, the focal agent's belief dynamics do not destabilize the opponent, preventing the correlated collapse seen in self-play.
• vs. Win-Stay Lose-Shift (WSLS):
  • Both formulations collapse into exploitation (DC/DD cycles) at any $\epsilon > 0$. The POMDP's forgiving disposition delays this transition, resulting in marginally lower scores than the MDP, which adapts faster to exploit WSLS.

Mechanism of Failure

Analysis of the Expected Free Energy decomposition reveals that in the POMDP, the known observation model absorbs noise, allowing state information gain to resolve to zero. In contrast, the MDP suffers from irreducible state uncertainty, leading to persistent indecision and uncommitted policies.

The critical noise threshold is derived as the point where the cooperative survival probability ($p_t(CC)^h$) falls below 0.5. Beyond this point, the "hostile" hypothesis becomes the more probable explanation for observed defections over the planning horizon, making retaliation a rational response even if the agent can perfectly distinguish noise from intent.

Significance and Claims

The paper claims that the representational choice of what constitutes a state versus an observation determines whether principled cooperation under noise is structurally possible.

• Necessity of Representation and Drive: The cooperative advantages arise specifically from the interaction between the POMDP structure (which isolates aleatoric noise) and the epistemic drive (which actively probes the opponent's policy). Neither component alone suffices; removing epistemic terms eliminates the "explore-then-commit" trajectory, while using the POMDP without epistemic drive yields no improvement.
• Context-Dependent Value: Intention inference is beneficial against reciprocating opponents but becomes a liability in high-noise self-play scenarios where mutual inference creates a feedback loop of retaliation.
• Strategic Structure: The advantage is specific to games where intent attribution is decision-relevant (e.g., IPD where $T > R$). In games like the Stag Hunt ($T < R$), where there is no temptation to exploit, the POMDP and MDP perform similarly, as the distinction between error and intent is less consequential.
• Limitations: The authors note that the sharp collapse in self-play is a property of mutual inference rather than a flaw in the model itself. The model assumes a stationary opponent policy; when both agents co-evolve, this assumption is violated, leading to model misspecification. The paper suggests that forgetting mechanisms or change-point detection could mitigate this in future work.

In conclusion, the paper argues that accurate inference under noise is bounded by the context in which it operates: while separating noise from intent allows for robust cooperation against fixed or reciprocating strategies, it introduces a fragility in mutual settings where noise can trigger a self-reinforcing transition to mutual defection.