Evolution of noisy learning in games

1. Problem Statement

In evolutionary game theory and learning theory, individuals adapt their strategies based on the performance (payoffs) of those strategies. A critical parameter in these models is payoff sensitivity (often denoted as $\beta$ or selection strength), which determines how deterministically an individual switches to a strategy that yields a higher payoff.

• Traditional View: Most models treat $\beta$ as a fixed parameter. Low $\beta$ implies "noisy" or erratic learning (random switching), while high $\beta$ implies deterministic, rational learning.
• The Gap: It is generally assumed that higher sensitivity (more rational learning) is always evolutionarily advantageous because it allows individuals to better identify and exploit profitable strategies.
• Research Question: The authors challenge this assumption by asking:
  1. Do individuals with fixed but different sensitivities systematically outperform one another?
  2. If sensitivity ($\beta$) is itself an evolvable trait, does natural selection drive it toward infinity (perfect rationality), or can "noisy" learning (finite $\beta$) be an evolutionarily stable strategy?

2. Methodology

The authors employ a two-timescale framework combining short-run learning dynamics with long-run evolutionary dynamics.

A. Short-Run Dynamics: Introspection Dynamics

• Setup: Two players interact repeatedly in a symmetric $2 \times 2$ stage game (e.g., Prisoner's Dilemma, Snowdrift, Stag-Hunt).
• Mechanism: At each time step, a player compares their current payoff ($\pi$) with the hypothetical payoff of switching strategies ($\pi'$), keeping the opponent's strategy fixed.
• Update Rule: The probability of switching follows a Fermi function (logit rule):
  $$\phi = \frac{1}{1 + e^{-\beta_i (\pi' - \pi)}}$$
  where $\beta_i$ is the player's payoff sensitivity.
  • $\beta \to 0$: Switching is random (coin toss).
  • $\beta \to \infty$: Switching is deterministic (always switch if $\pi' > \pi$).
• Analysis: The authors model this as a Markov chain to analytically derive the stationary distribution of game outcomes ($CC, CD, DC, DD$) and calculate the expected average payoff for players with specific $\beta$ values.

B. Long-Run Dynamics: Adaptive Dynamics

• Setup: A large, well-mixed population where $\beta$ is an inheritable trait.
• Process:
  1. Individuals interact and learn strategies based on their fixed $\beta$.
  2. Fitness is defined by the average payoff obtained.
  3. Reproduction occurs based on fitness; offspring inherit the parent's $\beta$ with a small probability of mutation.
• Tools: The authors use Pairwise Invasibility Plots (PIPs) and calculate the selection gradient ($D(x)$) to determine evolutionary stability.
  • Convergence Stability: Does the population evolve toward a singular point?
  • Evolutionary Stability (ESS): Can a mutant invade a population at that point?
  • Branching: Does the population split into distinct sub-populations with different $\beta$ values?

3. Key Contributions

1. Co-evolution of Strategy and Sensitivity: The paper moves beyond fixed parameters to model the co-evolution of strategic behavior and the learning mechanism (sensitivity) itself.
2. Strategic Value of Noise: It demonstrates that "noisy" learning (finite $\beta$) is not merely a byproduct of cognitive constraints but can be a strategic advantage. In specific game structures, being less sensitive allows a player to manipulate the opponent's response.
3. Generalization of Learning Rules: While the baseline model uses Introspection Dynamics, the authors verify that similar results (finite $\beta$ or branching) hold for other learning rules, including Experience-Weighted Attraction (EWA), Stochastic Fictitious Play, and Reinforcement Learning.

4. Key Results

The evolutionary outcome depends heavily on the type of game interaction:

A. Prisoner's Dilemma

• Result: Sensitivity ($\beta$) tends to increase indefinitely ($\beta \to \infty$).
• Mechanism: In a Prisoner's Dilemma, the dominant strategy is Defection. Higher sensitivity allows players to identify and stick to Defection more reliably, maximizing individual payoff against any opponent. There is no strategic benefit to being "noisy."
• Outcome: The population evolves toward perfect rationality, leading to the Nash equilibrium of mutual defection.

B. Snowdrift Game (Chicken)

• Result: Sensitivity converges to a finite, evolutionarily stable value ($\beta^*$).
• Mechanism: This game requires anti-coordination (one cooperates, one defects).
  • If a player is too sensitive ($\beta$ is high), they rigidly stick to their preferred equilibrium.
  • If a player is less sensitive (noisy), they occasionally deviate from their preferred equilibrium. This "erratic" behavior forces the highly sensitive opponent to switch strategies to avoid the worst outcome (mutual defection).
  • The noisy learner effectively "traps" the sensitive learner into a more favorable equilibrium for the noisy player.
• Outcome: Evolution stops at a finite $\beta$ where the benefit of being slightly erratic balances the cost of making suboptimal moves.

C. Stag-Hunt Game

• Result: Evolutionary Branching occurs.
• Mechanism: The population splits into two sub-populations:
  1. Low Sensitivity ($\beta_{low}$): These players act randomly, occasionally coordinating on the efficient (but risky) mutual cooperation equilibrium ($CC$).
  2. High Sensitivity ($\beta_{high}$): These players are rigid and tend to play the risk-dominant strategy (Defection).
• Outcome: The coexistence of noisy and rational learners allows the population to access the high-payoff cooperative equilibrium more often than a population of purely rational agents could.

D. Harmony Games & Generalization

• In some Harmony games, finite $\beta$ can evolve due to similar strategic manipulation mechanisms (creating a "super-game" that resembles a Prisoner's Dilemma).
• The results hold for multi-player public goods games (linear, diminishing returns, and threshold benefits), showing that the strategic advantage of noise is robust across different interaction structures.

5. Significance and Implications

• Reframing Cognitive Noise: The study suggests that what appears to be "irrational" or "noisy" behavior in human decision-making may not be a cognitive limitation but an evolved adaptation to specific social environments.
• Red King Effect Parallel: The findings parallel the biological "Red King effect," where slower-evolving (or less responsive) species gain an advantage in mutualistic interactions. Here, the "slower" learner (less sensitive) gains an advantage by forcing the "faster" (more sensitive) learner to accommodate them.
• Multi-Agent Systems: The results have implications for Multi-Agent Reinforcement Learning (MARL). Agents that dynamically adapt their learning rates (sensitivity) based on the dilemma faced may achieve better collective outcomes than agents with fixed, high learning rates.
• Evolutionary Branching: The paper provides a theoretical mechanism for how heterogeneity in learning styles (some people being impulsive/noisy, others being calculated) can be maintained in a population through evolutionary pressure.

In summary, the paper establishes that noise in learning is a strategic asset in games with multiple equilibria or coordination requirements, challenging the dogma that evolution always favors perfect rationality.