Decoding fairness: a reinforcement learning perspective

Technical Summary: Decoding Fairness: A Reinforcement Learning Perspective

Problem Statement
The Ultimatum Game (UG) serves as a primary model for understanding the emergence of fairness. While orthodox Economics, relying on the Homo economicus assumption, predicts that rational, self-interested agents will propose the minimum possible offer and accept any non-zero offer (resulting in extreme inequality), behavioral experiments consistently show that humans prefer fair divisions (typically 40–50%) and reject low offers. Existing theoretical explanations for this discrepancy largely rely on the imitation learning framework, attributing fairness to exogenous factors such as population structure, reputation, spite, or empathy. This paper investigates whether fairness can emerge solely from endogenous factors within a Reinforcement Learning (RL) paradigm, where agents optimize strategies through self-reflection and interaction with their environment rather than by imitating successful peers.

Methodology
The authors model a two-player UG scenario using Q-learning, a standard RL algorithm.

• Agent Architecture: Each player is assigned two distinct Q-tables: one for the role of the proposer and one for the responder.
• State and Action Space:
  • Actions: Proposers choose from three offers ($p_l < 0.5$, $p_m = 0.5$, $p_h > 0.5$); responders choose from three acceptance thresholds ($q_l, q_m, q_h$).
  • States: The state space consists of nine combinations ($S = \{s_1, ..., s_9\}$) representing the action pairs of the previous round.
• Learning Dynamics:
  • Agents update their Q-values using the Bellman equation: $Q_{s,a}(t+1) = (1-\alpha)Q_{s,a}(t) + \alpha(\pi(t) + \gamma \max_{a'} Q_{s',a'}(t))$.
  • $\alpha$ (Learning Rate): Controls the weight of current experience; high $\alpha$ implies "forgetfulness," while low $\alpha$ implies retaining historical experience.
  • $\gamma$ (Discount Factor): Controls the weight of future rewards; high $\gamma$ implies a "long-term vision," while low $\gamma$ implies "short-sightedness."
  • Exploration: Agents explore random actions with probability $\epsilon$ and exploit the highest Q-value otherwise.
• Metrics: The study tracks the stationary fractions of offers and acceptance thresholds, specifically monitoring the prevalence of the fair option ($p_m, q_m$) versus the rational option ($p_l, q_l$).

Key Results

1. Emergence of Fairness via Endogenous Factors: Fairness emerges prominently when agents possess both low learning rates ($\alpha$, valuing historical experience) and high discount factors ($\gamma$, valuing future rewards). In this parameter regime, the system stabilizes with a high probability of fair offers ($p_m$) and fair acceptance thresholds ($q_m$).
2. Two-Phase Evolutionary Mechanism:
   • Phase I (Reaching Deals): Starting from random configurations, strategies leading to deal failures (e.g., high offers $p_h$ or high thresholds $q_h$) are eliminated due to low payoffs. The system converges to three surviving strategies: $(p_m, q_m)$, $(p_l, q_l)$, and $(p_m, q_l)$.
   • Phase II (Branching Process): A transition dynamic occurs where the mixed strategy $(p_m, q_l)$ is unstable. The system branches into two stable states: the fair strategy $(p_m, q_m)$ or the rational strategy $(p_l, q_l)$. The transition path is identified as $(p_m, q_l) \to (p_l, q_l) \to (p_m, q_m)$.
3. Alignment with Behavioral Experiments: The model reproduces key experimental observations:
   • The probability of successful deals increases with higher offers.
   • Offers below 20% are frequently rejected (approx. 80% rejection probability in the model), while offers between 45–50% are highly accepted.
4. Parameter Sensitivity:
   • $\alpha$ and $\gamma$: High $\alpha$ (forgetful) or low $\gamma$ (short-sighted) agents default to the rational, unfair strategy.
   • Game Parameters ($l, h$): The lower bound of offers ($l$) significantly impacts the equilibrium; as $l$ increases, the fraction of rational strategies rises. The upper bound ($h$) has a marginal impact.
5. Robustness: The emergence of fairness holds when role assignment is changed from rotating to random or fixed, and when the scenario is extended to a latticed population.

Significance and Claims
The paper claims that endogenous motivation within the reinforcement learning paradigm is sufficient to explain the emergence of fairness, rendering exogenous factors (like reputation or social norms) unnecessary for this specific outcome. By demonstrating that agents who "appreciate past experiences" and "plan for future rewards" naturally evolve toward fair strategies, the study provides a fresh perspective distinct from the imitation learning framework. The findings suggest that human fairness may be a result of adaptive learning processes aimed at maximizing accumulated payoffs over time, rather than a purely social or cultural construct. The work concludes that the self-reflective nature of RL offers a fundamental explanation for why humans deviate from the Homo economicus prediction in the Ultimatum Game.