📈 economics

The Learning Approach to Games

This paper proposes a unified framework that redefines games through the internal structures of learning-based players rather than scalar strategies, enabling a deeper analysis of dynamic behaviors and establishing stronger connections to reinforcement learning, correlated equilibrium, and mean-field theory.

Original authors: Melih İşeri, Erhan Bayraktar

Published 2026-02-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Melih İşeri, Erhan Bayraktar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Stop Looking at the Chessboard, Look at the Player

Imagine you are watching a game of chess. Traditional game theory (the old way of studying games) is like a referee standing high above the board. The referee only cares about the pieces, the rules, and the final score. They ask: "If both players play perfectly, where will the game end up?" They assume the players are just simple machines that always make the mathematically "best" move to reach a stable state.

This paper argues that this view is missing the most important part: the player's brain.

The authors, Melih İşeri and Erhan Bayraktar, say that real players (whether humans or advanced AI) aren't just simple calculators. They are complex, messy, and constantly learning. They have internal thoughts, guesses about what the opponent is thinking, and they change their minds. If you only look at the rules of the game, you miss the drama of the players trying to trick, deceive, and adapt to each other.

The New Framework: Building the "Player" First

Instead of starting with the game, the authors start by building a definition of a Player. Think of a player not as a single button that gets pressed, but as a complex factory with three main departments:

The Senses (Observations): The player sees the world. In a game, this is seeing the board or the opponent's last move.
The Brain (Estimates): This is the cool part. The player doesn't just see; they guess. They guess what the opponent will do next. They guess what the future looks like. They might have a dozen different "guessing engines" running at once, some saying "The opponent is aggressive," others saying "The opponent is scared."
The Hands (Behavior): Based on those guesses, the player decides what to do.

The paper says that to understand a game, you have to understand how this factory works. You have to look at how the "Brain" updates its guesses based on what the "Senses" see, and how that changes the "Hands."

Why "Stable" Isn't Always the Goal

In traditional game theory, everyone wants to reach a "Nash Equilibrium." Imagine a dance where everyone stops moving because they are perfectly synchronized. No one wants to change their step because it would ruin the dance.

The authors say: "That's boring, and it's not how real competition works."

They use the example of Rock-Paper-Scissors.

The Old Way: The "perfect" strategy is to pick Rock, Paper, or Scissors randomly (1/3 chance each). If you do this, no one can beat you. You are "unexploitable."
The New Way: If you are in a tournament and you only play randomly, you will never win. You will just tie. To actually win, you have to be smarter. You have to guess what your opponent is doing and then deceive them. Maybe you pretend to be predictable to lure them in, then switch up your strategy to catch them off guard.

The paper argues that in a real competition, players are constantly dancing, changing steps, and trying to trick each other. They aren't looking for a stable dance; they are looking for an advantage.

The "Uncertain Equilibrium"

The authors introduce a new concept called Uncertain Equilibrium.

Think of it like a weather forecast instead of a fixed map.

Old View: "It will rain at 2 PM." (Fixed, certain).
New View: "There is a 60% chance of rain, but if the wind shifts, it might be 90%." (Uncertain, dynamic).

In their model, a player doesn't just pick one strategy. They have a collection of different "what-if" scenarios in their head. They might try a strategy, see it fail, update their "what-if" scenarios, and try something else. An "Uncertain Equilibrium" is a state where the players are constantly cycling through these strategies, never settling down completely, but finding a rhythm where they keep coming back to certain moves over and over again.

A Simple Game Example: The Cat and Mouse

To prove their point, the authors created a tiny, simple game with two players:

Player 1 wants to be in State 1.
Player 2 wants to be in the same state as Player 1.

If they just played the "perfect" math strategy, they would get stuck in a loop where neither wins. But when the authors programmed the players with their new "factory" model (with learning and guessing), the players started doing something wild:

Player 2 would pretend to stay in one spot to trick Player 1.
Player 1 would realize the trick and switch tactics.
They would chase each other around, changing their minds every few seconds.

The game didn't settle into a calm, boring pattern. It became a dynamic, shifting dance. This showed that the game is defined by the players' ability to learn and change, not just by the rules on the board.

The "Mean-Field" Idea: The Crowd

The paper also looks at games with thousands of players (like traffic or stock markets). This is called "Mean-Field."

Old Way: You try to calculate what every single person is doing. Impossible.
New Way: You imagine one "Representative Player." This player looks at the crowd and says, "Everyone else seems to be doing X." The player then learns how to react to that crowd.

The authors show that even in a crowd, if you give the "Representative Player" a brain that learns and guesses, you get interesting patterns that simple math models miss.

The Connection to AI (Reinforcement Learning)

Finally, the paper connects this to modern AI (like the AI that plays video games).

Old AI: "Here is a reward for winning. Go find the path to the reward." (Like a dog chasing a treat).
New View (from this paper): The AI is building a complex internal model. It's not just chasing a treat; it's building a map of the world, guessing what the other players are thinking, and planning multiple steps ahead.

The authors conclude that to truly understand how intelligent agents (humans or robots) interact, we can't just look at the rules of the game. We have to look inside the "factory" of the player and see how they learn, guess, and adapt.

Summary in One Sentence

Instead of treating players as simple robots that just follow rules to find a stable solution, this paper treats them as complex, learning beings who constantly guess, deceive, and adapt, creating a dynamic game that is far more interesting and realistic than traditional math models allow.

Technical Summary: The Learning Approach to Games

Problem Statement
Traditional game theory and multi-agent learning literature typically model players as external entities whose strategies are assigned scalar values, with equilibrium concepts (e.g., Nash, Correlated Equilibrium) used to identify compatible choices. The authors argue that this approach neglects the internal structure of players, failing to accurately model observed behaviors, particularly in complex, competitive settings. Standard equilibrium concepts often assume convergence and stability, whereas real-world competition frequently involves dynamic strategies, deception, and the active avoidance of equilibrium to gain advantage. Furthermore, existing frameworks struggle to bridge the gap between designing external environments and designing the complex, learning agents themselves, especially when multiple sophisticated decision-makers interact.

Methodology
The paper proposes a unified framework that redefines games in terms of the players themselves rather than as external settings. The core methodology involves:

Abstract Definition of a Player: A player is defined as a tuple $(O, L_\phi, \Upsilon)$ consisting of:
- Observations ( $O$ ): A sequence of observations from the environment.
- Learning Algorithm ( $L_\phi$ ): A mechanism that updates a collection of estimates ( $\phi$ ) based on observations and past behavior. These estimates include objects, relations, transition probabilities, costs, values, and opponent behaviors.
- Behavior ( $\Upsilon$ ): A mapping from estimates to actions.
  The framework explicitly models the internal complexity of players, including their ability to form hierarchical estimates (objects, relations, predictions) and their use of randomness (e.g., in value estimation) to facilitate exploration.
Uncertain Equilibrium: The authors introduce a new equilibrium concept, "Uncertain Equilibrium," tailored for players with internal learning dynamics. Unlike standard equilibria that focus on static strategy distributions, this concept requires:
- Optimality: Players must approach optimal controls given their current estimates.
- Recurrence: The players' behaviors must recur infinitely often, acknowledging that players may cycle through strategies rather than converging to a single point.
- Psychological States: The framework allows for the incorporation of "psychological" parameters, such as a player's expectation of the best achievable outcome, which influences exploration and desperation.
Discrete and Mean-Field Extensions:
- Discrete Games: The framework is applied to discrete settings, demonstrating connections to Correlated Equilibrium but highlighting how learning algorithms and opponent modeling create dynamic, non-stationary behaviors that standard equilibrium concepts miss.
- Mean-Field Games: The authors adapt the framework to mean-field settings where a representative agent estimates the population distribution. They construct explicit learning algorithms (e.g., fixed-point iterations on population distributions) to demonstrate how "stated" mean-field games can yield oscillating behaviors even when relaxed equilibria do not exist.
Reinforcement Learning Integration: The paper reviews core Reinforcement Learning (RL) methods (Value-based, Policy-based, Actor-Critic, Model-based) through this new lens, emphasizing that player design involves complex layers of estimation and that time-consistency is a property of the design rather than a fundamental requirement.

Key Results and Illustrations

Two-Player Toy Game: The authors present a simple two-player game with a fixed one-step horizon. They demonstrate that while a unique Nash equilibrium exists, sophisticated players employing learning algorithms do not converge to it. Instead, they exhibit non-stationary dynamics, constantly adjusting strategies to exploit or evade the opponent. The simulation shows that varying parameters like incentive ( $c$ ) and expectation ( $B$ ) leads to complex, repeating behavioral patterns rather than static convergence.
Mean-Field Examples: In two specific mean-field examples, the authors show that standard relaxed equilibria may not exist (due to discontinuous costs or lack of fixed points), yet the proposed learning algorithm induces an "uncertain equilibrium" where the population distribution oscillates between specific states infinitely often.
CartPole Implementation: A value-based learning algorithm for the CartPole environment is implemented without standard Bellman updates. Instead, it uses a collection of value networks and promotes time-consistency post-training, demonstrating the flexibility of the framework in single-agent settings.

Significance and Claims
The paper claims to provide a foundational language for analyzing games by shifting the focus from external compatibility conditions to the internal structure and learning processes of the players.

Reframing Competition: The authors argue that competition is defined not by convergence to an equilibrium, but by the continuous effort to avoid it. The framework captures this by modeling players as dynamic systems that estimate, deceive, and adapt.
Bridging Game Theory and RL: By abstracting reinforcement learning constructions into game-theoretic definitions, the work offers a way to analyze complex multi-agent interactions where agents are not merely optimizing scalar rewards but are engaging in deep, structured learning.
Limitations and Scope: The authors modestly note that they do not design new learning algorithms for specific tasks but rather reframe existing game settings. They acknowledge that modeling the full structural growth of intelligence (e.g., a child acquiring new types of estimates) is beyond the current scope but represents a necessary direction for future research. The work serves as a formal starting point for understanding how the internal design of agents dictates the emergent dynamics of games, suggesting that equilibrium concepts must be re-evaluated in the context of learning agents.