📈 economics

Rationalizing Boltzmann Rationality: An Axiomatic Characterization of Entropy-Regularized Policies

This paper resolves the theoretical tension between the standard softmax policy and Markov decision process axioms by distinguishing between environmental chance and agent choice, demonstrating that imposing independence of irrelevant alternatives and monotonicity on choice nodes uniquely derives the Boltzmann policy and entropy-regularized representation as a normative design choice reflecting whether an agent values its own ability to choose.

Original authors: Silviu Pitis

Published 2026-07-21

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Silviu Pitis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a robot brain that needs to learn how to navigate a world full of choices. In the world of computer science, this field is called Reinforcement Learning, and it's all about teaching agents (like robots or game-playing AIs) to make decisions that get them the best rewards. To do this, the agent has to balance two things: sticking to what it knows works (exploitation) and trying new things to see if they're even better (exploration). For decades, the standard way to handle this "trying new things" has been a mathematical recipe called the softmax function. Think of it like a chef deciding how much of each ingredient to add to a soup: if one ingredient tastes slightly better, the chef adds a bit more of it, but never completely ignores the others. This recipe is so common that it's the default setting for almost every modern AI system. But here's the mystery: while everyone uses this recipe, no one could quite explain why it was the only right way to do it from the very beginning. In fact, using this recipe seemed to break some of the fundamental rules of how we usually think about making rational choices, creating a confusing tension between "what works" and "what makes sense."

This paper, titled "Rationalizing Boltzmann Rationality," acts like a detective story that solves that mystery. The author, Silviu Pitis, asks a simple but profound question: Can we prove that this specific "soup recipe" (the softmax policy) is the logical choice if we assume the agent wants to be rational and values its own ability to choose? The paper argues that the confusion came from mixing up two different types of randomness. Imagine you are at a crossroads. Chance is like the weather suddenly changing, forcing a detour on you no matter what you want. Choice is you deciding which path to take. The paper shows that if you treat the weather (chance) and your decision (choice) as separate things, and you apply a few common-sense rules about how you should value your options, the softmax recipe pops out as the unique solution. It turns out that the "entropy bonus"—a mathematical term that sounds like a penalty but is actually a reward for having options—is the option premium. It is the extra value gained from the freedom to choose. The paper proves that if an agent values its own ability to pick its path, it must use this specific formula to remain consistent. If it doesn't, it's not being rational; it's just being inconsistent.

The Story of the Robot Chef

Let's dive into the heart of the discovery. Imagine you are a robot chef in a kitchen with a menu of three dishes: Pizza, Tacos, and Sushi. You have a "score" for how good each dish tastes based on your past experience.

The Old Way (The Hard Choice): In the old, strict way of thinking, if Pizza has a score of 10 and Tacos has a score of 9, you would only ever pick Pizza. If Sushi had a score of 5, you'd ignore it completely. This is called a "hard" decision.
The New Way (The Soft Choice): But in the real world, and in most modern AI, we use a "soft" choice. We still pick Pizza the most often, but we give Tacos a small chance and Sushi a tiny chance. This is the Boltzmann policy (or softmax). It's like saying, "Pizza is the best, but maybe I'm hungry for something else today."

For a long time, scientists used this soft choice because it helped robots explore and learn faster. But there was a nagging problem. The math behind "hard" choices (called Expected Utility Theory) said that if you add a new, terrible option to the menu (like "Burnt Toast"), it shouldn't change your preference between Pizza and Tacos. This rule is called Independence of Irrelevant Alternatives (IIA). However, the soft choice math seemed to break this rule. If you added "Burnt Toast," the math suggested your love for Pizza might change just because the menu got bigger. This made the whole system feel shaky, like a house built on a foundation that didn't quite fit.

The Great Distinction: Weather vs. Will

The paper solves this by making a brilliant distinction between Chance and Choice.

Chance is when the universe flips a coin for you. If you are forced to eat a random dish from a bag, the value of that bag is just the average of the dishes inside. This is the "weather" part.
Choice is when you get to pick. When you have a menu, you aren't just eating a random dish; you have the power to choose. The paper argues that having a menu is valuable in itself. It's like having a key to a treasure chest versus having the treasure already in your pocket. The treasure is the same, but the ability to open the chest adds extra value.

The author shows that if you apply the rules of rationality only to the "Chance" parts (the weather) and apply a new set of rules to the "Choice" parts (your will), the magic happens. The rules for choice are:

Independence of Irrelevant Alternatives (IIA): Your preference between Pizza and Tacos shouldn't change just because Burnt Toast is on the menu.
Monotonicity: If Pizza gets a better score, you should be more likely to pick it.

When you combine these two simple rules with the idea that "having options is valuable," the math forces you into a single, specific shape. That shape is the Boltzmann policy. It turns out that the "entropy bonus" (the extra value we give to having options) is exactly the option premium—the value gained from keeping your options open. The paper proves that this isn't just a lucky guess or a convenient trick; it is the only way to be rational if you believe that having the power to choose is worth something and that your options are truly independent.

What This Means for the Future

The paper doesn't just say "this is cool"; it gives us a clear map of when to use this recipe and when to throw it away.

When to use it: If your robot is choosing between independent things (like different flavors of ice cream that don't affect each other), the Boltzmann policy is the perfect, mathematically proven choice. It's the only way to be consistent under these conditions.
When NOT to use it: The paper warns that if your options are "clumped" together, this recipe fails. Imagine a menu with 10 different shades of "Red Bus" and 1 "Blue Bus." If you treat every Red Bus as a totally separate choice, the robot might spend 90% of its time picking a Red Bus just because there are so many of them, even if it actually prefers the Blue Bus. This is called the "similarity effect." In these cases, the simple Boltzmann recipe breaks down because the assumption of independence is violated, and the robot needs a smarter menu system (like grouping the red buses together).

The authors also discovered something surprising about how "rational" the robot needs to be. They found a "minimum rationality threshold." If the robot is too unsure (too low on the "temperature" scale, or $\beta$ ), and it faces a situation where one choice could make its rewards explode (like an investment that could double or triple), the math breaks down completely. The robot's value becomes infinite and nonsensical. However, the paper shows that if we assume the robot understands that the future eventually stops mattering (a concept called "horizon continuity"), this problem disappears, and the math works again for any level of rationality.

The Big Picture

In the end, this paper does something rare in the world of complex math: it connects the dots between economics, information theory, and artificial intelligence. It shows that the same logic that explains why humans might choose a Gumbel-flavored ice cream (a specific type of random noise) also explains why AI should use the softmax function. It proves that the "entropy bonus" isn't just a fudge factor to make learning faster; it is the option premium. It is the value of being the one who makes the choice, rather than having the choice made for you.

So, the next time you see an AI making a decision with a little bit of randomness, remember: it's not just guessing. It's following a deep, mathematical law that says, "Having the freedom to choose is valuable, and here is the mathematically perfect way to honor that freedom when your choices are independent." The paper doesn't just justify the tool we've been using for years; it tells us exactly why it works, when we need to use it, and when we need to build a new one.

Technical Summary: Rationalizing Boltzmann Rationality

Problem Statement

The softmax policy, defined as $\pi(a | s) \propto \exp(\beta Q(s, a))$ , serves as the default model for stochastic choice in Reinforcement Learning (RL), underpinning methods like Soft Actor-Critic, entropy-regularized RL, and Reinforcement Learning from Human Feedback (RLHF). Despite its ubiquity, the literature lacks a derivation of the softmax form from normative first principles. Existing justifications rely on heuristics such as robustness, exploration smoothing, or translation invariance, but none uniquely derive the form while resolving a fundamental theoretical tension: the entropy bonus in the soft Bellman equation violates the Independence axiom of Expected Utility Theory (EUT). Specifically, the standard von Neumann–Morgenstern (VNM) Independence axiom implies that the value of a lottery is the weighted average of its components (Compound Lottery Reduction), whereas the soft Bellman equation adds an entropy term that strictly exceeds this average, suggesting that adding an option can decrease an agent's value.

Methodology

The paper resolves this tension by distinguishing between two types of randomness in an MDP: chance (environmental lotteries) and choice (behavioral selection). The methodology involves constructing an axiomatic framework that restricts the scope of standard rationality axioms to these distinct domains:

Domain Restriction: The authors restrict VNM Independence to "base prospects," which are deterministic policies subject only to environmental randomness. This preserves the standard Bellman relation for environmental transitions.
Axiomatic Characterization of Choice: At choice nodes, the paper imposes two new axioms on the policy and value functions:
- Independence of Irrelevant Alternatives (IIA):
  - Policy IIA: The ratio of probabilities between two actions depends only on the difference in their Q-values ( $Q_a - Q_b$ ).
  - Value IIA: The probability of selecting an action depends only on the loss in the menu's total value caused by removing that action.
- Monotonicity: Better actions receive higher selection probabilities, and expanding the menu of options cannot decrease the value of the choice node.
Derivation: The authors prove that within infinite-horizon MDPs (potentially with state-action-dependent discounting $\gamma(s, a)$ ), these axioms uniquely determine the functional form of the policy and value function.

Key Contributions

1. Axiomatic Characterization (Theorem 3.1)

The paper provides a complete axiom-to-algorithm path for the Boltzmann policy. It demonstrates that if an agent satisfies VNM preferences over base prospects and adheres to Policy IIA, Value IIA, and Monotonicity at choice nodes, the following must hold:

Policy: The choice rule is uniquely the Boltzmann policy: $\pi(a | s) = \frac{\exp(\beta Q(s, a))}{\sum_{a'} \exp(\beta Q(s, a'))}$ .
Representation: The policy is the unique maximizer of an entropy-regularized objective: $\max_{\hat{\pi}} [\sum \hat{\pi}_a Q(s, a) + \frac{1}{\beta} H(\hat{\pi})]$ .
Value Function: The value function is the Log-Sum-Exp (LSE) of the Q-values: $V(s) = \frac{1}{\beta} \text{LSE}(\beta Q(s, \cdot))$ .
Soft Bellman Equation: The value function satisfies the soft Bellman equation, where the entropy bonus emerges naturally as the "option premium"—the value of having the ability to choose versus having an outcome externally imposed.

The derivation relies on a novel step (Step 2) where Value IIA is used to derive the entropy bonus by comparing the cardinal value of a menu before and after an action is removed, without assuming an additive optimization structure a priori.

2. RL-Specific Consequences

The paper analyzes the implications of this framework for RL algorithms:

Convergence under Generalized Discounting: In settings with amplifying actions where $\gamma(s, a) > 1$ , the paper identifies a minimum rationality threshold $\beta_0$ . Below this threshold, the soft Bellman equation may lack a finite fixed point because amplifying actions receive positive weight even when suboptimal. However, the paper shows that the "horizon continuity" axiom (from prior work by Pitis, 2019) ensures the spectral radius of the discount-weighted transition matrix remains less than 1, resolving this divergence and guaranteeing convergence for all $\beta > 0$ .
Return Monotonicity: The actual (unregularized) expected return of the Boltzmann policy is non-decreasing with respect to the inverse temperature $\beta$ . Higher rationality (higher $\beta$ ) never harms the actual return, even though it reduces the entropy bonus.
Entropy Uniqueness: Within the Tsallis entropy family, only Shannon entropy (the limit as $q \to 1$ ) satisfies the IIA axioms. Other forms, such as sparsemax ( $q=2$ ), violate IIA, leading to qualitatively different choice behaviors.

3. Cross-Disciplinary Synthesis

The paper synthesizes independent lines of research from economics (Luce, Yellott, McFadden), information theory (Shore-Johnson), and bounded rationality (Fudenberg et al.) that converge on the Boltzmann form. It offers a normative assessment of when IIA is appropriate:

Appropriate: When actions are genuinely independent (e.g., distinct skills). In this case, Boltzmann is the unique consistent policy.
Inappropriate: When actions are correlated or redundant (e.g., the "red bus/blue bus" problem or LLM paraphrases). In these cases, IIA leads to probability mass distortion, and the paper suggests nested logit models as a remedy.

Results and Significance

The paper claims to resolve the basic tension between stochastic choice and expected utility theory by reframing the entropy bonus not as a violation of rationality, but as a consequence of valuing the ability to choose.

Normative Justification: The work establishes that the softmax form is not merely a mathematical convenience or an empirical heuristic, but the unique solution to a specific set of rationality constraints (IIA and Monotonicity) applied to the choice node.
Design Decision: The choice between "hard" (deterministic) and "soft" (stochastic) Bellman equations is framed as a design decision regarding whether the agent values its own agency. If the agent values the option premium, the soft Bellman equation is the axiomatic result.
Diagnostic Utility: The framework provides a concrete diagnostic for agent design: if the relative preference between two actions changes when irrelevant alternatives are added or removed, the agent violates IIA, and the Boltzmann policy is theoretically unjustified for that domain.
RLHF Implications: The paper notes that in RLHF, where pairwise comparisons often involve near-duplicate responses, the standard Bradley-Terry assumption (which implies IIA) can lead to perverse incentives. The axiomatic analysis suggests that grouping similar responses into "nests" (nested logit) is necessary to preserve rationality in such settings.

In summary, the paper provides a first-principles derivation of the Boltzmann policy and the soft Bellman equation, clarifying the conditions under which they are the uniquely rational choice for an agent and identifying the specific failure modes (correlated actions) where alternative structures are required.