⚡ electrical engineering

On Reward-Balancing Methods for Reinforcement Learning

This paper introduces and theoretically analyzes reward-balancing methods for discounted-return reinforcement learning by recasting them as an optimal control problem, extending the framework to handle model uncertainty via stochastic sampling, and demonstrating superior performance through scenario model predictive control simulations.

Original authors: Simone Baroncini, Bahman Gharesifard, Giuseppe Notarstefano

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Simone Baroncini, Bahman Gharesifard, Giuseppe Notarstefano

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a dog to perform tricks. In the world of Reinforcement Learning (RL), the "dog" is an AI agent, the "tricks" are actions it can take, and the "treats" are rewards.

Usually, we teach the dog by giving it a fixed set of rules: "If you sit, you get a treat. If you bark, you get nothing." The dog then tries millions of combinations to figure out which path leads to the most treats. This is how most AI learns today.

But what if the rules themselves were slightly confusing? What if the treats were scattered in a way that made the dog's brain work overtime just to understand the game, even though the best trick was obvious?

This paper introduces a clever new way to teach the dog: Reward Balancing. Instead of just waiting for the dog to learn, we change the rules of the game while the dog is playing, making the game easier to solve without changing the actual goal.

Here is the breakdown of their ideas using simple analogies:

1. The Problem: A Confusing Map

Imagine you are hiking in a forest (the State). You have a map (the Model) and a compass (the Policy).

The Goal: Reach the summit.
The Reward: You get points for getting closer to the top.
The Issue: Sometimes, the terrain is tricky. Even if you know the best path, the "points" system might be confusing. Maybe the path to the top gives you 10 points, but a path to a dead-end gives you 12 points because of a weird glitch in the scoring. The AI gets confused and wastes time exploring dead ends.

2. The Solution: "Normalizing" the Game

The authors propose a method called Reward Balancing. Think of this as a Game Designer who sits next to the hiker and constantly rewrites the scorecard.

The Old Way: The hiker tries to find the best path based on the original, confusing scorecard.
The New Way (Reward Balancing): The Game Designer looks at the hiker's current path. If the hiker is doing a good job, the Designer adjusts the scorecard so that the hiker's current path becomes the obviously perfect path.
- They don't change the forest (the physics of the world).
- They don't change the summit (the goal).
- They just tweak the numbers on the scorecard so that the "greedy" choice (the one that looks best right now) is actually the best long-term choice.

They call this "Normalization." It's like taking a messy, tangled knot of string and straightening it out so the ends are easy to find, without cutting the string.

3. The Secret Sauce: The "Group" of Changes

The paper dives deep into the math to prove that these changes are safe. They discovered that these scorecard adjustments form a "Group" (a mathematical club of rules).

The Analogy: Imagine you have a Rubik's Cube. You can twist it in many ways. Some twists scramble it; others solve it. The authors found a specific set of twists (transformations) that never change the fact that the cube can be solved; they just make the solution easier to see.
They proved that no matter how they twist the scorecard, the "best path" remains the same. It's like changing the currency in a video game from dollars to euros; the prices change, but the value of the items and the best strategy to win remain identical.

4. The Control Theory Twist: Steering the Ship

The authors then looked at this problem through the lens of Control Theory (how engineers steer rockets or self-driving cars).

The Analogy: Imagine the "Reward Scorecard" is a boat floating on a river. The "Normal" state (where the game is easy to solve) is a calm harbor.
The AI's job is to steer the boat into the harbor.
The "Reward Balancing" method is the steering wheel. The paper shows how to design the perfect steering wheel (a control law) that automatically guides the boat to the harbor, even if the wind (randomness) is blowing.

5. Dealing with Uncertainty: The Foggy Forest

In the real world, we don't always have a perfect map. The forest might be foggy, and we only have a rough guess of where the trees are.

The Problem: If you try to steer the boat using a bad map, you might crash.
The Solution: The authors created a "Scenario MPC" (Model Predictive Control) method.
- The Analogy: Instead of guessing the wind once, the captain looks at many possible futures (scenarios). "What if the wind blows left? What if it blows right?"
- The AI calculates the best steering move that works well on average across all these possible futures.
- The Result: Their simulations showed that this "Scenario" approach was much better at finding the optimal path than the old methods when the map was imperfect. It was like having a navigator who checks the weather forecast for 10 different days before deciding which route to take.

Summary: Why Does This Matter?

Faster Learning: By "balancing" the rewards, the AI doesn't waste time figuring out confusing math. It sees the best move immediately.
Parallel Power: Unlike some AI methods that have to learn step-by-step (like reading a book one page at a time), this method can do many calculations at once (like reading the whole book at once).
Robustness: It works even when we don't know the world perfectly, which is crucial for real-world robots, self-driving cars, and financial trading.

In a nutshell: This paper gives us a new way to "tune" the rules of a game so that the smartest move is always the easiest one to spot, even when the world is messy and uncertain. It turns a confusing puzzle into a clear path.

1. Problem Statement

The paper addresses the problem of solving Discounted-Return Reinforcement Learning (RL) problems within the Markov Decision Process (MDP) framework.

Context: Traditional RL algorithms (like Q-learning) iteratively improve a policy while keeping the reward function fixed.
Core Idea: The authors investigate Reward-Balancing Methods, a class of algorithms that take an alternative approach: they fix the policy to be "greedy" with respect to the current reward function and iteratively adjust the reward function itself.
Goal: The objective is to transform a given MDP into an equivalent "Normal MDP" where the optimal value function is zero. In such a normal form, finding the optimal policy becomes trivial: one simply selects the action with the zero reward in each state.
Challenge: The transformation must preserve the relative ordering of policies (i.e., the optimal policy must remain the same) and handle scenarios where the MDP model is unknown or subject to uncertainty (stochastic model sampling).

2. Methodology and Theoretical Framework

The paper develops a rigorous mathematical and control-theoretic framework for these methods.

A. Mathematical Foundations (Group Actions and Geometry)

Reward Transformations: The authors define a group of advantage-preserving transformations. These are affine transformations of the reward function $r \mapsto r + b(\delta)$ , parameterized by a state-dependent function $\delta$ .
Invariance: They prove that these transformations preserve the advantage function ( $adv_{\pi, r}$ ) and shift the value function by exactly $\delta$ ( $v_{\pi, r'} = v_{\pi, r} - \delta$ ).
Normal Form: An MDP is in "normal form" if its optimal value function is zero. The authors characterize the set of all normal reward functions (the "Normal Set," $\mathcal{N}$ ) and show that the space of all reward functions forms a principal fiber bundle over this normal set.
Symmetry: The framework incorporates MDP symmetries, showing that if the original MDP has symmetries, the reward transformations can be constrained to preserve them, allowing for dimensionality reduction.

B. Control-Theoretic Reformulation

The normalization process is recast as an Output Stabilization Problem and subsequently as an Optimal Control Problem (OCP).

System Dynamics: The reward vector $R_t$ is treated as the state of a linear time-invariant system:
$R_{t+1} = R_t + B\Delta_t$
where $B = \gamma F - S$ (derived from the transition matrix and projection), and $\Delta_t$ is the control input (the adjustment to the value function).
Objective: Design a feedback control law $\Delta_t = h(R_t)$ that steers the reward vector into the Normal Set $\mathcal{N}$ (where $\max_{u \in U_x} r(u) = 0$ for all states $x$ ).
Admissible Set: The authors define a set of admissible transformations that ensure the reward remains non-positive (a necessary condition for convergence to the normal set) and moves closer to the optimal value function.

C. Handling Model Uncertainty (Stochastic Setting)

The paper extends the framework to cases where the transition model $F$ is unknown and only available through stochastic samples ( $\hat{F}_t$ ).

Stochastic Dynamics: The update rule becomes a stochastic system with multiplicative noise: $R_{t+1} = R_t + \hat{B}_t \Delta_t$ .
Robustness: The authors identify the largest model-invariant admissible set ( $\mathcal{F}_I(R)$ ), a subset of control inputs that guarantees the reward remains non-positive regardless of the specific model realization.
Probabilistic Bounds: They derive rigorous probabilistic bounds on the deviation between the stochastic trajectory and the ideal (exact model) trajectory, proving that the system converges geometrically to the normal set under specific conditions.

D. Scenario Model Predictive Control (MPC)

To design robust control laws for the stochastic setting, the authors propose a Scenario MPC approach:

Instead of using a fixed heuristic, they solve a finite-horizon optimal control problem at each step using $N$ sampled model realizations (scenarios).
The cost function minimizes the cumulative "output" (distance to the normal set) while respecting the model-invariant constraints.
This allows for the design of control laws that are more robust and performant than standard heuristics under uncertainty.

3. Key Contributions

Theoretical Characterization: A comprehensive algebraic and geometric analysis of reward-balancing methods, establishing them as a group action on the space of reward functions and defining the "Normal Set" as a section of a principal fiber bundle.
Control-Theoretic Reformulation: Recasting the normalization problem as an optimal control problem. This provides a systematic way to design normalization laws using constraints and cost functions, moving beyond ad-hoc algorithms.
Robustness to Uncertainty: Extension of the framework to stochastic model sampling. The paper provides:
- A characterization of the largest model-invariant feasible set.
- Rigorous guarantees for normalization under uncertainty.
- Probabilistic bounds on the deviation of the stochastic process from the ideal trajectory.
Scenario MPC Design: Introduction of a Scenario MPC framework for designing robust reward-balancing laws that outperform existing state-of-the-art methods in uncertain environments.

4. Results

Convergence Guarantees: The paper proves that specific control laws (like "Full-Output Feedback" and "Safe Reward-Balancing") are normalizing. For the stochastic case, it proves that if the control law stays within the model-invariant set and satisfies a contraction condition, the reward converges geometrically to the normal set.
Probabilistic Bounds: Theorem 40 provides an exponential bound on the probability that the deviation between the stochastic reward trajectory and the ideal trajectory exceeds a threshold. This quantifies the stability of the method under model mismatch.
Simulation Studies:
- The authors tested the proposed Scenario MPC against the Full-Output Feedback method (a state-of-the-art baseline from previous literature).
- In a 2-state, 5-action MDP with model uncertainty (uniform noise), the Scenario MPC approach demonstrated significantly higher performance in terms of the percentage of runs where the resulting greedy policy was truly optimal.
- While Full-Output Feedback is computationally cheaper, the MPC approach showed that leveraging multiple scenarios allows for better handling of model uncertainty, leading to more reliable policy optimization.

5. Significance

Paradigm Shift: The paper shifts the perspective of RL from "learning a policy for a fixed reward" to "learning a reward that makes a greedy policy optimal." This offers a new angle for improving sample efficiency.
Parallelizability: The underlying reward-balancing methods are fully parallelizable, offering a potential speedup over sequential methods like Q-learning.
Robustness: By providing a control-theoretic framework with probabilistic guarantees, the paper bridges the gap between theoretical RL and practical applications where models are imperfect.
Design Framework: The formulation as an Optimal Control Problem opens the door for using advanced control techniques (like MPC, robust control, and constraint handling) to design next-generation RL algorithms that are more stable and efficient under uncertainty.

In conclusion, this paper provides a deep theoretical foundation for reward-balancing methods, transforming them from heuristic algorithms into a structured control problem with rigorous convergence guarantees, particularly in the presence of model uncertainty.