💰 quantitative finance

Tackling Decision Processes with Non-Cumulative Objectives using Reinforcement Learning

This paper introduces a general mapping that transforms Non-Cumulative Markov Decision Processes (NCMDPs) into standard MDPs, enabling the direct application of existing reinforcement learning techniques to optimize arbitrary reward functions and demonstrating improved performance and training efficiency across diverse tasks.

Original authors: Maximilian Nägele, Jan Olle, Thomas Fösel, Remmy Zen, Florian Marquardt

Published 2026-10-01

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Maximilian Nägele, Jan Olle, Thomas Fösel, Remmy Zen, Florian Marquardt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, there is a powerful framework used to teach machines how to make decisions. Imagine a robot learning to walk, a computer program mastering a video game, or a trading algorithm managing a stock portfolio. These systems operate by taking a series of actions, one after another, in response to their surroundings. With every move, the system receives a signal, often called a reward, telling it whether that action was good or bad. For decades, the standard rule for success in these scenarios has been simple: maximize the total sum of all rewards collected over time. If a robot gets a small point for every step forward, the goal is to get as many points as possible by the end of the journey. This approach, known as a Markov decision process, has been incredibly successful, guiding everything from industrial robots to self-driving cars.

However, real life is often more complicated than a simple tally sheet. Sometimes, the most important outcome isn't the total amount of good things that happened, but rather the worst moment that occurred, or the consistency of performance over time. Consider a spacecraft landing on a planet. The goal isn't just to land safely; it is to ensure the craft never exceeds a dangerous speed during the entire descent, regardless of how smooth the rest of the flight was. In finance, an investor might care less about the total profit made over a year and more about how much that profit fluctuated, seeking a steady return rather than a risky gamble. These scenarios involve what researchers call non-cumulative objectives, where the final score depends on a specific function of the entire history of rewards, such as the maximum value reached or the ratio of average gain to volatility. Until now, teaching artificial intelligence to optimize these complex, history-dependent goals has been difficult, often requiring custom-built algorithms that are hard to apply to new problems.

A team of researchers from the Max Planck Institute for the Science of Light and Friedrich-Alexander-Universität Erlangen-Nürnberg has developed a general solution to this problem. They discovered a way to translate these complex, non-cumulative challenges into the standard format that existing, powerful artificial intelligence tools already know how to solve. Instead of inventing a new type of learning algorithm from scratch, they created a bridge. They showed that by slightly changing how the machine perceives its current situation and how it calculates its immediate feedback, any complex goal can be converted into a standard "sum of rewards" problem. This allows researchers to take the most advanced, off-the-shelf learning software available today and apply it directly to problems that were previously out of reach, without needing to modify the software itself.

The core of their method involves giving the artificial agent a bit more memory. In a standard setup, an agent only needs to know its current state to make a decision. But when the goal depends on the entire history of rewards—like remembering the highest speed reached so far—the agent needs to carry that information with it. The researchers proposed a system where the agent's "state" is expanded to include a running summary of the past, such as the highest or lowest reward seen up to that moment. Simultaneously, they adjusted the immediate reward the agent receives at each step. Instead of getting a reward that simply reflects the current action, the agent receives a calculated value that, when added up over the whole journey, perfectly reconstructs the complex goal. For example, if the goal is to minimize the maximum speed, the agent is rewarded in a way that penalizes it only when it sets a new speed record, effectively turning the "minimum of the maximums" problem into a standard sum.

This approach was tested across a wide variety of difficult tasks, proving its versatility. In a simulation of a lunar lander, the researchers trained an agent to land a spacecraft while strictly limiting its maximum speed. They compared their method against a standard approach that tried to approximate the goal by adding a penalty at the very end of the flight. The new method, which treated the speed limit as a continuous part of the learning process, found a much better balance between landing safely and moving efficiently. In the realm of finance, they applied the technique to portfolio optimization, where the goal is to maximize the Sharpe ratio—a measure of risk-adjusted return that divides average profit by the volatility of those profits. Previous methods had to rely on rough approximations of this ratio. By using the new mapping, the agents could learn to maximize the exact ratio directly, resulting in significantly better investment strategies during training.

The researchers also explored discrete optimization problems, such as finding the most efficient arrangement of quantum logic gates or simplifying complex diagrams used in quantum computing. In these tasks, the goal is often to find the single best state reached during a long search, rather than the sum of all improvements made along the way. Here, the new method allowed the agents to explore more boldly. Because the agent was not penalized for temporary setbacks that were necessary to reach a better solution later, it learned faster and found higher-quality solutions than agents trained with standard cumulative rewards. In one experiment involving quantum error correction, the new method improved performance by a significant margin, finding better solutions in less time.

The strength of this work lies in its simplicity and generality. The researchers did not create a new learning algorithm; they created a translation layer. This means that any expert in a specific field, from robotics to finance, can take their existing problem, apply this mapping, and immediately use the most powerful reinforcement learning tools available. The method works in both predictable environments and those full of random noise, and it handles both simple and complex goals. While the researchers noted that the expanded memory required for the agent can make the problem slightly larger, modern deep learning techniques are well-equipped to handle this. The result is a unified framework that removes the barrier between complex, real-world objectives and the sophisticated tools of artificial intelligence, opening the door for machines to learn strategies that were previously too difficult to define.

Technical Summary: Tackling Decision Processes with Non-Cumulative Objectives using Reinforcement Learning

Problem Statement

Standard Markov Decision Processes (MDPs) model decision-making by maximizing the expected sum of rewards over a trajectory, defined as $E_\pi [\sum_{t=0}^{T-1} r_t]$ . However, many real-world applications require optimizing Non-Cumulative Markov Decision Processes (NCMDPs), where the objective is to maximize the expected value of an arbitrary function $f$ of the rewards, rather than their sum:
$E_\pi [f(\tilde{r}_0, \tilde{r}_1, \dots, \tilde{r}_{T-1})]$
Examples of such objectives include maximizing the minimum reward (weakest-link problems), optimizing the Sharpe ratio (mean divided by standard deviation of returns), or minimizing the maximum velocity in control tasks. Existing methods for solving NCMDPs are limited: they often restrict objectives to specific functions (e.g., only the maximum), operate only in deterministic environments, or require modifying the core reinforcement learning (RL) algorithms (e.g., redefining temporal difference errors). Consequently, state-of-the-art RL approaches often rely on approximations or fail to handle stochastic NCMDPs effectively.

Methodology

The authors propose a general theoretical framework that maps any NCMDP to a standard MDP, thereby allowing the direct application of existing MDP solvers (such as dynamic programming or deep RL algorithms like PPO) without algorithmic modification.

The Mapping Mechanism

The core innovation is the construction of an augmented state space and adapted reward function. Given an NCMDP $\tilde{M}$ with states $\tilde{s}_t$ , rewards $\tilde{r}_t$ , and objective function $f$ , the corresponding MDP $M$ is defined as follows:

Adapted States ( $s_t$ ): The state space is expanded to include a history vector $h_t$ that encapsulates all necessary information from the reward history required to compute the objective function.
$s_t = (\tilde{s}_t, h_t)$
The vector $h_t$ is updated at each step via a function $u$ : $h_{t+1} = u(h_t, \tilde{r}_t)$ . For many common functions (e.g., max, min, Sharpe ratio), $h_t$ can be maintained with constant size.
Adapted Rewards ( $r_t$ ): The immediate reward in the new MDP is defined as the incremental change in the objective function value:
$r_t = f(\tilde{r}_0, \dots, \tilde{r}_t) - f(\tilde{r}_0, \dots, \tilde{r}_{t-1}) = \rho(h_t, \tilde{r}_t)$
This ensures that the cumulative sum of the adapted rewards in the MDP equals the value of the non-cumulative objective in the original NCMDP:
$\sum_{t=0}^{T-1} r_t = f(\tilde{r}_0, \dots, \tilde{r}_{T-1})$
Equivalence: The authors prove that the optimal policy for the constructed MDP is equivalent to the optimal policy for the original NCMDP. Furthermore, any policy yields the same expected return in both formulations. This allows standard convergence guarantees (e.g., for Q-learning or policy gradients) to apply directly to NCMDPs.

Implementation

The approach treats the NCMDP environment and the MDP solver as black boxes. A simple intermediate layer computes the adapted states and rewards, requiring no changes to the underlying RL algorithm. This facilitates the use of standard libraries (e.g., Stable-Baselines3) for non-cumulative problems.

Key Contributions

Theoretical Framework: Introduction of a general mapping from NCMDPs to standard MDPs that accommodates arbitrary non-cumulative objectives in both deterministic and stochastic environments.
Algorithmic Agnosticism: The method enables the use of any existing MDP solver without modifying the learning algorithm itself, unlike prior works that required redefining update rules.
State Space Management: Identification of conditions under which the additional state information $h_t$ remains constant in size, making the approach scalable for deep RL techniques despite the expanded state space.
Empirical Validation: Demonstration of the method's effectiveness across diverse domains, including classical control, financial portfolio optimization, and discrete optimization.

Experimental Results

The authors evaluated their approach against standard MDP formulations and prior NCMDP-specific methods across three domains:

Classical Control (Lunar Lander): The task involved maximizing a reward function penalized by the maximum velocity during a trajectory. The proposed method (MAXVELPPO) achieved a superior trade-off between landing safety and maximum speed compared to a cumulative approach (FINALPPO) that added the penalty only at the end. Adding history to the cumulative agent did not improve performance, whereas the mapping approach naturally integrated the history.
Portfolio Optimization: The goal was to maximize the Sharpe ratio. The proposed method (SHARPE) directly optimized the exact Sharpe ratio, outperforming agents using the differential Sharpe ratio approximation (DIFFSHARPE) or a final-reward formulation (FINALSHARPE).
Discrete Optimization:
- Peak Environment: In a toy problem requiring intermittent cost increases to find a global minimum, the non-cumulative objective (MAXREINFORCE) trained significantly faster than the cumulative baseline. Analysis showed this was due to reduced gradient variance and a more advantageous gradient direction.
- Quantum Error Correction & ZX-Diagrams: In tasks involving quantum circuit discovery and ZX-diagram simplification, the non-cumulative approach (MAXPPO) generally outperformed standard PPO. In the ZX-diagram task, the non-cumulative agent explored more effectively (higher policy entropy) and reached optimal performance in roughly half the training steps required by the cumulative agent.

Significance and Claims

The paper claims that this work significantly expands the scope of reinforcement learning by unifying non-cumulative objectives with standard MDP solvers. The authors emphasize that:

The method removes the need for specialized, often deterministic-only, NCMDP algorithms.
It allows researchers to specify exact objectives (e.g., Sharpe ratio, min-reward) directly to the agent without approximation.
The implementation is straightforward, requiring only a mapping layer, making it accessible to domain experts who are not RL specialists.
The approach improves both final performance and training efficiency in tasks where the objective depends on the trajectory's history in a non-additive way.

The authors note limitations, such as the potential increase in state space size (though manageable by deep RL) and the observation that in some discrete optimization tasks with long trajectories, learning can be initially slow if the agent frequently incurs zero reward. They also acknowledge that while their method improves performance within the trained environment, generalization to unseen periods (e.g., in financial data) remains a challenge common to all RL agents in these domains.