🤖 AI

Ergodic Risk Measures: Towards a Risk-Aware Foundation for Continual Reinforcement Learning

This paper introduces the first formal theoretical framework for continual reinforcement learning under risk-aware decision-making by demonstrating the incompatibility of classical risk measures with continual learning and proposing a new class of "ergodic risk measures" that successfully bridge this gap, supported by empirical validation.

Original authors: Juan Sebastian Rojas, Chi-Guhn Lee

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Juan Sebastian Rojas, Chi-Guhn Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Forever Student" vs. The "Risk-Aware" Student

Imagine a robot (or an AI agent) that doesn't just learn one game and then stop. Instead, it is a "Forever Student" that has to keep learning new games, new rules, and new environments for the rest of its life. This is called Continual Reinforcement Learning.

For a long time, scientists taught these Forever Students to be Risk-Neutral. This means the student only cared about the average score.

Analogy: Imagine a student who only cares about their GPA. If they can get a 3.0 every single day, they are happy. They don't care if one day they get a 5.0 and the next day they fail a test, as long as the average is 3.0. They ignore the "bad days."

However, in the real world, ignoring the bad days is dangerous. Sometimes, you need to avoid the "failures" even if it means your average score drops slightly. This is Risk-Awareness.

This paper asks a big question: Can we teach a Forever Student to be Risk-Aware? Can we teach it to care about avoiding disasters, not just maximizing the average?

The Problem: The Old Rules Don't Work

The authors discovered that the mathematical tools we usually use to teach "Risk-Awareness" (called Risk Measures) break when you try to use them on a Forever Student.

The "Static" Rule (The Final Exam): Some old tools only look at the very end of a test.
- The Problem: A Forever Student never finishes the test. It keeps going forever. If you try to use a tool that waits for the end, the student will wait forever and never learn anything.
The "Nested" Rule (The Crystal Ball): Other tools try to predict the future risk at every single step, assuming the future is perfectly consistent with the past.
- The Problem: In a changing world, the future isn't always consistent. If the student tries to stick to a rigid prediction, it can't adapt when the world changes. If it tries to adapt, the math says it's "broken."

The authors proved that these old tools are incompatible with a student that must learn forever. They are like trying to use a map of a city that was built 100 years ago to navigate a city that changes its layout every day.

The Solution: The "Ergodic" Compass

To fix this, the authors invented a new tool called Ergodic Risk Measures.

The Analogy: Imagine the student is walking through a forest that changes shape every hour.
- The Old Tools tried to memorize the entire forest from the beginning of time (impossible) or predict the entire future path (impossible).
- The New Tool (Ergodic) tells the student: "Don't worry about the whole forest. Just look at the last 10 minutes of your walk. Based on what you saw recently, make a safe decision for the next step."

This new approach has two superpowers that make it perfect for a Forever Student:

Finite Memory: It only needs to remember a short, recent window of time. It doesn't need to remember everything that ever happened.
Plasticity (Flexibility): Because it only looks at the recent past, it can instantly change its mind if the environment changes. If the forest suddenly gets a river, the student notices immediately and adapts.

The "Red Pill / Blue Pill" Experiment

To prove their new tool works, the authors ran a simple experiment with a game called "Red Pill / Blue Pill."

The Setup: The student has to choose between a "Red World" and a "Blue World."
- Blue World: Usually gives a steady, safe reward (good for a risk-neutral student).
- Red World: Usually gives a lower reward, but sometimes gives a huge reward if you are lucky, or a terrible penalty if you are unlucky.

Scenario 1: Changing Attitude
The student starts out being "Risk-Neutral" (doesn't care about danger). It learns to stay in the Blue World because it's safe and steady.
Then, the student's "attitude" changes. It becomes "Risk-Averse" (it hates the possibility of a terrible penalty).

Result: Using the new Ergodic tool, the student immediately realizes, "Oh no, the Blue World is actually risky for me now because I'm scared of the penalty!" It switches to the Red World, which turns out to be safer for its new personality. The student successfully adapted its behavior without forgetting how to learn.

Scenario 2: Changing Environment
The student keeps the same attitude, but the "Red World" and "Blue World" swap their reward patterns.

Result: The student notices the change in the recent past, recalculates the risk, and switches to the new "better" world. It never stops learning.

The Bottom Line

This paper is the first time anyone has built a solid mathematical foundation for teaching AI agents to be Risk-Aware while they are Learning Forever.

Old Way: You could be Risk-Aware, but only if you stopped learning eventually.
Old Way: You could learn forever, but only if you ignored risks and just chased the average.
New Way (This Paper): You can do both. By using Ergodic Risk Measures, an agent can constantly adapt to new dangers and changing environments without needing a super-computer memory or a crystal ball. It just looks at what happened recently, makes a smart, safe choice, and keeps moving forward.

Technical Summary: Ergodic Risk Measures for Continual Reinforcement Learning

Problem Statement

Continual Reinforcement Learning (Continual RL) aims to formalize lifelong learning and endless adaptation, requiring agents to balance the stability-plasticity dilemma: retaining useful prior knowledge while remaining flexible enough to adapt to new streams of experience. To date, Continual RL has been explored almost exclusively through a risk-neutral lens, where agents optimize expected long-run performance (e.g., average reward).

However, real-world deployment often requires risk-aware decision-making, where agents optimize measures of performance beyond the mean (e.g., variance, tail risks). A critical gap exists: the classical theory of risk measures, which underpins non-continual risk-aware RL, is fundamentally incompatible with the requirements of continual learning. Specifically, existing frameworks (static and nested risk measures) fail to satisfy the conditions necessary for an agent to be risk-aware over an infinite time horizon with finite memory.

Methodology

1. Formalizing Continual Risk-Aware RL

The authors first generalize the Continual RL framework (Abel et al., 2025) to a risk-aware setting. They define a risk-aware observation as a tuple of an environmental observation and the agent's current risk attitude. Consequently, a risk assessment is defined as a mapping from a sequence of risk-aware observations to a scalar value representing the risk of that sequence.

2. Axioms for Continual Risk-Awareness

Based on the assumption that continual agents possess finite memory and non-zero plasticity, the paper establishes two necessary and sufficient axioms for risk-awareness in continual settings:

Feasibility Axiom: Risk assessments must be computable based on a finite time interval of observations and the agent's risk attitude during that interval.
Plasticity Axiom: The agent's risk assessments and behavior must be influenced by new observations and changes in risk attitude infinitely often.

3. Incompatibility of Existing Measures

The paper demonstrates that classical risk measure theories fail these axioms:

Static Risk Measures: Fail the Feasibility axiom as the time horizon approaches infinity (requiring infinite memory) or fail the Plasticity axiom if the horizon is finite (ignoring future observations).
Nested (Dynamic) Risk Measures: Fail the Plasticity axiom because they enforce time consistency. Time consistency implies that if an outcome is deemed less risky at time $t$ , it must remain less risky at all future times, preventing the agent from adapting its risk assessment based on new information.

4. Proposed Solution: Ergodic Risk Measures

To resolve this incompatibility, the authors introduce Ergodic Risk Measures. These are defined as a sequence of conditional risk measures where:

The risk at any time $t$ can be accurately computed using only a finite, rolling window of preceding history.
Time consistency is not strictly enforced, allowing the agent to update its risk assessment as new information arrives.

The authors map this concept to the average-reward Markov Decision Process (MDP) formulation. They propose a risk-aware objective that optimizes the time-averaged risk of the limiting per-step reward distribution. Under standard ergodicity assumptions (e.g., unichain or communicating MDPs), they prove that this objective corresponds to an ergodic risk measure, satisfying both the Feasibility and Plasticity axioms.

Key Contributions

First Formal Treatment: This work presents the first formal theoretical treatment of risk-aware decision-making specifically within a continual (lifelong) learning setting.
Theoretical Incompatibility Proof: The paper rigorously proves that classical static and nested risk measures are incompatible with the axioms of continual learning due to their requirements for infinite memory or strict time consistency.
Ergodic Risk Measures: The authors propose a new class of ergodic risk measures that are compatible with continual learning. These measures allow for finite-memory computation and infinite adaptability (plasticity).
Theoretical Guarantees: The paper demonstrates that optimizing an ergodic risk measure in an average-reward MDP setting admits a stationary, deterministic optimal policy, circumventing the non-Markovian optimal policy issues often associated with risk-aware RL.
Empirical Validation: A case study utilizing Conditional Value-at-Risk (CVaR) as an ergodic risk measure is presented.

Results

The authors validate their theory using the RED CVaR Q-learning algorithm on two continual learning variations of the "Red-Pill Blue-Pill" (RPBP) task:

$\tau$ -RPBP Task (Changing Risk Attitude): The agent's risk parameter ( $\tau$ $τ$ ) shifts from risk-neutral to risk-averse over time.
- Result: The agent successfully learns to stay in the "blue world" (higher mean, lower CVaR) initially, and then correctly adapts to prefer the "red world" (lower mean, higher CVaR) once its risk attitude becomes risk-averse.
S-RPBP Task (Changing Environment): The reward distributions of the states change over time, requiring the agent to find the state with the best CVaR under a fixed risk attitude.
- Result: The agent continually adapts and successfully identifies the state with the superior CVaR as the environment changes.

These results demonstrate that ergodic risk measures enable agents to maintain sensible risk-aware behavior while adapting to changing risk attitudes and environmental dynamics.

Significance and Claims

The paper claims to provide a risk-aware foundation for Continual RL. Its primary significance lies in bridging the gap between risk-aware decision-making and lifelong learning.

Theoretical Foundation: It establishes that risk-awareness in continual settings requires a departure from time-consistent, nested risk measures toward ergodic measures that prioritize adaptability and finite computability.
Practical Utility: By showing that ergodic risk measures (like time-averaged CVaR) admit stationary optimal policies, the work offers a tractable alternative to complex, non-Markovian risk-aware objectives often found in non-continual settings.
Modest Scope: The authors position this work as the "first steps" and a "formal theoretical exploration." They do not claim to solve all challenges of risk-aware lifelong learning but rather provide the necessary theoretical conditions (axioms) and a compatible mathematical framework (ergodic risk measures) to enable future progress in developing "risk-aware lifelong agents."

The paper concludes that ergodic risk measures offer a new, appealing alternative for risk-aware learning that balances interpretability (like static measures) with a form of stability and adaptability (unlike strict nested measures), making them suitable for both continual and non-continual average-reward settings.