💰 quantitative finance

Robust Control under Stationary Ambiguity

This paper proposes the concept of "stationary ambiguity," a simulator design principle where parameter uncertainty is maintained as a non-decaying, state-dependent filter rather than resolving over time, to train control policies that preserve robustness against shifting latent factors in real-world sequential decision-making tasks like financial hedging.

Original authors: Konrad J. Mueller, Amira Akkari, Ben Wood, Lukas Gonon

Published 2026-08-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Konrad J. Mueller, Amira Akkari, Ben Wood, Lukas Gonon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to play a video game, but you don't know the exact rules of the game's physics engine. Maybe the gravity is slightly heavier than you think, or the friction on the ground is a bit slippery. This is the world of stochastic control, a branch of science where we try to find the best way to make decisions when the future is fuzzy and the rules aren't 100% clear. Usually, we train these robots in a computer simulation. But here's the catch: if we just guess the rules and stick with that guess, the robot might become a master of our specific guess but a total failure in the real world where the rules might be different. To fix this, scientists often use a trick called domain randomization. They tell the simulation, "Hey, let's pretend the gravity changes randomly every time we start a new game!" This forces the robot to learn a strategy that works no matter what the gravity is.

However, there is a sneaky problem with this standard trick. In a typical simulation, once the robot starts playing, it gets to watch the game unfold. If the gravity is set to "heavy" for the whole game, the robot quickly figures out, "Oh, it's heavy today!" and stops worrying about other possibilities. It becomes an expert at handling heavy gravity but forgets how to handle light gravity. This is called vanishing ambiguity: the robot's confusion disappears as it learns the specific conditions of that one game. But in the real world—especially in places like the stock market—the rules don't stay fixed. The "gravity" (or market volatility) can shift suddenly and unpredictably. If your robot has already decided, "I know exactly how this works," and the rules change, it will crash and burn.

This paper, titled "Robust Control Under Stationary Ambiguity," tackles this exact headache. The authors, working at the intersection of computer science and finance, argue that we need a new way to train our robots. Instead of letting the robot figure out the rules and then locking them in, they propose a method called stationary ambiguity. In this setup, the "rules" of the simulation are designed to keep shifting in a way that the robot can never fully pin down. It's like training a surfer in a pool where the waves change their size and shape constantly, not just once at the start, but throughout the entire session. The goal is to keep the surfer (or the trading robot) in a state of healthy uncertainty, so they stay ready for anything.

The researchers tested this idea on financial problems, specifically hedging, which is like buying insurance for a stock portfolio. They compared two types of training simulators. The first was the old-school "static" version, where the hidden rules (like market volatility) were picked once at the start and stayed the same. The second was their new "refresh" version, where the rules could randomly jump to a new value every now and then, keeping the ambiguity alive.

The results were clear. The robots trained on the old-school simulator got really good at predicting the market until the market suddenly changed. Once the "regime shift" happened, their performance tanked because they had become too confident in their old predictions. They specialized too quickly. On the other hand, the robots trained with stationary ambiguity never got too cocky. They kept a healthy level of doubt, which meant they were much better at adapting when the market rules changed. When the authors tested these robots on real historical stock market data, the "stationary" robots consistently lost less money and handled market crashes better than their overconfident counterparts.

The paper suggests that this isn't just a trick for finance; it's a fundamental design principle for any system where the environment is unpredictable and the controller can't force the environment to stay the same. By building simulators that maintain a steady level of mystery, we can create AI that doesn't just memorize the past, but stays robust enough to survive a surprise future.

Technical Summary: Robust Control Under Stationary Ambiguity

Problem Statement

In stochastic control problems, particularly in finance (e.g., optimal investment and derivative hedging), policies are often optimized within simulators. These simulators rely on parameters $x$ estimated from limited historical data, introducing parameter uncertainty (ambiguity). A common approach to handle this is Domain Randomization, where the simulator parameter $x$ is treated as a random variable drawn from a distribution $\nu$ at the start of each training episode.

However, this standard approach, termed Static Latent Model (SLM), suffers from a critical flaw: vanishing ambiguity. Because the parameter $x$ is fixed for the duration of a training episode, the control policy progressively observes the system's output $Y$ and infers the value of $x$ with increasing precision. Consequently, the policy's posterior belief about $x$ concentrates over time, causing the policy to specialize to the specific regime it has inferred.

This specialization creates a sim-to-real gap in ambiguity dynamics. In real-world systems like financial markets, latent regimes (e.g., volatility or correlation) often shift or drift over time. A policy trained under vanishing ambiguity becomes overconfident in its estimate of the current regime and fails to adapt when the regime changes, leading to poor performance and a loss of robustness.

Methodology

The authors propose a new modeling principle called Stationary Ambiguity to address the decay of robustness.

1. Formal Definition

Stationary ambiguity requires that the simulator induces a stationary filter process $\pi_t := P(X_t \in \cdot \mid Y_{\le t})$ .

In a stationary regime, the average amount of ambiguity about the latent state $X_t$ (measured by conditional variance) remains constant over time.
This prevents the systematic decay of uncertainty that occurs when a static parameter is inferred from an accumulating history of observations.

2. The Refresh Latent Model (RLM)

To approximate stationary ambiguity in practice, the authors introduce the Refresh Latent Model (RLM).

Instead of drawing a static parameter $X \sim \nu$ once per episode, the latent parameter $X_t$ is modeled as a stationary Markov process.
Specifically, the authors use a transition kernel where $X_t$ remains the same as $X_{t-1}$ with probability $1-\alpha$ , or "refreshes" to a new value drawn from $\nu$ with probability $\alpha$ .
Key Property: Because the latent parameter occasionally jumps to a new value, observations from the distant past become less informative about the current state. This prevents the filter from concentrating, maintaining a non-zero level of ambiguity throughout the control horizon.

3. Simulation Setup

To ensure the policy operates under stationary ambiguity from the start of the control problem:

Warm-up Period: The simulator is initialized in the infinite past (or a sufficiently long finite past $H$ ) in its stationary distribution. The policy observes this history before making its first control decision.
Initialization: The authors emphasize that deterministic initialization of latent states (e.g., setting variance $V_0 = v_0$ ) violates stationary ambiguity by creating a systematic increase in ambiguity after $t=0$ . They advocate for initializing latent states from the stationary distribution or using conditional initialization based on observed history.

Key Contributions

Analysis of Vanishing Ambiguity: The paper demonstrates that policies optimized under static randomization (SLM) progressively specialize to the inferred latent regime. In optimal investment and hedging tasks, this leads to a loss of robustness when the underlying regime shifts, as the policy fails to adapt quickly to new conditions.
Stationary Ambiguity Principle: The authors formalize the requirement for simulators to induce a stationary filter process. They provide sufficient conditions (e.g., the joint process $(X, Y)$ being a stationary Markov process) and a practical recipe (RLM) to achieve this.
Empirical Validation in Hedging: Through three distinct hedging problems (Volatility uncertainty in Black-Scholes, Correlation uncertainty in Heston, and Covariance uncertainty in multi-asset models), the authors show that policies trained under RLM (stationary ambiguity) maintain robustness over time, whereas SLM policies degrade.
Real-World Backtesting: In a large-scale case study using S&P 100 stock data, policies trained with stationary ambiguity outperformed those trained with static randomization and standard Black-Scholes benchmarks, particularly during periods of regime shifts (large volatility changes).

Results

Simulation Experiments

Investment Problem: Under static randomization, the optimal investment strategy becomes deterministic and specialized to the inferred drift. When the drift regime shifts, the SLM policy adjusts slowly, incurring significant utility loss. The RLM policy, trained with stationary ambiguity, maintains a distribution of actions that allows for faster adaptation to regime changes.
Hedging Problems (BS-VOL, HESTON-CORR, BS-COV):
- Initial Robustness: Both SLM and RLM policies outperform "plug-in" policies (trained on a single fixed parameter) when evaluated across a range of fixed regimes.
- Continual Robustness: When evaluated on paths with regime shifts (pre-training regime $\neq$ trading regime), SLM policies show a large spread in performance depending on the pre-training regime, indicating they have specialized. RLM policies show minimal spread, maintaining consistent performance regardless of the pre-training history.
- Filtering Behavior: Probing the internal state of neural network policies reveals that SLM policies learn to filter the latent state with high confidence (low variance) over time, while RLM policies maintain a higher, stable level of uncertainty, allowing them to treat new observations as potential evidence of a regime change.

Real Market Backtest

Using daily data from the S&P 100, the authors backtested hedging policies for ten different payoff structures. The training period covers January 2006 to December 2015, while the evaluation (backtest) period spans January 2016 to December 2025.
Overall Performance: RLM policies generally achieved lower spectral risk than SLM policies and standard Black-Scholes benchmarks (BS-EWMA, BS-HIST).
Regime Shifts: The performance gap widened significantly during periods of high volatility changes. The SLM policy, having specialized to the pre-training regime, performed poorly when the market regime shifted, whereas the RLM policy remained robust.

Significance and Claims

The paper claims that stationary ambiguity is a critical modeling principle for designing simulators for control problems driven by exogenous stochastic processes with shifting latent structures.

Robustness Preservation: The primary significance is that stationary ambiguity prevents the "over-specialization" of policies to specific inferred regimes, ensuring that robustness is maintained over time rather than decaying.
Practical Design Guide: The authors provide a concrete "recipe" for simulator design, advising practitioners to:
1. Randomize parameters using a stationary Markov process (like the RLM) rather than a static draw.
2. Initialize simulators in their stationary regime (via warm-up or conditional initialization) rather than deterministically.
3. Ensure the simulator's latent dynamics allow for regime shifts that the policy cannot predict with certainty.
Applicability: While the experiments focus on financial hedging, the authors argue that this principle applies to any sequential control problem where the environment's latent dynamics are uncertain and expected to change (e.g., inventory control, energy storage), provided the controller cannot influence the latent state through its actions (no exploration).

The paper concludes that ignoring the dynamics of ambiguity (i.e., allowing it to vanish) creates a fundamental gap between simulation and reality, and that enforcing stationary ambiguity is a necessary step for deploying robust control policies in dynamic, uncertain environments.