📊 statistics

Optimized Certainty Equivalent Risk Minimization Using Samples: Algorithms, Convergence Rates, and Applications

This paper proposes a stochastic gradient algorithm for optimizing Optimized Certainty Equivalent (OCE) risk by leveraging a novel characterization linking OCE to utility-based shortfall risk to derive non-asymptotic convergence rates and mean-squared error bounds for both the estimator and its gradient, with demonstrated applications in portfolio optimization and machine learning.

Original authors: Sumedh Gupte, Prashanth L. A., Sanjay P. Bhat

Published 2026-08-10

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Sumedh Gupte, Prashanth L. A., Sanjay P. Bhat

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a spaceship navigating through a sea of asteroids. You have a map, but it's blurry. You know where the asteroids usually are, but you can't be 100% sure where the next one will pop up. In the real world, this is exactly what financial investors, doctors, and machine learning engineers face every day. They have to make decisions based on data that is full of surprises. This is the world of risk management.

For a long time, people tried to measure risk by just looking at the "average" outcome. But averages are tricky; they hide the scary stuff. If you have a 99% chance of winning a dollar and a 1% chance of losing your entire life savings, the average looks great, but the risk is terrifying. To fix this, mathematicians invented special tools called risk measures. Think of these as a "worry-meter" that doesn't just count the average, but also screams when things get really bad. One of the most powerful of these tools is called the Optimized Certainty Equivalent (OCE). You can think of OCE as a super-smart calculator that asks, "How much guaranteed money would I need to feel just as safe as I do with this risky gamble?" It helps us find the perfect balance between being brave and being safe.

However, there's a catch. In the real world, we rarely know the exact rules of the game. We only have a pile of past data (samples) to guess what might happen next. The big question has always been: "Can we use this pile of data to find the best possible strategy using OCE, and can we prove it actually works?"

This paper is like a master mechanic handing you a new, high-performance engine for that spaceship. The authors, Sumedh Gupte, Prashanth L. A., and Sanjay P. Bhat, have built a brand-new way to calculate and optimize OCE using just a sample of data. They didn't just say, "It works." They built a mathematical proof showing exactly how fast their method converges to the right answer, even when the data is messy or the risks are huge. They created a "gradient estimator," which is basically a compass that points the way toward the safest path, even when you can't see the destination. They tested this compass on three different missions: picking the best mix of stocks (portfolio optimization), teaching computers to spot diseases (classification), and figuring out how unsure a computer is about its own predictions (uncertainty quantification). In every case, their new method didn't just work; it often beat the old, standard ways of doing things, proving that with the right math, we can navigate the stormy seas of uncertainty with much more confidence.

The Core Discovery: A New Compass for Risk

The main finding of this paper is a set of algorithms that allow us to find the "safest" decision using the OCE risk measure, even when we only have a limited number of data samples. The authors didn't just guess; they derived a specific formula to calculate the "slope" (or gradient) of the risk. Imagine you are walking down a foggy mountain trying to find the lowest valley (the safest spot). You can't see the whole mountain, but you can feel the ground under your feet. This paper gives you a tool to feel exactly which way is "down" based on a few samples of the ground, and it proves mathematically how many steps you need to take to reach the bottom.

They also tackled a tricky problem: what if the data includes "unbounded" variables? In plain English, this means what if the worst-case scenario isn't just "bad," but theoretically "infinite" bad? Previous methods often broke down here. The authors showed that their method holds up even when the risks are extreme, provided certain mathematical conditions are met. They proved that their estimates for the risk and the direction to move get better and better as you feed them more data, and they gave exact formulas for how fast that improvement happens.

What They Ruled Out and What They Proved

The paper is very careful about what it claims. It explicitly argues against the idea that you need to know the entire future distribution of data to make good decisions. You don't. You only need samples. It also rules out the notion that you must stick to simple, "smooth" utility functions (the math behind the risk measure). The authors showed their method works even for "rough" or "non-smooth" functions, which covers many popular risk measures like the Entropic Risk and Mean-Variance Risk.

However, the paper does not claim to have solved every risk problem in the universe. The results are based on mathematical proofs and computer simulations. They did not test this on live stock markets with real money changing hands, nor did they test it on live patients in a hospital. The "proof" is in the math and the simulations. The authors are very confident in their convergence rates (how fast the algorithm works) because they derived non-asymptotic bounds—mathematical guarantees that tell you exactly how close you are to the answer after a specific number of steps, rather than just saying "it gets better eventually."

The Three Missions: Where the Compass Worked

To show off their new engine, the authors ran three distinct experiments, like testing a new car on a racetrack, a dirt road, and a snowy mountain.

1. The Stock Market (Portfolio Optimization)
Imagine you have a basket of different stocks. You want to decide how much money to put in each one to get the best return without losing your shirt. The authors used their algorithm to pick the best mix of assets from real stock market data (S&P 500, FTSE, and Nasdaq). They compared their "OCE-optimal" portfolios against standard benchmarks like "Equal Weights" (putting the same money in everything) and "Maximum Sharpe Ratio" (a classic risk-adjusted return metric).

The Result: In simulations, the portfolios built with their OCE method often outperformed the benchmarks. For instance, on the S&P 500 data, the "Entropic Risk" portfolio achieved an annualized return of 0.1924 (19.24%), beating the "Benchmark - Sharpe" which got 0.1814. Even more impressively, when the market was tricky, their method adapted well, showing that it's a viable way to manage real-world money.

2. The Medical Detective (Classification)
Here, the goal was to teach a computer to tell the difference between sick and healthy patients using data from the UCI Heart Disease dataset and the Breast Cancer Detection dataset. The computer had to learn a model that minimizes risk, not just error.

The Result: The models trained with OCE were surprisingly good. On the Breast Cancer dataset, the "Smooth CVaR" model achieved an accuracy of 0.9883 and an AUROC (a measure of how well it separates sick from healthy) of 0.9963. This was comparable to, and in some metrics better than, standard methods like Logistic Regression. Crucially, the OCE models had lower "Expected Calibration Error" (ECE), meaning they were better at knowing how sure they were. For example, the Smooth CVaR model had an ECE of 0.0239, while the standard "NN BCE Loss" had 0.0437. This means the OCE models were less likely to be confidently wrong.

3. The Uncertainty Detective (Uncertainty Quantification)
This is about teaching a computer to say, "I don't know." In machine learning, it's dangerous when a model is confident but wrong. The authors used their method to improve a technique called Mean Variance Estimation (MVE), which tries to predict both the answer and how uncertain it is.

The Result: They replaced the standard "average" loss with their OCE risk criterion. The results were dramatic. On a test dataset, the "MVE with OCE" approach had a Mean Absolute Calibration Error (MACE) of 0.0285, compared to 0.0718 for the standard MVE and 0.0772 for Deep Ensembles (a very advanced competitor). This suggests that by using OCE, the model became much better at quantifying its own uncertainty, a critical feature for safety in AI.

The Takeaway

This paper doesn't just offer a new way to calculate risk; it offers a new way to optimize it. It provides a rigorous, mathematically proven toolkit that works with limited data, handles extreme risks, and has been tested in simulations to show it can beat traditional methods in finance, healthcare, and AI safety. While it hasn't been deployed in a live stock exchange or a hospital yet, the simulations suggest that if you want to navigate the fog of uncertainty, this new compass points the way more accurately than the old ones.

Technical Summary: Optimized Certainty Equivalent Risk Minimization Using Samples

Problem Statement

The paper addresses the optimization of the Optimized Certainty Equivalent (OCE) risk measure, a class of convex risk measures that generalizes popular metrics such as Conditional Value-at-Risk (CVaR), entropic risk, and mean-variance risk. While OCE is theoretically well-established, its practical application in machine learning and finance is hindered by a lack of efficient algorithms for optimization under uncertainty, particularly when dealing with unbounded random variables (r.v.s) and non-Lipschitz utility functions.

Existing literature often relies on asymptotic convergence guarantees or restricts analysis to bounded r.v.s and Lipschitz utility functions. Furthermore, previous optimization approaches have largely been limited to specific portfolio optimization settings or empirical risk minimization without non-asymptotic convergence rates for stochastic gradient (SG) methods. The core problem is to develop a framework for OCE estimation and optimization that:

Handles unbounded random variables.
Accommodates non-Lipschitz utility functions (e.g., entropic risk).
Provides non-asymptotic error bounds for both estimation and optimization.
Offers a generalizable gradient-based algorithm applicable to diverse domains like classification, regression, and portfolio management.

Methodology

1. Characterization and Extension to Unbounded Variables

The authors first extend the OCE formalism to unbounded r.v.s by imposing an integrability condition. A central theoretical contribution is a characterization linking OCE to Utility-Based Shortfall Risk (UBSR).

Proposition 1 establishes that the OCE of a random variable $X$ can be expressed as $OCE_u(X) = SR_{u',1}(X) + E[u(X - SR_{u',1}(X))]$ , where $SR_{u',1}(X)$ is the UBSR with a loss function $l = u'$ and threshold $\lambda = 1$ .
This characterization allows the authors to leverage existing results on UBSR to derive properties of OCE, including convexity and Wasserstein-distance bounds, even for non-Lipschitz utilities (Examples 1–4 in the paper).

2. Estimation via Sample Average Approximation (SAA)

The paper proposes an estimator for OCE using i.i.d. samples $\{Z_i\}_{i=1}^m$ :

Estimator Construction: The OCE estimator is constructed by first estimating the UBSR root ( $SR_m$ ) via a bisection search on the sample average of the derivative of the utility function, and then computing the OCE value using this root.
Error Bounds: The authors derive Mean-Squared Error (MSE) and Mean Absolute Error (MAE) bounds for this estimator. Crucially, these bounds hold under moment assumptions on the underlying r.v. and do not require the utility function to be Lipschitz, covering cases like entropic risk and mean-variance risk.

3. Gradient Derivation and Optimization

To optimize OCE with respect to a parameter $\theta$ (where the loss is $F(\theta, \xi)$ ), the authors derive an explicit expression for the gradient:

Gradient Expression: Using the OCE-UBSR link, Theorem 3 shows that $\nabla OCE(\theta) = E[u'(F(\theta, \xi) - SR(\theta)) \nabla F(\theta, \xi)]$ .
Gradient Estimator: A stochastic gradient estimator is formed by replacing the expectation with a sample average and using the estimated UBSR root from the current batch.
Convergence Analysis: The authors propose an OCE-SG (Stochastic Gradient) algorithm. They derive non-asymptotic convergence rates for three regimes:
- Strongly Convex: $O(1/n)$ rate for the parameter error.
- Convex: $O(1/\sqrt{n})$ rate for the function value gap.
- Non-Convex: $O(1/\sqrt{n})$ rate for the minimum gradient norm.
  These rates are achieved by using increasing batch sizes to control the bias of the gradient estimator.

Key Contributions

Theoretical Extension: The paper extends OCE analysis to unbounded random variables and non-Lipschitz utility functions, establishing convexity and providing Wasserstein-distance stability bounds.
Estimation Guarantees: It provides the first non-asymptotic MSE and MAE bounds for OCE estimation under general utility functions, bridging the gap between OCE and UBSR.
Optimization Algorithm: The authors introduce a gradient-based OCE-SG algorithm. Unlike prior works that offered only asymptotic guarantees or were limited to specific utility functions, this work provides non-asymptotic convergence bounds for strongly convex, convex, and non-convex objectives.
Gradient Characterization: A novel expression for the OCE gradient is derived, enabling the use of standard stochastic optimization techniques for a broad class of risk measures.

Experimental Results

The authors validate their approach through three application domains:

Portfolio Optimization: Using S&P 500, FTSE 100, and NASDAQ data, the OCE-SG algorithm was tested against benchmarks (Equal Weights, Maximum Sharpe, Hierarchical Risk Parity). Results indicate that OCE-optimal portfolios often outperform benchmarks in terms of cumulative returns and risk-adjusted metrics (Sharpe, Sortino), particularly when using smooth variants of CVaR or entropic risk. The algorithm successfully optimized various OCE instances (Examples 1–7).
Classification: On UCI Heart Disease and Breast Cancer Detection datasets, neural networks trained with OCE objectives (Entropic, CVaR, Monotone Mean Variance) were compared against Logistic Regression, Random Forest, and standard BCE-loss NNs. OCE models achieved higher accuracy, F1-scores, and AUROC on the Heart Disease dataset and lower Expected Calibration Error (ECE) on the Breast Cancer dataset.
Uncertainty Quantification: In a Mean-Variance Estimation (MVE) framework, replacing the standard expectation loss with an OCE risk criterion (using exponential utility) significantly outperformed both vanilla MVE and Deep Ensembles on a heterogeneous regression dataset, demonstrating improved calibration and uncertainty quantification.

Significance and Claims

The paper claims to provide a comprehensive, non-asymptotic framework for OCE risk minimization. Its significance lies in:

Bridging Theory and Practice: By providing non-asymptotic bounds, the work offers practical convergence guarantees for stochastic optimization, which are essential for finite-data scenarios in machine learning and finance.
Generality: The methodology is not restricted to specific utility functions or bounded data, making it applicable to a wider range of real-world problems where risk measures like entropic risk or quartic risk are preferred.
Algorithmic Efficiency: The proposed gradient-based approach allows for the optimization of complex risk measures using standard stochastic gradient descent, facilitating integration into existing deep learning pipelines.

The authors modestly note that while hyperparameter analysis and utility function selection are subjective and left for future work, the proposed algorithms serve as a viable and theoretically grounded candidate for optimizing OCE in diverse settings.