💻 computer science

Risk-Aware Preference Learning for Stochastic Outcomes

This paper demonstrates that incorporating Cumulative Prospect Theory to model human risk sensitivity significantly improves reward function recovery and reduces regret in preference learning for stochastic robot navigation compared to traditional expected utility assumptions.

Original authors: Yi-Shiuan Tung, Yuni Wu, Wei Jiang, Alessandro Roncone, Bradley Hayes

Published 2026-07-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yi-Shiuan Tung, Yuni Wu, Wei Jiang, Alessandro Roncone, Bradley Hayes

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to behave like a polite human. You can't just write a long list of rules like "don't hit people" or "walk fast," because life is messy and full of surprises. Instead, scientists have found a clever trick: show the robot two different ways of moving and ask a human, "Which one looks better?" By doing this thousands of times, the robot can learn a hidden "scorecard" of what humans actually value. This is called preference-based reward learning.

However, there is a tricky assumption hiding in the background of most of these studies. It assumes that humans are like perfect calculators who simply add up the good and bad parts of a situation, weighing them by how likely they are to happen. This is called Expected Utility. But real humans aren't calculators; we are emotional. We worry way too much about rare disasters (like a car crash) and we hate losing things more than we love gaining them. This paper asks a simple question: What happens if we teach a robot using a model that assumes humans are perfect calculators, when the humans are actually risk-averse, emotional decision-makers?

The researchers at the University of Colorado Boulder decided to test this idea in a simulated world where a robot has to navigate through a crowd of pedestrians. In this world, every move the robot makes isn't a single, guaranteed path. Instead, it's like rolling a dice: the robot might glide smoothly, or it might bump into someone, or it might get stuck. The outcome is a cloud of possibilities, not a straight line.

The team set up a digital experiment with two types of "teachers" (the humans whose preferences the robot tries to learn). One teacher was a "perfect calculator" who only cared about the average outcome (Expected Utility). The other teachers were "risk-sensitive" humans who used a complex psychological model called Cumulative Prospect Theory (CPT). This model accounts for how real people overestimate the chance of rare, scary events and feel the pain of a collision much more sharply than the joy of a smooth path.

The researchers then trained two types of "students" (the learning algorithms). One student assumed the teacher was a perfect calculator (the EU learner), while the other student assumed the teacher was a risk-sensitive human (the CPT learner). They fed both students thousands of "which path is better?" questions generated by the teachers and watched how well they learned the true scorecard.

Here is what they found in their simulations. When the teacher was a perfect calculator, the calculator-student learned perfectly, while the risk-sensitive student got a little confused by the extra complexity. But when the teacher was a risk-sensitive human, the calculator-student performed significantly worse. It tried to explain the human's fear of rare crashes by distorting the recovered scorecard, making the robot think collisions were just "a little bit bad" instead of "catastrophic," resulting in a biased reward. The robot learned a less accurate lesson.

In contrast, the risk-sensitive student (the CPT learner) performed much better. It realized that the human wasn't just valuing the outcome differently; they were weighing the uncertainty differently. By separating the "value" of the outcome from the "fear" of the risk, the CPT learner recovered a scorecard with substantially lower error than the calculator-student.

The paper suggests that if we want robots to be safe and aligned with human values in the real world—where rare accidents are scary and people are naturally anxious about them—we can't just assume humans are logical math machines. We need to build robots that understand human fear and risk sensitivity, or else they might learn to take dangerous shortcuts because they think we don't mind the risk. While this result comes from computer simulations and not yet from real people testing real robots, the evidence points to a clear need: to teach robots how to be safe, we must first teach them how humans actually feel about the unknown.

Technical Summary: Risk-Aware Preference Learning for Stochastic Outcomes

Problem Statement
Preference-based reward learning is a standard paradigm for aligning autonomous systems with human intent, particularly in domains like social robot navigation where objectives (safety, efficiency, norms) are difficult to specify manually but easy to express via comparisons. However, existing approaches predominantly assume that human evaluators assess uncertain outcomes using Expected Utility (EU), which linearly aggregates outcome utilities weighted by their probabilities.

The authors identify a critical mismatch: behavioral economics demonstrates that humans are systematically risk-sensitive, often overweighting rare catastrophic events (e.g., collisions) and exhibiting loss aversion. In stochastic environments, assuming EU when the true human decision-maker is risk-sensitive can lead to structural errors in reward learning. Specifically, the learner may incorrectly encode risk sensitivity as differences in outcome utility, conflating what users value with how they weight uncertainty. This paper investigates whether modeling human risk sensitivity improves reward recovery when learning from preferences over stochastic robot outcomes.

Methodology
The study employs a Bradley-Terry preference learning framework to compare two models of human decision-making:

Expected Utility (EU): The standard baseline where the value of an action $A$ is the probability-weighted sum of outcome utilities: $V_{EU}(A) = \sum p_i \theta^\top \phi(o_i)$ .
Cumulative Prospect Theory (CPT): A nonlinear model that captures risk sensitivity through:
- Value Function: An asymmetric transformation of gains and losses, $v(x)$ , which incorporates loss aversion ( $\lambda > 1$ ) and diminishing sensitivity ( $\alpha, \eta$ ).
- Probability Weighting: A distortion function, $w(p)$ , that overweights rare events when the parameter $\gamma < 1$ .
- Decision Weights: Computed via rank-dependent transformation over cumulative probabilities.

Experimental Setup
The authors constructed a 2D social navigation simulation where a robot selects from seven meta-actions (e.g., forward, turn, stop) in dynamic environments populated by pedestrians (modeled via the Helbing–Molnár social force model).

Stochasticity: Each robot action induces a distribution over outcomes. For every action, $K=50$ stochastic rollouts were simulated to generate an empirical distribution of feature vectors (minimum clearance, deviation, progress, smoothness, collision count).
Synthetic Teachers: Preference data was generated by "teachers" using either the EU model or the CPT model. CPT teachers utilized two risk profiles: T&K (based on Tversky and Kahneman's empirical parameters) and Strong (amplified risk sensitivity).
Learners: Two classes of learners were trained on the generated pairwise comparisons:
- EU Learner: Assumes linear utility aggregation.
- CPT Learner: Jointly estimates reward weights ( $\theta$ ) and CPT parameters.
Evaluation: Performance was measured by held-out action regret, defined as the difference between the value of the action chosen by the true user model and the action selected by the learned model.

Key Results
The experiments yielded the following findings regarding reward recovery:

EU Teachers: When preferences were generated by an EU teacher, the EU learner achieved the lowest regret. The CPT learner did not outperform the EU learner, suggesting that introducing CPT parameters when the underlying process is linear adds unnecessary flexibility and reduces sample efficiency.
CPT Teachers: When preferences were generated by risk-sensitive (CPT) teachers, the CPT learner substantially outperformed the EU learner.
- As the number of training pairs increased, the regret for the CPT learner decreased steadily.
- The EU learner remained at a significantly higher regret level.
- The results indicate that when users exhibit risk sensitivity, an EU learner attempts to explain these choices by distorting the recovered reward function, whereas a CPT learner successfully separates reward weights from risk sensitivity.

Significance and Claims
The paper concludes that explicitly modeling human risk sensitivity via Cumulative Prospect Theory improves reward learning under uncertainty, particularly when the preference-generating process involves strong risk sensitivity. The authors argue that matching the learner's preference model to the teacher's risk-sensitive choice process is crucial for recovering accurate reward functions in stochastic domains.

The authors maintain a modest stance regarding the scope of these findings:

The results are presented as preliminary evidence derived from synthetic data.
The study demonstrates the theoretical and algorithmic advantage of CPT-based learners in simulation but explicitly states that human-subject validation remains future work.
The paper does not claim immediate deployment in real-world robots but highlights the importance of addressing the EU-CPT mismatch to avoid biased reward learning in safety-critical applications like social navigation.

Technical Summary: Risk-Aware Preference Learning for Stochastic Outcomes

More like this