💻 computer science

Counterfactual Reasoning and Environment Design for Active Preference Learning

This paper introduces CRED, a novel active preference learning framework that enhances reward estimation in long-horizon robotic tasks by jointly optimizing environment design and trajectory selection through counterfactual reasoning to generate diverse, informative queries for human preference ranking.

Original authors: Yi-Shiuan Tung, Bradley Hayes, Alessandro Roncone

Published 2026-07-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yi-Shiuan Tung, Bradley Hayes, Alessandro Roncone

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to navigate the world, but you can't just hand it a rulebook. You can't say, "Always take the fastest route" or "Never cross the grass," because real life is messy. Sometimes you want speed, even if it means a bumpy ride; other times, you want safety, even if it takes longer. This is the heart of Preference Learning: a field where robots learn what humans want not by being told, but by watching them choose between different options. Think of it like a taste-tester at a restaurant. The robot doesn't know if you prefer spicy or sweet until it shows you two dishes and asks, "Which one looks better?"

However, there's a catch. If the robot just randomly shows you two paths, it might waste your time asking about routes that are obviously terrible or identical. This is called Active Preference Learning. The goal is to be a smart interviewer: ask the right questions to figure out your true preferences as quickly as possible. But in complex, long journeys, figuring out which questions to ask is incredibly hard. The robot often gets stuck guessing, failing to learn your unique style of decision-making, especially when it encounters new, strange environments it hasn't seen before.

This is where the paper introduces CRED, a clever new method that acts like a super-powered imagination engine for robots. The researchers, Yi-Shiuan Tung, Bradley Hayes, and Alessandro Roncone, propose that instead of just asking about the current world, the robot should "imagine" new worlds and ask "What if?" questions.

Here is how CRED works, using a simple analogy: Imagine you are trying to guess a friend's favorite ice cream flavor. A normal robot might just ask, "Do you like vanilla or chocolate?" over and over. CRED, however, is different. First, it uses Counterfactual Reasoning. It asks itself, "What if my friend loves mint but hates chocolate? What would they choose then?" It simulates these "what if" scenarios in its head, creating a diverse set of imaginary choices that cover all the possible ways its friend might think. This helps the robot generate much more interesting questions to ask, rather than boring, repetitive ones.

Second, CRED uses Environment Design. Imagine that your friend only likes chocolate when it's served in a fancy bowl, but vanilla in a plain cup. If you only ever serve them in plain cups, you'll never learn their true preference. CRED realizes that the setting matters. It "imagines" changing the environment—maybe turning a grassy field into a gravel road or changing the weather in a simulation—to see how that changes the choice. By tweaking the world itself, the robot can find the perfect scenario to ask a question that reveals the most about your preferences.

The researchers tested this idea in two ways. First, they used a digital grid world with different terrains like sand, grass, and gravel. Second, they used real-world map data from OpenStreetMaps to simulate delivery routes. They compared CRED against other top methods that just pick random paths or stick to the current environment.

The results were promising. In their simulations, CRED learned the "true" preferences much faster than the other methods. While other robots took about 25 tries to figure out the pattern, CRED often converged in just 15 iterations. More importantly, when the robot was dropped into a completely new environment it had never seen before, CRED's learned rules worked much better. It didn't just memorize the specific roads it practiced on; it learned the logic of the preferences, allowing it to generalize. For example, in the map tests, CRED reduced the error in reward prediction by a massive margin compared to the best previous method, achieving near-perfect accuracy in some cases.

The paper suggests that by combining "what if" thinking with the ability to redesign the world, robots can become much better at understanding us. They don't need to be told every rule; they can learn our values by exploring the space of possibilities, both in our choices and in the worlds we live in. While the current method requires some heavy computing power to run these simulations, the authors believe this approach could be the key to making robots that truly adapt to human needs in the real world.

Technical Summary: CRED for Active Preference Learning

Problem Statement

Effective deployment of autonomous robots, particularly in domains like delivery routing, requires adapting to human preferences that balance competing objectives such as time, cost, safety, and terrain conditions. While Preference Learning (PL) allows robots to learn reward functions from human trajectory rankings without manual specification, standard Active Preference Learning (APL) faces significant challenges. Existing methods often struggle to explore the full trajectory space efficiently, particularly in long-horizon tasks. They frequently rely on pre-generated trajectory sets or random rollouts, which may fail to identify informative queries. Consequently, learned reward functions often lack generalization to novel environments or scenarios not present during the initial training phase.

Methodology: CRED

The authors propose CRED (Counterfactual Reasoning and Environment Design), a query generation method designed to improve reward estimation by jointly optimizing environment design and trajectory selection. CRED operates within a Bayesian framework where the human reward function is modeled as a linear combination of trajectory features ( $R_H(\xi) = w^T \Phi(\xi)$ ), and the system maintains a belief distribution over the reward weights $w$ .

CRED addresses the limitations of prior work through two core mechanisms:

1. Counterfactual Reasoning

Instead of generating queries based on a single policy derived from the mean of the current belief, CRED explores "what if" scenarios.

Sampling: The system samples diverse reward weight vectors ( $w$ ) from its current belief distribution.
Policy Training: For each sampled weight vector, a distinct policy is trained (e.g., via PPO or value iteration) to maximize the corresponding reward.
Trajectory Generation: These policies are rolled out to generate a set of counterfactual trajectories, each representing a different hypothesized human preference.
Selection: The system evaluates pairs of these trajectories to maximize the information gain objective (mutual information between the query and the estimated weights), selecting the pair that most effectively reduces uncertainty in the belief distribution.

2. Environment Design

Recognizing that the environment itself influences the information value of queries, CRED treats environment parameters (e.g., terrain types, edge weights) as optimizable variables.

Bilevel Optimization: The problem is formulated as a bilevel optimization where the upper level selects environment parameters ( $\theta_E$ ) to maximize information gain, and the lower level finds the optimal trajectory pair for that environment.
Bayesian Optimization: Since the information gain objective is generally non-differentiable with respect to environment parameters, CRED employs Bayesian Optimization using a Gaussian Process (GP) with an Upper Confidence Bound (UCB) acquisition function to efficiently search the environment parameter space.
Dimensionality Reduction: In high-dimensional grid environments, Variational Autoencoders (VAEs) are used to compress the environment space into a lower-dimensional latent space to make the optimization tractable.

Key Contributions

Environment Design for APL: An approach that modifies environment parameters to generate diverse contexts, enabling the system to query human preferences across a wider range of scenarios and improving generalization.
Counterfactual Query Generation: A method that utilizes counterfactual reasoning to generate trajectory pairs reflecting varied hypothesized preferences, rather than relying on random rollouts or mean-belief policies.
Empirical Validation: A demonstration of the method's ability to generalize to novel environments through experiments in two distinct domains: a GridWorld with varying terrain types and real-world navigation using OpenStreetMap data.

Experimental Results

The authors evaluated CRED against state-of-the-art baselines, including Random Rollouts (RR), Mean Belief Policy (MBP), and ablated versions (MBP + ED, CR alone). The experiments utilized 10 simulated users with diverse ground-truth reward weights.

Information Gain: CRED generated preference queries with significantly higher initial information gain compared to baselines. This led to a more rapid decrease in belief entropy during the early stages of training.
Convergence Speed: CRED converged to the ground-truth reward function faster, typically within 15 iterations, whereas the best-performing baseline (RR) required approximately 25 iterations.
Generalization and Accuracy: In test environments (out-of-distribution scenarios), CRED demonstrated superior performance:
- GridWorld: Achieved an 84% reduction in reward difference and a 5% absolute increase in policy accuracy compared to the best baseline.
- OpenStreetMaps: Achieved a 97% reduction in reward difference and a 1% absolute increase in policy accuracy.
- While absolute gains in policy accuracy and Jaccard similarity for OpenStreetMaps appeared small, the authors note that baseline performance was already high (approx. 90-99%) due to strong biases in the reward function toward goal-reaching; thus, marginal improvements represent significant adherence to specific user preferences.

Significance and Limitations

The paper claims that CRED enables robots to learn human preferences more effectively and with fewer iterations by explicitly addressing the exploration of the trajectory space and the influence of environmental context. By "imagining" new scenarios and reasoning about counterfactual preferences, the system learns reward functions that generalize better to novel environments.

The authors acknowledge a primary limitation: the computational cost associated with training a reinforcement learning policy for each sampled reward weight. They suggest that future work could address this through parallelization or meta-learning approaches to develop a single policy capable of rapid fine-tuning for different reward weights.