💻 computer science

CRED: Counterfactual Reasoning and Environment Design for Active Preference Learning

The paper proposes CRED, a novel active preference learning framework that enhances reward inference efficiency and accuracy by jointly optimizing environment design and counterfactual reasoning to generate highly informative trajectory comparisons for human feedback.

Original authors: Yi-Shiuan Tung, Gyanig Kumar, Wei Jiang, Bradley Hayes, Alessandro Roncone

Published 2026-03-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yi-Shiuan Tung, Gyanig Kumar, Wei Jiang, Bradley Hayes, Alessandro Roncone

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to behave, but you can't speak its language. You can't just write a rulebook saying, "Always take the grass path" or "Never hover over the laptop." Instead, you have to show the robot two different paths it could take and ask, "Which one do you like better?"

This is the core of Active Preference Learning (APL). The robot learns by asking you questions. But here's the problem: if the robot asks you the same boring questions over and over, or if it only asks about paths that look exactly the same, it will learn very slowly. It's like trying to guess someone's favorite flavor of ice cream by only asking them to choose between vanilla and vanilla.

The paper introduces a new method called CRED (Counterfactual Reasoning and Environment Design) to help the robot ask better questions. Think of CRED as a "Super-Teacher" that uses two special tricks to make learning faster and easier for you.

The Two Super-Tricks of CRED

1. The "What If?" Game (Counterfactual Reasoning)

Usually, a robot might just pick two random paths and ask you to choose. CRED is smarter. It plays a mental game of "What if?"

Imagine the robot has a hunch about what you like, but it's not sure. It thinks, "Maybe you hate grass, but maybe you actually love it."

The Old Way: The robot picks two paths based on its current best guess.
The CRED Way: The robot imagines different versions of you. It asks, "What if my guess is wrong? What if you actually prefer the gravel path?" It then creates a path that would be perfect for that "imaginary you."

By comparing a path for its "current guess" against a path for its "imaginary guess," the robot creates a question that highlights the biggest difference between your possible preferences. This forces you to make a choice that gives the robot the most useful information. It's like a detective comparing two very different suspects to figure out who the real culprit is, rather than comparing two people who look almost identical.

2. The "Stage Designer" (Environment Design)

Even if the robot asks a great question, the question might be useless if the setting is boring. Imagine trying to teach a robot to drive on a road, but you only ever show it a straight, empty highway. It will never learn how to handle a bumpy dirt road or a gravel path.

CRED realizes that the environment itself is a variable it can change.

The Old Way: The robot asks questions in the same fixed room or on the same fixed road every time.
The CRED Way: The robot acts like a stage designer. It says, "Okay, to figure out if you hate gravel, let's change the road to be all gravel for this specific question." It actively designs the world (changing terrain, moving obstacles, or altering wind) to create a scenario where your preference really matters.

By changing the "stage," the robot can create situations where the difference between "good" and "bad" behavior is crystal clear, making it much easier for you to answer.

How It Works Together

CRED combines these two tricks in a loop:

Imagine: It guesses different things you might like (Counterfactuals).
Design: It builds a specific environment where those different likes would lead to very different paths (Environment Design).
Ask: It shows you those two very different paths and asks, "Which one?"
Learn: Based on your answer, it updates its understanding of you.

The Results: Faster and Less Tiring

The researchers tested this in three scenarios:

Lunar Lander: A robot landing on the moon with wind blowing.
Tabletop: A robot carrying coffee across a messy table full of electronics.
Navigation: A delivery robot choosing between paved roads and grassy paths.

The findings were clear:

Accuracy: CRED learned the "true" human preference much faster and more accurately than older methods. It needed fewer questions to get it right.
Human Experience: When real people participated in the study, they found CRED's questions easier to answer. They felt less mentally tired (lower "mental workload") because the robot wasn't asking confusing or repetitive questions. The choices were clear and meaningful.

In a Nutshell

CRED is a smarter way for robots to learn from humans. Instead of randomly guessing what you want, it uses imagination to create "What if?" scenarios and changes the environment to make those scenarios obvious. This helps the robot learn your preferences quickly, with fewer questions, and without driving you crazy.

Technical Summary: CRED – Counterfactual Reasoning and Environment Design for Active Preference Learning

Problem Statement
As robotic systems operate in increasingly complex and unpredictable environments, explicitly specifying and balancing optimization objectives to achieve desired behaviors becomes infeasible. While Active Preference Learning (APL) offers a solution by inferring human reward functions from trajectory rankings, existing methods face significant limitations. Current approaches typically sample trajectory queries from fixed datasets or replay buffers. This restriction limits query diversity, often failing to identify informative comparisons necessary to distinguish between competing reward functions. Furthermore, generating queries within a single, fixed environment bounds the information value of the data, potentially leading to learned rewards that do not generalize well to unseen scenarios. The core challenge is to efficiently generate preference queries that maximize information gain while ensuring the learned reward function generalizes across diverse contexts.

Methodology
The paper proposes CRED (Counterfactual Reasoning and Environment Design), a novel framework for APL that jointly optimizes environment design and trajectory selection. The system operates as a bilevel optimization problem (Fig. 2):

Counterfactual Reasoning (Inner Optimization):
Instead of relying on random rollouts or fixed buffers, CRED maintains a Bayesian belief distribution over the unknown reward weights $w$ . It samples diverse weight vectors from this belief and asks counterfactual questions: "What if this specific weight vector were the true preference?" For each sampled weight $w_i$ , the system generates an optimal trajectory $\xi_i$ (using Reinforcement Learning or trajectory optimization methods like CHOMP) that maximizes the reward under that specific hypothesis. By evaluating the information gain (mutual information between the query and the reward weights) across pairs of these counterfactual trajectories, CRED identifies the most informative comparison to present to the user.
Environment Design (Outer Optimization):
CRED recognizes that the environment itself dictates which trajectories are feasible and which features are observable. To maximize query informativeness, the system treats environment parameters $\theta$ (e.g., terrain types, obstacle placements, wind conditions) as optimizable variables. Using Bayesian Optimization, CRED searches for environment configurations that, when combined with the counterfactual trajectories, yield the highest expected information gain. This allows the system to "imagine" new scenarios (e.g., altering terrain from grass to gravel) that better expose differences between competing reward hypotheses.

Key Contributions

Novel Query Generation Framework: CRED integrates counterfactual reasoning and environment design to move beyond fixed trajectory sets, enabling the synthesis of diverse and highly informative preference queries.
Bilevel Optimization Formulation: The paper formalizes the problem as a bilevel optimization where the outer loop selects the environment $\theta$ and the inner loop selects the trajectory pair $(\xi_A, \xi_B)$ , effectively coupling environment design with query synthesis.
Bayesian Inference Integration: The method utilizes an adaptive Metropolis algorithm to maintain a belief over reward weights, sampling diverse hypotheses to drive the counterfactual generation process.

Experimental Results
The authors evaluated CRED across three simulation domains (Lunar Lander, Tabletop Manipulation, and Navigation) and a user study involving 25 participants.

Reward Accuracy and Efficiency: In simulations, CRED consistently outperformed state-of-the-art baselines (Random Rollouts and Mean Belief Policy). It achieved higher final reward accuracy (e.g., 0.998 in Lunar Lander, 0.979 in Tabletop) and converged significantly faster, often surpassing 95% accuracy within 10 iterations.
Ablation Studies: Experiments confirmed that both counterfactual reasoning and environment design are complementary; removing either component resulted in lower accuracy or slower convergence.
Comparison with Domain Randomization: CRED significantly outperformed domain randomization (which samples environments uniformly at random), demonstrating that actively searching for informative environments via Bayesian Optimization is superior to random sampling.
User Study: In human-robot interaction tasks, participants interacting with CRED reported lower mental workload (measured by NASA-TLX) and found the trajectory comparisons easier to evaluate compared to baseline methods. CRED also achieved higher reward correlation with ground-truth preferences in both manipulation and navigation tasks.

Significance and Claims
The paper claims that CRED represents a significant advancement in active preference learning by addressing the scalability and generalization limitations of existing methods. By jointly optimizing the environment and the query, CRED enables robots to learn reward functions that are not only more accurate but also robust across different scenarios. The authors assert that their approach allows for more efficient learning with fewer user queries and reduces the cognitive burden on human users. The framework is presented as generalizable across different policy classes (e.g., PPO, value iteration) and domains, though the authors acknowledge the computational complexity of the bilevel formulation as a current limitation.