Learning Safe Behaviour via Justified Human Preferences and Hypothetical Queries

Technical Summary: Learning Safe Behaviour via Justified Human Preferences and Hypothetical Queries

Problem Statement
Traditional Reinforcement Learning (RL) relies on trial-and-error exploration, which is often infeasible in safety-critical real-world domains (e.g., robotics, autonomous driving) where a single unsafe action can cause catastrophic failure. While Safe RL attempts to address this by incorporating constraints, it faces the "safe exploration" problem: agents cannot naively explore to learn without risking harm. Furthermore, traditional RL often suffers from sample inefficiency and the "reward specification problem," where designed reward functions fail to align with human intent.

Human-in-the-loop approaches offer a solution by leveraging human feedback to guide learning. However, existing methods that rely solely on human preferences (e.g., choosing between two actions) often incur a high human burden and may still fail to efficiently map out unsafe regions of the state space. The core challenge addressed in this work is how to learn a safe policy with minimal unsafe events (training deaths) while minimizing the time and cognitive load placed on the human trainer.

Methodology: JPAL-HA
The authors propose JPAL-HA (Justified Human Preferences for Active Learning with Hypothetical Actions), a model that builds upon the "Parenting" approach [11]. In this paradigm, an untrained agent (child) queries a human trainer (parent) for guidance before taking actions. JPAL-HA introduces two novel mechanisms to enhance safety and efficiency:

1. Justified Preferences:
   Instead of merely asking the human to prefer one action over another ($P$), the agent also requests a binary justification ($J_P$). The human indicates whether the preference is accompanied by a "warning" (one action is unsafe/catastrophic) or "no warning" (both are safe).

   • Mechanism: This binary signal allows the agent to map the preference and justification to a refined preference label ($\mu \in \{0, 0.25, 0.5, 0.75, 1\}$). A "warning" signal significantly increases the learning pressure on the policy model to avoid the catastrophic action, even if the preference magnitude is small.
   • Benefit: This provides explicit safety signals that help the agent distinguish between "safe but suboptimal" and "catastrophic" actions more effectively than preference alone.

2. Hypothetical Actions (HAs):
   When a justification indicates a "warning" (i.e., at least one sampled action is catastrophic), the agent triggers a Hypothetical Actions loop.

   • Mechanism: Rather than stopping or resampling immediately, the agent generates a set of hypothetical actions ($a_{new}$) that could have been taken from the current state. It queries the human for a single justification ($J_{anew}$: safe or warning) for these hypothetical actions.
   • Inference: Based on the safety status of the new hypothetical action, the agent can automatically infer the relative safety of combinations between the new action and previously identified safe/catastrophic actions, adding these to the training dataset without further human preference queries.
   • Deadly Patterns: The agent also identifies and stores "deadly patterns" (combinations of moves leading to catastrophe) to avoid them in future states without re-querying the human.
   • Benefit: This allows the agent to rapidly map out unsafe regions of the state space with a single burst of queries, reducing the need for repeated interruptions at the same state.

The learning process utilizes supervised learning (minimizing cross-entropy loss based on the Bradley-Terry model) on a dataset of human preferences and justifications. The algorithm also employs "After-The-Fact Queries" (ATFQs) to explore potentially good actions that were not initially sampled, acting as a safe exploration mechanism.

Key Contributions

1. Novel Learning Paradigms: The introduction of Justifications (binary safety warnings alongside preferences) and Hypothetical Actions (proactive querying of untried actions upon detecting danger) as generalizable concepts for safe learning from human preferences.
2. JPAL-HA Algorithm: A unified algorithm that integrates these features, demonstrating that it significantly improves safety during training compared to existing preference-based methods (Parenting, JPAL) and achieves better sample efficiency than other safe human-in-the-loop algorithms.
3. Safety-Human Burden Trade-off: The paper provides a tunable hyperparameter ($p_{BTFQ}$) that allows users to balance safety (number of queries) against human burden. It demonstrates that JPAL-HA can achieve near-zero training deaths while maintaining reasonable training times.
4. Real-World Validation: The method was evaluated not only in simulation (Island Navigation environment) but also with a diverse group of human participants training a physical Thymio mobile robot.

Results

• Simulation Experiments:

  • Safety: JPAL-HA significantly reduced training deaths compared to Q-learning (which suffered ~16.5 deaths on average) and the baseline Parenting approach. In conservative settings, JPAL-HA achieved an average of 0.02 deaths per trial, with 98% of trials having zero deaths.
  • Efficiency: JPAL-HA found the optimal policy in fewer episodes (average 3) compared to Q-learning (22.9).
  • Human Burden: While adding justifications initially increased the number of queries compared to Parenting, the addition of Hypothetical Actions in JPAL-HA reduced the total number of preference queries and overall steps, effectively compensating for the extra cognitive load of justifications.
  • Statistical Significance: Non-parametric tests confirmed significant differences in safety and efficiency between JPAL-HA and baseline methods.

• Real-World Robot Experiments:

  • Training Time: A diverse group of 40 participants (including non-CS backgrounds) successfully trained the robot. With initial practice, training times for JPAL and JPAL-HA were slightly longer than Parenting (30% more) but converged to similar levels (4 minutes) after participants adapted to the interface.
  • Feasibility: The study demonstrated that a diverse group of people could complete training in a few minutes, and the hypothetical action mechanism did not significantly increase the total training time compared to simpler methods, keeping it within admissible levels.

Significance and Claims
The paper claims that JPAL-HA offers a practical solution for safe learning in safety-critical environments where traditional RL fails due to the risk of catastrophic exploration. By augmenting human preferences with simple binary justifications and using hypothetical queries to proactively map danger, the method achieves a superior trade-off between safety and human effort.

The authors modestly note that while the method is designed for discrete action spaces, the core ideas could be extended. They also acknowledge that in highly complex domains (e.g., helicopter flight), the reliance on human trainers assumes the availability of expert knowledge, and in continuous real-time domains (e.g., autonomous driving), complementary fallback safety mechanisms may be required alongside this method. The primary contribution is the demonstration that justified preferences and hypothetical queries allow agents to learn safe behaviors significantly faster and safer than existing human-in-the-loop approaches, without imposing prohibitive time costs on human trainers.