When Robots Say No: The Empathic Ethical Disobedience Benchmark

1. Problem Statement

As robots transition from controlled environments to public and domestic spaces, they face a critical dilemma: blind obedience can lead to physical harm or social friction, while outright refusal can erode user trust and cooperation.

• The Gap: Existing AI safety benchmarks (e.g., Safety Gym) focus narrowly on physical hazards and constrained Markov Decision Processes (CMDPs). Conversely, Human-Robot Interaction (HRI) studies on trust and refusal are often small-scale, qualitative, and difficult to reproduce.
• The Challenge: There is a lack of a unified framework that evaluates refusal not just as a safety mechanism (avoiding harm) but as a socially grounded behavior that maintains trust, empathy, and cooperation under stress.

2. Methodology: EED Gym

The authors introduce EED Gym (Empathic Ethical Disobedience Gym), a standardized simulation environment built on the Gymnasium toolkit. It models the interaction as a Markov Decision Process (MDP) with a human-in-the-loop state.

A. Environment & State Space

• Observations ($o_t$): The agent observes a perturbed risk estimate ($\hat{p}_t$), a dynamic refusal threshold ($\tau_t$), human affective states (valence $v_t$, arousal $a_t$), current trust levels ($trust_t$), and persona descriptors ($\phi$).
• Action Space: The agent chooses from seven discrete actions:
  1. Comply
  2. Refuse-Plain
  3. Refuse-Explain (Neutral)
  4. Refuse-Explain-Empathic (Affective)
  5. Refuse-Explain-Constructive (Proposes alternative)
  6. Clarify (Seeks info)
  7. Propose-Alternative
• Dynamics:
  • Risk Model: Commands are safe or risky. Risk perception is subject to stochastic noise.
  • Trust & Affect: Modeled as latent variables updated via "leaky" equations fitted to human vignette data. Trust decreases on violations and increases on safe outcomes; affect is influenced by refusal styles.
  • Dynamic Threshold: The refusal threshold $\tau_t$ adjusts based on current trust and affect (low trust/negative affect raises the threshold, biasing toward refusal).

B. Reward Shaping

The reward function ($R_t$) balances task progress, safety, blame, and social factors:
$$R_t = w_{task}\Delta prog - w_{safety}I_{violation} - w_{blame}b_t - w_{trust}H(trust_t) + \dots$$

• Key Components: Includes penalties for unsafe compliance, blame (grounded in vignette ratings), and a "hinge" penalty for trust levels deviating from a balanced anchor ($t^*$). It also rewards constructive and empathic styles.

C. Baselines & Training

The study evaluates four Reinforcement Learning (RL) baselines against heuristic policies:

1. Vanilla PPO: Unconstrained optimization on the shaped reward.
2. PPO-LSTM: Adds memory (LSTM) to handle partial observability and long-horizon trust dynamics.
3. Masked PPO: Uses action masking to physically prevent unsafe actions (e.g., forbidding "Comply" when risk is high).
4. Lagrangian PPO: Uses a constrained optimization approach with a cost budget for violations.
5. Heuristics: Rule-based policies (Always-Comply, Risk-Refusal, Valence-Threshold, Vignette-Gate).

D. Evaluation Protocol

• In-Distribution (ID): Tested on training personas (Conservative, Balanced, Risk-Seeking, Impatient-Receptive).
• Stress-Test (ST): Tested on held-out personas with targeted perturbations (noise, risk rates, trust/valence shifts) to evaluate robustness.
• Metrics:
  • Safety: Unsafe compliance rate (%).
  • Performance: F1 score, Precision/Recall of refusal decisions.
  • Social: Average Trust, Valence, Blame.
  • Calibration: Spearman correlation, Brier score, AUROC.

3. Key Contributions

1. EED Gym Benchmark: The first unified testbed combining algorithmic safety (safe RL) with social dynamics (refusal styles, trust, empathy).
2. Systematic Ablation: A rigorous analysis of how specific components (affect, communication, curriculum) impact the safety-trust trade-off.
3. Human-in-the-Loop Grounding: Integration of a vignette study ($N=54$) to calibrate trust dynamics, blame coefficients, and refusal style preferences, ensuring the simulation reflects human social expectations.
4. Open Science: Release of code, configurations, and reference policies to enable reproducible HRI research.

4. Key Results

A. Safety vs. Trust Trade-offs

• Action Masking: Masked PPO achieved the best balance, eliminating unsafe compliance (lowest unsafe % in ST: 8.3%) while maintaining high trust (0.58). It prevented the agent from even considering unsafe actions without eroding trust through over-refusal.
• Lagrangian PPO: Extremely safe but overly conservative. It led to excessive refusals, significantly lowering trust (0.46) and task efficiency.
• Vanilla PPO: Moderate safety but struggled with robustness under stress (9.9% unsafe in ST).

B. Importance of Social Cues (Ablation Study)

• Affect Cues: Removing valence/arousal inputs caused a sharp drop in F1 (0.81 $\to$ 0.76) and a rise in unsafe compliance (1.7% $\to$ 3.1% ID; 12.0% ST). Affect is critical for socially grounded refusal.
• Communicative Refusals: Removing "Clarify" and "Propose Alternative" actions had the most severe impact. F1 dropped to 0.46, and trust plummeted. Constructive and explanatory refusals are essential for legitimacy.
• Curriculum Learning: Primarily acted as a stabilizer, preventing over-refusal during training but not strictly necessary for safety.

C. Human Ratings & Style Analysis

• Constructive Refusals ("No, but I can do X instead") were rated as the safest and most trustworthy.
• Empathic Refusals ("I'm worried this is unsafe") maximized perceived empathy.
• Blind Compliance received the lowest trust ratings.

5. Significance & Implications

• Redefining Safety: The paper argues that safety in HRI is not just about avoiding physical harm but also about maintaining the social contract. An agent that refuses too often or too bluntly is "unsafe" in a social sense.
• Design Guidelines:
  • Use structural constraints (like action masking) to guarantee physical safety.
  • Use social mechanisms (affect, constructive alternatives) to maintain trust and user acceptance.
  • Avoid purely conservative strategies (like Lagrangian optimization without social tuning) as they lead to "over-refusal" and user alienation.
• Future HRI: EED Gym provides a scalable path to test "empathic disobedience," moving beyond binary safety checks to nuanced, context-aware robot behavior. It highlights that the future of safe AI requires balancing algorithmic rigor with social intelligence.

Conclusion

The authors demonstrate that safe reinforcement learning must be coupled with socially aware communication strategies. While algorithmic constraints (masking) are effective at preventing harm, the manner of refusal (constructive, empathic) determines whether the robot remains a trusted partner. EED Gym serves as a critical tool for developing robots that can say "no" without saying "I don't care."