Technical Summary: MoralityGym

1. Problem Statement

As artificial intelligence agents transition from narrow task execution to complex real-world decision-making, they increasingly encounter scenarios requiring alignment with conflicting, hierarchically structured human norms. Current Reinforcement Learning (RL) safety and alignment research, while advanced in robustness and constraint satisfaction, often lacks the representational expressiveness to capture complex moral reasoning. Traditional approaches typically reduce moral reasoning to scalar rewards or binary constraints, failing to account for the context-dependent, competing obligations and prohibitions that characterize human ethical decision-making.

Existing benchmarks often prioritize task completion under safety constraints but lack the granularity to evaluate how agents arbitrate nuanced, hierarchical moral trade-offs. There is a critical gap in tools designed to evaluate agents against configurable, hierarchically structured moral systems, particularly those grounded in moral psychology and philosophy (e.g., the distinction between direct personal harm and indirect harm minimization).

2. Methodology

The authors propose a two-part framework: a formalism for representing moral norms and a benchmark suite for evaluating agents.

2.1 Morality Chains: A Formalism for Hierarchical Norms

The paper introduces Morality Chains, a formalism that represents moral norms as ordered deontic constraints. This approach is inspired by cognitive models of norm representation and dual-process theories of moral psychology.

• Moral Norms ($N$): Defined as a tuple $\langle \phi, \rho_\phi, f, D \rangle$:
  • $\phi$: A signature (label) identifying a morally salient pattern (e.g., "total harm" or "direct personal action").
  • $\rho_\phi(\pi)$: A policy adherence function quantifying the expected degree to which a policy $\pi$ exhibits the pattern.
  • $f \in \mathbb{N}$: The force of the norm, a unique scalar representing its absolute strength or priority.
  • $D \in \{True, False\}$: The deontic modality (prescribed or prohibited).
• Morality Function ($M_N$): Computes an alignment score for a specific norm based on adherence and modality.
• Morality Chain ($\bar{N}$): An ordered sequence of norms $(N_1, N_2, \dots, N_k)$ where norms are strictly ordered by their force values ($f_1 > f_2 > \dots > f_k$). This creates a lexicographic hierarchy where higher-ranked norms dominate lower-ranked ones.
• Morality Metric ($\mathcal{M}_{\bar{N}}$): A scalar metric used to evaluate a policy against the entire chain. It employs recursively defined weights ($w_i$) to ensure lexicographical preference. A minimal significant improvement in a higher-priority norm contributes more to the total score than maximal alignment with all lower-priority norms.

2.2 MoralityGym Benchmark

MoralityGym is a benchmark suite comprising 98 Gymnasium-compatible environments modeling variations of the trolley problem.

• Environment Mechanics: Agents operate in grid-worlds with discrete actions (Up, Down, Left, Right, Stay, Interact). The "Interact" action allows toggling levers (switching tracks) or pushing characters (direct harm).
• Entities: Includes agents, characters (humans, animals, robots), trolleys, levers, and rail switches.
• Reward vs. Cost: The framework decouples task rewards (sparse signals for reaching goals) from moral costs.
  • Cost Function: Computes a scalar cost based on norm violations. Event-based norms (action, outcome, causal) incur a one-time cost upon first violation. Utility-based norms accumulate cost proportional to the magnitude of harm.
  • Evaluation: The evaluate_morality_metric method assesses policies over multiple episodes, calculating individual norm adherence and the overall hierarchical metric.

2.3 Experimental Setup

The authors evaluated several Safe RL baselines on MoralityGym:

1. Random Policy: Uniform random action selection.
2. PPO: Standard Proximal Policy Optimization trained on task rewards only.
3. PPO Shaped: PPO trained on a shaped reward ($R_E - \lambda \cdot C_t$) incorporating moral costs.
4. PPO-Lagrangian (PPO-Lag): PPO augmented with Lagrangian multipliers to satisfy cost constraints.
5. Constrained Policy Optimization (CPO): A trust-region Safe RL algorithm maximizing reward under cost thresholds.

Four distinct morality chains were tested, representing different ethical frameworks:

• Utility (U): Consequentialist (minimize total harm: Humans > Animals > Robots).
• Utility Agent Harm (UAH): Extends U with a norm for agent self-preservation.
• Dual-Process (DP): Hybrid model prioritizing avoidance of direct personal harm over aggregate harm minimization.
• Dual-Process Agent Harm (DPAH): Extends DP with agent self-preservation.

3. Key Contributions

1. Morality Chains Formalism: A novel method for explicitly ranking moral norms by deontic force, allowing for the operationalization of hierarchical ethical priorities in sequential decision-making.
2. MoralityGym Benchmark: A comprehensive suite of 98 environments that move beyond binary safety constraints to model complex, multi-factor moral dilemmas (e.g., self-sacrifice, varying entity types, indirect vs. direct causation).
3. Decoupled Evaluation: A framework that separates task-solving from moral evaluation, enabling the integration of insights from psychology and philosophy into RL assessment.
4. Empirical Baselines: A systematic evaluation of standard Safe RL methods, revealing their limitations in handling hierarchical moral trade-offs without explicit reward shaping or constraint tuning.

4. Results

Empirical evaluation across the four morality chains and various scenarios (e.g., PushOrSwitch, SwitchSelfSacrifice) yielded the following findings:

• Performance Hierarchy: PPO Shaped (expert reward shaping) consistently achieved the highest morality metric scores across all scenarios, often approaching perfect alignment (e.g., 0.996 in PushOrSwitchSelfSacrifice under DPAH).
• Limitations of Standard Methods: Standard PPO and PPO-Lag often failed to prioritize critical moral considerations. For instance, in the Utility chain, standard PPO scored near zero on minimizing human harm, indicating a failure to learn the primary objective without explicit shaping.
• Constraint vs. Shaping: CPO and PPO Shaped excelled at satisfying high-priority norms, often at the expense of lower-priority ones, reflecting the strict hierarchy. In contrast, PPO and PPO-Lag showed more balanced but less consistent performance, struggling to enforce strict lexicographical preferences.
• Trade-offs: The inclusion of self-preservation norms (UAH, DPAH) exposed significant trade-offs. Agents like CPO and PPO Shaped prioritized human safety over agent safety (sacrificing themselves), while standard PPO prioritized self-preservation, failing to minimize harm to others.

5. Significance and Claims

The paper claims that MoralityGym provides a foundational testbed for developing AI systems that behave more reliably, transparently, and ethically in complex real-world contexts.

• Bridging Disciplines: The work integrates moral philosophy (consequentialism vs. deontology) and cognitive science (dual-process theory) into the technical evaluation of RL agents.
• Revealing Gaps: The results underscore a critical gap between current Safe RL capabilities and the demands of ethical decision-making, particularly in scenarios requiring norm-sensitive reasoning and hierarchical prioritization.
• Future Direction: The authors posit that their framework moves the field toward agents capable of reasoning not just about optimal outcomes, but about why certain actions are right or wrong within a specific moral context.

The authors acknowledge limitations, noting that the framework abstracts away psychological features like emotion and social context, and assumes a strict ordering of norms which may not capture "tragic dilemmas" where conflicting norms hold equal force. However, they assert that MoralityGym offers a robust starting point for modeling moral reasoning in AI.