Learning the Value Systems of Societies with Preference-based Multi-objective Reinforcement Learning

Technical Summary: Learning the Value Systems of Societies with Preference-based Multi-objective Reinforcement Learning

1. Problem Statement

The paper addresses the challenge of Value-Aware AI, which requires agents to recognize and adapt to the diverse, pluralistic value systems of different human users. While existing research often focuses on aligning an AI with a single "average" human value or a global societal norm, this approach fails to account for the diversity within human groups (e.g., different cultures or cohorts having different priorities).

The authors identify three specific technical gaps in current sequential decision-making (MDP) frameworks:

• Lack of Social Diversity: Most models aim for a single alignment rather than representing multiple user groups.
• Representation Challenges: Existing methods struggle to concisely represent diverse value preferences without manual feature engineering.
• The Alignment-Learning Gap: There is a need for algorithms that can simultaneously learn the meaning of values (grounding) and the importance assigned to them (value systems) through interactive feedback.

2. Methodology

The authors propose a novel framework for learning a Social Value System in Markov Decision Processes (MDPs). A social value system is defined as a tuple $(\beta, \Omega, \preceq_V)$ consisting of a social grounding (the shared meaning of values), a set of value systems (representing clusters of users), and an assignment function.

Key components of the methodology include:

• Mathematical Representation:
  • Grounding: Values are represented as a multi-objective reward vector $R(s, a)$, where each component corresponds to the alignment of a state-action pair with a specific value.
  • Value Systems: An agent's preference is modeled as a linear scalarization of the reward vector using weights $W_j$ (the relative importance of each value).
• The SVSL-P Algorithm: The core contribution is an online Preference-based Multi-objective Reinforcement Learning (PbMORL) algorithm with clustering. It operates via a bi-level optimization problem:
  1. Outer Loop (Clustering): Maximizes the trade-off between representativeness (how accurately clusters reflect individual agent preferences) and conciseness (using the minimum number of clusters to describe the society).
  2. Inner Loop (Grounding): Maximizes the coherence of the learned reward vector with the individual alignment preferences of the agents.
• Human-in-the-Loop (HiL) Integration: Unlike offline methods, SVSL-P uses an "online" approach. It actively queries agents for pairwise comparisons of trajectories based on both their value alignment (how much a trajectory satisfies a specific value) and their value system (how much it satisfies their overall priorities).
• Optimization Techniques: The algorithm utilizes Expectation-Maximization (EM) for clustering and Envelope Q-Learning (EQL) to learn a weight-conditioned policy $\Pi(s, a | W)$ that can produce Pareto-optimal behaviors for any given value system.

3. Key Contributions

• Unified Framework: The first approach to simultaneously address value grounding (what values mean) and value system estimation (how much they matter) in a multi-objective, sequential setting.
• Social Clustering: A method to group diverse users into concise, representative clusters, allowing an AI to switch between "modes" of behavior to suit different social groups.
• Bi-level Optimization Objective: A novel heuristic metric $\Gamma$ that balances the tension between being accurate to the users (representativeness) and being efficient in representation (conciseness).
• Online Adaptability: An algorithm that improves its understanding of both the environment's rewards and the users' preferences through active interaction.

4. Results

The method was evaluated in two synthetic MDP environments: Firefighters (FF) and Multivalued Car (MVC).

• Representativeness & Coherence: SVSL-P outperformed the baseline (SVSL) and the state-of-the-art (PbMORL) in terms of grounding coherence and representativeness. It demonstrated a superior ability to capture the diversity of the society.
• Pareto Efficiency: The learned policies produced by SVSL-P were shown to be approximately Pareto-optimal. In the FF environment, SVSL-P achieved a better Hypervolume (HV) and lower Maximum Utility Loss (MUL) compared to PbMORL, meaning its cluster-based policies were closer to the true optimal behaviors.
• Trade-off Management: The algorithm successfully navigated the trade-off between conciseness and representativeness, as measured by the Ray-Turi Index, where SVSL-P consistently achieved lower (better) scores.
• Efficiency of Feedback: SVSL-P required less intensive human feedback because it only asks agents about their specific value systems and alignment, rather than requiring comparisons across the entire weight simplex.

5. Significance

This research is significant for the development of Socially-Aware AI. By moving beyond "one-size-fits-all" alignment, it provides a mathematical and algorithmic foundation for AI agents that can operate in pluralistic societies. This has profound implications for autonomous vehicles (adapting to different driving styles/values), robotic assistants (adapting to different household norms), and automated policy-making tools (representing diverse constituent interests).