🤖 AI

Accounting for Context: Shaping Moral Credences for Value Alignment

This paper argues that aggregating moral evaluations under uncertainty must account for contextual factors, demonstrating that ignoring these factors leads to violations of the weak Pareto principle—a phenomenon identified as a variation of Simpson's paradox that reveals the limitations of standard aggregation mechanisms.

Original authors: Jazon Szabo, Sanjay Modgil

Published 2026-06-08

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jazon Szabo, Sanjay Modgil

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Moral Committee" Problem

Imagine you are building a robot (let's call it FROBO) to help people. You want FROBO to make moral decisions that match what humans think is right. But here's the problem: humans don't agree on what "right" is.

Some people believe in Utilitarianism (the "Greatest Good" rule): Do whatever saves the most lives or creates the most happiness, even if it means breaking a rule.
Others believe in Deontology (the "Rule Book" rule): Follow the rules (like "never harm a person"), even if the outcome isn't the best possible one.

To make FROBO smart, researchers usually ask a bunch of humans: "If you had to choose between Theory A and Theory B, which one do you trust more?" Let's say 60% of people trust Utilitarianism and 40% trust Deontology. The robot then uses these percentages (called moral credences) to decide what to do.

The Paper's Main Argument:
The authors say this standard method is flawed because it ignores context. It treats the robot's brain like a static calculator that uses the same 60/40 split for every single situation. In reality, the "trustworthiness" of a moral theory should change depending on the specific situation the robot is in.

The Running Example: The Burning Hospital

To prove their point, the authors use a story about a firefighter robot named FROBO trapped in a burning hospital during a military blockade. FROBO has to choose between two rooms:

The Left Room: Contains a huge stash of insulin. If saved, it could cure dozens of diabetics later, but only if the blockade stays in place.
The Right Room: Contains an unconscious patient who will die immediately if not helped.

The Dilemma:

The Utilitarian View: "Save the insulin! It could save 50 lives later." (But this requires complex guessing about the future).
The Deontological View: "Save the person in front of you! You have a duty to help the immediate victim." (This is a clear rule).

The Context Problem:
The authors argue that the robot's ability to calculate the "Utilitarian" outcome is limited. FROBO is a small robot with limited battery and processing power. It cannot perfectly predict if the blockade will end or if the insulin will actually be needed. Its "Utilitarian math" is shaky and prone to error because of these resource limits.

However, the "Deontological rule" (save the person right here) is simple and doesn't require complex math. It works perfectly even with limited resources.

The Solution:
Instead of using a fixed 60/40 split, the robot should adjust its trust based on the situation:

In the Left Room: Because the math is hard and risky, the robot should lower its trust in Utilitarianism and raise its trust in Deontology.
In the Right Room: The rule is clear and easy to follow, so the trust levels stay the same.

The Surprising Result: Simpson's Paradox

When the authors ran the math with these "context-aware" adjustments, something weird happened. They found that the robot sometimes chose the "worse" option according to every single theory, just because the context shifted the weights.

They call this a violation of the Weak Pareto Principle.

In plain English: If everyone (Utilitarians AND Deontologists) agrees that Option A is better than Option B, the group should pick Option A.
What happened here: The math showed that even though both theories preferred the Left Room (in their raw calculations), the robot ended up picking the Right Room.

The Analogy: The University Admissions Paradox
The authors compare this to a famous real-world puzzle called Simpson's Paradox.
Imagine a university where, overall, more men are accepted than women. It looks like the university is sexist against women.
But, when you look at each individual department, women actually have a higher acceptance rate than men!
How? Because women applied mostly to the "super hard" departments (like Physics), while men applied mostly to the "easier" departments (like English). The "hardness" of the department skewed the overall numbers.

How it applies to the robot:
The robot's decision was skewed by the "hardness" of the situation.

For the Left Room, the "Utilitarian" theory was weak (because the robot couldn't calculate well), so its vote didn't count much.
For the Right Room, the "Deontological" theory was strong (because the rule was clear), so its vote counted a lot.
Even if both theories wanted the Left Room, the context made the Right Room look better in the final tally.

The authors argue this isn't a bug; it's a feature. It shows that ignoring context leads to bad decisions. If you ignore the fact that the robot is "bad at math" in this specific room, you get the wrong answer.

Key Takeaways

Context Matters: You can't just ask people "Which moral theory do you like?" and apply that answer to every situation. You have to ask, "Which theory works best right now, given the robot's limits and the specific danger?"
Trust is Dynamic: A robot should trust a "Rule-Based" approach when it's in a chaotic, high-pressure situation where it can't do complex math. It should trust a "Consequence-Based" approach when it has plenty of time and data to predict the future.
The "Thick" vs. "Thin" View: Current AI research treats moral theories like simple math equations (Thin). The authors say we need to treat them like complex tools (Thick). A hammer is great for nails, but terrible for screws. You shouldn't use a hammer just because 60% of people said they like hammers; you should use the tool that fits the job.

What the Paper Does NOT Say

It does not say we should stop using simulations to train robots.
It does not claim that robots should have feelings or consciousness.
It does not provide a medical or clinical solution for human doctors.
It does not say that one moral theory is "better" than the other in a general sense. It only says that the appropriateness of a theory changes based on the situation.

In short, the paper is a warning: Don't let a robot follow a rigid moral recipe book when the kitchen is on fire. It needs to know when to follow the rules and when to do the math, depending on how much time and brainpower it has.

Technical Summary: Accounting for Context: Shaping Moral Credences for Value Alignment

Problem Statement
Value alignment aims to ensure that intelligent agents behave in accordance with human ethical values. However, human societies exhibit moral pluralism, where individuals and groups evaluate actions through conflicting ethical theories (e.g., deontology vs. utilitarianism). Current approaches to pluralistic alignment often rely on a pipeline that elicits preferences, derives utility functions, and aggregates them using moral uncertainty frameworks. These frameworks typically assign static "moral credences" (weights) to different ethical theories based on population proportions or individual confidence.

The central problem identified in this paper is that standard aggregation mechanisms under moral uncertainty often ignore contextual factors. Specifically, the applicability and reliability of an ethical theory can vary significantly depending on the specific situation an agent faces. For instance, consequentialist theories (like utilitarianism) require accurate prediction of outcomes, which may be computationally infeasible or epistemically uncertain in real-world settings with resource bounds. Conversely, deontological rules may remain robust under the same constraints. Treating moral credences as static, context-independent values can lead to aggregation errors where the agent's decision-making process fails to account for the differential reliability of theories in specific contexts.

Methodology
The authors formalize a framework for context-adjusted moral credences within the decision-theoretic model of moral uncertainty (building on MacAskill, Bykvist, and Ord 2020).

Formal Definitions:
- Ethical Framework: Defined as a pair $\langle T, c \rangle$ , where $T$ is a set of ethical theories and $c$ is a credence function mapping theories to weights summing to 1.
- Credence Profile: To accommodate context, the authors define a credence profile $C = \langle c_a, c_b, \dots \rangle$ , where each $c_x$ is a credence function specific to a particular action $x$ . This allows weights to vary per action based on context.
- Contextual Features: Defined as functions $g: A \times T \to (0, 1]$ that modulate the reliability of a theory $t$ regarding an action $a$ . A value closer to 1 implies high reliability/appropriateness, while lower values indicate reduced reliability due to constraints (e.g., resource bounds, uncertainty, novelty).
- Adjustment Functions: The paper introduces two mechanisms to update initial credences $c$ $c$ into context-adjusted credences $c_x$ $c_{x}$ based on a context $con = \langle g_1, \dots, g_m \rangle$ $co n = ⟨ g_{1}, \dots, g_{m} ⟩$ :
  - prod: Multiplies the initial credence by the product of all contextual features, followed by normalization. This is "risk-neutral," allowing high values in one feature to compensate for low values in another.
  - mini: Takes the minimum of the initial credence and all contextual features, followed by normalization. This is "risk-averse," ensuring the adjusted credence does not exceed the reliability of the weakest contextual factor.
Metanormative Theory:
The authors utilize Maximizing Expected Choiceworthiness (MEC) as the primary aggregation function. MEC orders actions based on the weighted arithmetic mean (WAM) of their choiceworthiness across theories, using the context-adjusted credences from the credence profile.
Running Example (FROBO):
The paper employs a scenario involving a firefighter robot (FROBO) in a burning hospital under military blockade. FROBO must choose between saving a single unconscious patient (Right Room) or preserving insulin for many potential future patients (Left Room).
- Context: Resource bounds limit FROBO's ability to calculate long-term consequences accurately.
- Effect: The contextual feature for resource bounds ($rb$) significantly lowers the credence for utilitarianism regarding the Left Room (where complex prediction is needed) but leaves it high for the Right Room (where the consequence is immediate and certain).

Key Results
The paper demonstrates that incorporating context-adjusted credences leads to a violation of the Weak Pareto Principle, a standard property in social choice theory stating that if all theories prefer action $b$ over $a$ , the aggregate should also prefer $b$ over $a$ .

Violation of Weak Pareto: The authors prove that for a broad class of metanormative theories (specifically those that are inter-theoretically responsive) and adjustment functions (specifically those that are context-surjective, including prod and mini), the Weak Pareto principle is violated.
- Mechanism: It is possible to construct a scenario where every individual ethical theory prefers action $a$ over $b$ ( $t(a) < t(b)$ for all $t$ ), yet the context-adjusted aggregation prefers $b$ over $a$ .
- Example: In the FROBO scenario, even if utilitarianism and deontology both nominally prefer the Left Room (insulin) over the Right Room (patient) in a vacuum, the resource constraints might lower the utilitarian credence for the Left Room so drastically that the deontological preference for the Right Room dominates the aggregate WAM, causing the agent to choose the Right Room.
Simpson's Paradox Interpretation: The authors interpret this violation not as a flaw, but as a manifestation of Simpson's Paradox. Just as aggregated data can reverse trends present in subgroups (e.g., admission rates by gender vs. department), aggregating moral evaluations across different contexts (actions) can reverse the preferences of individual theories. The violation occurs because the "weight" of a theory changes dynamically based on the action's context, allowing a theory with a lower initial credence to dominate in a specific context where it is more reliable.

Significance and Claims
The paper argues that the violation of the Weak Pareto principle in this context is not undesirable but rather an inevitable and necessary consequence of accounting for contextual factors.

Limitations of Context-Insensitive Aggregation: The results highlight that standard aggregation mechanisms, which ignore how context impacts the reliability of ethical theories, are insufficient for real-world value alignment. Ignoring context can lead to "fanatical" or unreliable outcomes where agents follow theories that are computationally or epistemically unsuited for the specific situation.
New Research Direction: The work motivates a shift in moral uncertainty research toward contextually-weighted social choice. It suggests that criteria for evaluating aggregation methods (like Weak Pareto) must be reassessed when context is a variable.
Thickening Ethical Representations: The paper posits that the need for context adjustment arises because current models represent ethical theories "thinly" (as mere utility functions). By adjusting credences based on contextual factors (like computational feasibility), the authors are effectively "thickening" these evaluations to reflect the algorithmic and epistemic realities of the theories.
Future Integration: The authors suggest that this formalism could be integrated with argumentation-based dialogues. Instead of static preference elicitation, agents could dynamically adjust moral credences "on the fly" through structured dialogues that explore contextual nuances, thereby supporting more robust value alignment in narrow and superintelligent systems alike.

The paper concludes that while the violation of standard social choice axioms appears counterintuitive, it reveals the conceptual limitations of ignoring context and provides a formal basis for more realistic, context-sensitive moral decision-making in AI.