Technical Summary: Bounded Morality

Problem Statement

Traditional approaches to formalizing morality in artificial systems typically encode established ethical theories (e.g., utilitarianism, deontology) as static rules or value functions. This strategy treats moral disagreement as a problem of selecting the correct theory, often failing to account for how moral reasoning is actually carried out, how it scales with cognitive capacity, or why distinct theories persist across cultures. While bounded rationality research has shown that human judgment reflects adaptive trade-offs under computational constraints, the "demand side" of moral reasoning remains underspecified: specifically, which features of moral situations impose the informational and computational burdens that necessitate these trade-offs.

This paper addresses the gap in characterizing the demand structure of moral problems. It asks how moral situations vary in ways that systematically increase or decrease the computational burden of moral reasoning for finite agents.

Methodology

The authors propose Bounded Morality, a formal framework that models moral reasoning as constrained inference under situational demands. The methodology proceeds in three stages:

1. Defining the Demand Space

The framework identifies two orthogonal dimensions that define the computational demands of moral situations:

• Moral Breadth: The scope of moral representation. This captures which entities, groups, timescales, or abstractions are treated as morally relevant (e.g., including out-groups, non-human animals, or future generations).
• Moral Depth: The inferential complexity. This captures how richly, recursively, and coherently represented entities and their interactions are reasoned about (e.g., counterfactual reasoning, multi-step deliberation, or integrating multiple perspectives).

Empirical evidence from developmental psychology, neuroscience, and social choice theory is used to demonstrate that these dimensions are dissociable but jointly constrained by finite cognitive resources.

2. Formal Modeling

The authors formalize moral reasoning as a controlled dynamical system on a graph:

• Moral Interaction Graph ($G^*$): A graph where nodes represent morally relevant entities and edges represent direct influence relationships.
• Ground-Truth Value: Defined as the discounted sum of welfare over infinite trajectories, where welfare is a graph-structured function of node and edge potentials.
• Bounded Approximation: Finite agents cannot compute the ground truth. Instead, they employ:
  • Abstraction ($G$): A coarse-grained projection of $G^*$ via an aggregation map $\pi_V$, reducing the number of distinct entities and interactions (Breadth).
  • Truncated Horizon ($H$): A rollout depth specifying how many steps of the dynamics are explicitly evaluated (Depth).
• Cost Model: A canonical cost model is introduced where total computational cost is $Cost(b, H) = \alpha b + \beta b H^p$. This establishes an inverse scaling law: under a fixed budget $B$, increasing representational breadth ($b$) strictly reduces the achievable depth ($H$), and vice versa.

3. Strategy and Regret

• Moral Strategy: Defined as an allocation rule $\rho$ that maps world states and resource budgets to specific pairs of $(G, H)$.
• Moral Regret: The difference between the ground-truth moral value of the optimal intervention and the value achieved by the bounded strategy.
• Moral Progress: Defined not as converging on a unique moral truth, but as reducing expected regret within a fixed resource budget (better allocation) or shifting the feasible frontier outward via technological or institutional changes.

Key Contributions

1. The Breadth–Depth Trade-off

The paper establishes a formal Pareto frontier for moral computation. It demonstrates that for finite agents, expanding the scope of moral concern (breadth) necessarily constrains the depth of inference, and deepening inference requires narrowing the scope. Ethical theories are reinterpreted not as competing accounts of moral truth, but as locally efficient strategies adapted to different regimes of resource demands along this frontier.

2. Reinterpretation of Ethical Theories

The framework maps canonical ethical theories to specific regions of the breadth–depth space:

• Act Utilitarianism: High breadth, moderate depth. Prioritizes representing many stakeholders with simple additive aggregation.
• Deontology: Moderate breadth, moderate depth. Relies on hard constraints to prune the search space, often requiring limited scope to maintain feasibility.
• Contractualism: Low breadth, high depth. Focuses on a few stakeholders but allocates significant resources to deep deliberation and worst-case objection analysis.
• Virtue Ethics: Minimal breadth, zero depth. Shifts computation offline into learned dispositions (amortized inference), eliminating online rollout.
• Care Ethics: Local breadth, moderate depth. Focuses on dense relational structures within a local neighborhood rather than global enumeration.

3. Formalizing Moral Disagreement and Progress

• Disagreement: The framework suggests that persistent moral disagreement can arise even among agents sharing the same underlying values. Disagreement may reflect different allocations of limited resources (different points on the Pareto frontier) rather than fundamental value conflicts.
• Progress: Moral progress is redefined as either (i) reducing regret by improving resource allocation within fixed constraints, or (ii) expanding the feasible frontier through cultural or technological advancements that increase available capacity.

4. Implications for AI Alignment

The paper argues that AI alignment should not focus solely on imitating human judgments, which are themselves bounded and potentially suboptimal. Instead, alignment should focus on designing systems where moral reasoning capacity scales with representational and computational resources. The goal is to replicate the scaling relationship between resources and moral capacity, rather than copying the specific outputs of bounded human agents.

Results and Claims

The paper does not present new experimental data or empirical validation of specific AI systems. Instead, its "results" are theoretical derivations:

• Inverse Scaling Law: A mathematical derivation showing that under a fixed budget, depth scales inversely with breadth ($H \leq C b^{-1/p}$).
• Regret Minimization: The definition of moral progress as the minimization of expected regret relative to an unbounded ideal, given specific resource constraints.
• Strategy Classification: A conceptual mapping of ethical theories to specific structural commitments regarding breadth, depth, and aggregation.

Significance

The paper claims to offer a unifying descriptive account of moral reasoning that bridges ethics, psychology, and AI. Its significance lies in:

1. Reframing Moral Failure: It shifts the explanation for moral shortcomings (e.g., rigidity, neglect of distant stakeholders) from defective values or irrationality to rational allocations of limited computational resources.
2. Non-Relativist Disagreement: It provides a structural explanation for moral disagreement that does not require moral pluralism; agents can disagree while sharing values simply by operating at different feasible points on the breadth–depth frontier.
3. Computational Lens for Ethics: It offers a minimal computational lens to integrate insights from ethics and psychology, treating moral theories as strategies for managing the trade-offs inherent in finite agents.
4. Diagnostic Tool for AI: It provides a technical vocabulary to analyze why morally competent behavior breaks down in artificial systems, suggesting that alignment failures may stem from mismatches between moral computation and context rather than incorrect objectives.

The authors explicitly state that the framework does not argue for any particular moral values or ethical theory. Instead, it studies the operational constraints any finite moral agent—human or artificial—must face when situational demands exceed available time, information, and computational capacity.