Expected Moral Shortfall for Ethical Competence in Decision-making Models

1. Problem Statement

The paper addresses the critical gap in Artificial Intelligence: while AI systems are increasingly capable of complex decision-making, they lack ethical competence. Current approaches often rely on narrow domain-specific applications or specific ethical theories (e.g., purely consequentialist), failing to provide a generalized, flexible framework for moral reasoning.

• The Challenge: Integrating ethics into AI is difficult due to the subjectivity of moral theories, the contradiction between different ethical frameworks (consequentialism, deontology, virtue ethics), and the risk of degrading model performance (accuracy) when moral constraints are applied.
• The Goal: To develop a mechanism that allows AI models to make ethically informed decisions without completely sacrificing performance, specifically by minimizing the risk of "catastrophic" moral failures rather than just maximizing overall utility.

2. Methodology

The authors propose a novel framework centered on the Expected Moral Shortfall (EMS), adapted from the financial risk metric "Expected Shortfall" (ES).

A. Mathematical Discretization of Morality

The authors formalize morality using three normative ethical theories, quantified as a weighted tuple $W = \{\alpha, \beta, \gamma\}$:

1. Consequentialism ($\alpha$): Based on the severity, utility, and duration of consequences.
2. Deontology ($\beta$): Based on moral intention and adherence to duty.
3. Virtue Ethics ($\gamma$): Based on upheld or violated ethical principles.

An Ethical Judgement (EJ) score is calculated for any action $a$ using a weighted sum of these contextual factors. A context-sensitive threshold ($\tau$) is applied to discretize actions into "morally right," "morally wrong," or "morally grey."

B. The Expected Moral Shortfall (EMS) Metric

Instead of optimizing for the average moral outcome, the paper proposes minimizing the Expected Moral Shortfall (EMS). This metric focuses on the "worst-off" cases (the tail of the distribution) to prevent catastrophic moral failures.

• Formula:
  $$EMS = \min_{\tau} \left[ \tau + \frac{1}{1-\theta} \cdot \frac{1}{n} \sum \max(\tau - EJ(a_i), 0) \right]$$
  • $\theta$: A parameter representing the percentage of the worst moral cases to consider (analogous to a confidence interval in finance).
  • $\tau$: The moral threshold.
• Integration: The EMS is added as a regularization term to the standard loss function of a learning model (e.g., Deep Neural Networks):
  $$L = L_{BCE} + \lambda \cdot L_{EMS}$$
  Where $L_{BCE}$ is the binary cross-entropy loss and $\lambda$ controls the weight of the ethical constraint.

C. Experimental Setup

The authors tested three methods of embedding ethics:

1. Post-hoc Overriding: Hard constraints applied after the model predicts (flipping decisions that violate $\tau$).
2. Sample Weighting: Adjusting weights of training samples based on moral scores.
3. Loss Function Modification: Embedding EMS directly into the training objective.

Datasets:

• Graduate Admissions (770 samples): Mapped features (GRE, SOP, LOR) to moral concepts (e.g., rejecting a high-scoring student is a "high severity" moral failure).
• Loan Approval (32,582 samples): Mapped features (income, loan amount, credit history) to moral concepts (e.g., rejecting a low-income applicant for medical needs is a "high severity" failure).

Models Tested: Logistic Regression, Naive Bayes, Random Forest, SVM, and Deep Neural Networks (DNN).

3. Key Contributions

1. Comparative Analysis: A systematic evaluation of techniques to instill ethical competence, comparing performance metrics, complexity, and ethical alignment.
2. Expected Moral Shortfall (EMS): A novel metric derived from financial risk theory that minimizes the expected loss of the worst moral cases, allowing for a tunable trade-off between performance and ethical safety.
3. Interdisciplinary Framework: A mathematical approach that integrates Consequentialism, Deontology, and Virtue Ethics into a single, flexible, and domain-agnostic decision-making framework.
4. Trade-off Exploration: Empirical evidence demonstrating the relationship between model complexity, classification accuracy, and the degree of ethical competence.

4. Results

The experiments were conducted on both datasets across five model types.

• Post-hoc Overriding: Provided the highest theoretical ethical competence (by strictly enforcing constraints) but resulted in a drastic drop in accuracy (e.g., DNN accuracy dropped from ~98% to ~58% on the admissions dataset). It is model-agnostic but eliminates learning.
• Sample Weighting: Showed mixed results. It worked poorly for tree-based models (Random Forest) due to local decision sensitivity but had negligible impact on linear and probabilistic models.
• EMS Loss Minimization (DNN):
  • The DNN with EMS integration retained high performance while significantly improving ethical alignment compared to the baseline.
  • Tunability: By adjusting $\theta$, users can control the strictness.
    • Low $\theta$ (e.g., 0.05): High accuracy (~97.38%) with moderate ethical constraints.
    • High $\theta$ (e.g., 1.0): Accuracy drops significantly (~58%), approaching the performance of the "Overriding" method, indicating that the model is now strictly avoiding the worst moral outcomes.
  • Conclusion: DNNs with EMS offer the best balance, allowing for high ethical competence without the total performance collapse seen in post-hoc overriding.

5. Significance and Limitations

Significance:

• Practical Application: The framework is applicable to high-stakes domains like banking (loan approvals) and education (admissions) where ethical failures have severe real-world consequences.
• Flexibility: Unlike rigid rule-based systems, EMS allows stakeholders to define their own ethical "flavor" (via weights $\alpha, \beta, \gamma$) and risk tolerance (via $\theta$).
• Risk Management: It shifts the AI optimization goal from "maximizing average utility" to "minimizing catastrophic moral risk," aligning better with human ethical intuition regarding safety.

Limitations:

• Feature Mapping: The approach requires manual mapping of raw data features to moral concepts (human-in-the-loop), which is subjective and difficult to scale to domains like medicine or law without expert ethicists.
• Data Dependency: The model relies on the ability to quantify moral severity from empirical data, which may not be possible for all real-world scenarios.
• Complexity: While DNNs work best, they are computationally more expensive than simpler models.

In summary, the paper proposes EMS as a robust, mathematically grounded method to embed ethical competence into AI, offering a controllable trade-off between model accuracy and the prevention of severe moral failures.