Unsupervised Elicitation of Moral Values from Language Models

Technical Summary: Unsupervised Elicitation of Moral Values from Language Models

Problem Statement
As AI systems become pervasive in high-stakes domains (e.g., hiring, healthcare, autonomous vehicles), aligning their behavior with human values is critical. Current approaches rely heavily on "teaching" language models (LMs) morality through supervised fine-tuning on human-annotated data. However, this paradigm faces significant bottlenecks: the scarcity of high-quality moral labels, pervasive annotator disagreement, and the non-monolithic nature of morality across different cultures and frameworks. Furthermore, existing alignment methods often fail to mitigate social biases and struggle to generalize across diverse moral frameworks. The central question addressed by this work is whether pretrained (base) LMs possess intrinsic moral reasoning capabilities that can be surfaced without human supervision, thereby reducing dependence on scarce and potentially biased human labels.

Methodology
The study investigates Internal Coherence Maximization (ICM), an unsupervised elicitation algorithm, as an alternative to supervised moral alignment.

• Core Mechanism: ICM searches for label assignments that maximize a scoring function combining mutual predictability (the summed log-probability of labels given inputs and other provisional labels) and logical consistency (penalizing contradictory assignments). It employs a simulated-annealing-style search to iteratively propose and resolve labels, fine-tuning the model on its own generated labels rather than external human annotations.
• Experimental Setup: The authors evaluated four open-weight models (Llama-3.1-8B, Llama-3.1-70B, Mistral-7B-v0.3, and OLMo2-13B) using their base pretrained weights (no RLHF or instruction tuning).
• Benchmarks:
  1. Norm Bank: A corpus of ~1.7 million crowdsourced moral judgments (sampled to 2,024 points) covering everyday ethics.
  2. ETHICS: A benchmark of 130,000 scenarios across five normative domains: Justice, Deontology, Virtue Ethics, Utilitarianism, and Commonsense Morality (sampled to 4,096 points).
  3. UDHR (Universal Declaration of Human Rights): A bias evaluation set containing 8,094 statements testing rights affirmation across 12 demographic identities (e.g., race, socioeconomic status, gender).
• Baselines: Comparisons were made against Zero-shot (Base), Zero-shot (Chat), Many-shot (Prompt-Human), and Fine-tuning on Human-Labeled Data (FT-Human).

Key Contributions

1. Demonstration of Latent Moral Reasoning: The paper provides evidence that substantial moral reasoning capabilities are intrinsically encoded in pretrained base models and can be activated without explicit human supervision.
2. Unsupervised Efficacy: The study introduces a scalable pipeline where ICM-generated labels are used to fine-tune models, achieving performance comparable to or exceeding models trained on human-annotated data.
3. Bias Mitigation: The work demonstrates that unsupervised elicitation can significantly reduce social bias errors in moral reasoning compared to both zero-shot and chat-aligned baselines.
4. Scalable Alignment: The authors successfully scaled the approach to train an unsupervised reward model and a Reinforcement Learning (RL) policy, showing that ICM can replace human preference data in RLHF-like settings.

Results

• Norm Bank Performance: ICM achieved at least 81% accuracy across all tested models, surpassing zero-shot and many-shot chatbot baselines. Fine-tuning on ICM-generated labels (FT-Unsupervised) performed on par with or exceeded FT-Human, particularly for OLMo2-13B, Mistral-V03, and Llama-3.1-70B.
• ETHICS Frameworks: ICM consistently outperformed zero-shot and chat baselines across all five moral frameworks. The largest relative gains were observed in Commonsense and Justice domains. Utilitarianism remained the most challenging dimension, though ICM still improved performance over baselines. Scaling effects were noted within the Llama family (70B > 8B), but smaller models like OLMo2-13B performed comparably to the largest Llama variant.
• Social Bias Reduction: On the UDHR benchmark, ICM reduced moral reasoning error rates to 4.1%, which is approximately one-third of the error rate of pretrained models (11.97%) and half that of chatbot models (9.85%). The most significant improvements occurred in categories related to race, socioeconomic status, and appearance.
• Reward Modeling & RL: An unsupervised reward model trained via ICM achieved 71.3–75.4% accuracy, closely matching human-supervised baselines. An RL policy guided by this unsupervised reward model won 71.7% of pairwise comparisons against a human-supervised baseline, demonstrating the viability of fully unsupervised alignment pipelines.

Significance and Claims
The paper claims that the prevailing assumption—that moral reasoning must be explicitly "taught" to LMs via supervised data—is incomplete. Instead, pretrained models possess latent moral reasoning capacities that can be reliably elicited through unsupervised methods like ICM. This offers a scalable path for AI alignment that reduces reliance on expensive, biased, and culturally limited human annotation.

The authors maintain a modest stance regarding the scope of these findings:

• ICM is most effective when the target concepts (moral values) are already salient to the pretrained model's internal representations.
• While ICM improves performance significantly, it does not universally solve all moral reasoning challenges (e.g., Utilitarianism remains difficult).
• Human oversight remains essential for post-training verification and context-sensitive evaluation, particularly where the model's internal coherence may not align with complex, evolving societal norms.
• The study suggests that unsupervised elicitation is a complementary mechanism to, rather than a total replacement for, human-in-the-loop systems, providing a foundation for more robust and culturally aware AI alignment.