The Moral Foundations Reddit Corpus

1. Problem Statement

Moral framing and sentiment significantly influence online and offline behaviors, including political engagement, polarization, and pro-social actions. While Natural Language Processing (NLP) has made strides in detecting moral sentiment, existing resources are limited:

• Platform Bias: The primary existing resource, the Moral Foundations Twitter Corpus (MFTC), is restricted to Twitter. Different platforms (e.g., Reddit vs. Twitter) possess distinct linguistic structures, character limits, anonymity levels, and community norms, which affect how moral rhetoric is expressed.
• Theoretical Obsolescence: The MFTC relies on the original Moral Foundations Theory (MFT) which treated "Fairness" as a single foundation. Recent theoretical revisions (Atari et al., 2023) split Fairness into two distinct foundations: Equality (equal outcomes) and Proportionality (merit-based rewards). Existing datasets lack annotations aligned with this updated taxonomy.
• LLM Limitations: There is a need to evaluate whether Large Language Models (LLMs) can effectively detect subjective moral sentiments without extensive human-annotated training data, or if fine-tuned encoder models remain superior.

2. Methodology

A. Corpus Construction (MFRC)

The authors constructed a new dataset comprising 16,123 English Reddit comments curated from 12 distinct subreddits. The selection criteria ensured diversity in topics and moral contexts:

• US Politics: r/politics, r/conservative, r/antiwork.
• French Politics: r/europe, r/worldnews, r/neoliberal, r/geopolitics, r/Conservative (filtered for keywords related to the 2017 French presidential election).
• Everyday Moral Life: r/nostalgia, r/AmItheAsshole, r/confession, r/relationshipadvice.

Sampling Strategy:
Due to the rarity of moral rhetoric in general discourse, a semi-supervised up-sampling method was employed. The authors used word embeddings and a "moral loading score" (based on Distributed Dictionary Representations) to identify highly moral comments. The final dataset was balanced to include 50% high moral loading (>95th percentile) and 50% lower moral loading comments to ensure diversity.

B. Annotation Protocol

• Annotators: Five trained undergraduate research assistants (selected from a pool of 27) with expert-level familiarity in MFT.
• Taxonomy: Annotations were based on the Revised Moral Foundations Theory (Atari et al., 2023), covering 8 categories:
  1. Care/Harm
  2. Equality/Inequality (New)
  3. Proportionality/Disproportionality (New)
  4. Loyalty/Betrayal
  5. Authority/Subversion
  6. Purity/Degradation
  7. Thin Morality: Moral judgments not clearly tied to a specific foundation (e.g., generic "good/bad").
  8. Implicit/Explicit: Whether the sentiment was stated directly or inferred.
• Process: Each comment was annotated by at least three annotators. Final labels were determined by majority vote (≥50% agreement). Annotators also provided confidence levels and demographic/psychological metadata (political ideology, income, etc.) to facilitate bias analysis.

C. Experimental Setup

The authors evaluated various models to establish baselines for moral sentiment classification (both single-label and multi-label):

1. Large Language Models (LLMs):
   • Models: Meta-Llama-3-8B and Ministral-8B.
   • Settings: Zero-shot prompting, Few-shot prompting (5-shot), and Parameter-Efficient Fine-Tuning (PEFT/LoRA).
2. Encoder-Only Models:
   • Model: BERT (Bidirectional Encoder Representations from Transformers).
   • Settings: Full Fine-Tuning (FFT).
3. Training Details: Weighted loss functions were used to address class imbalance (fewer moral posts than non-moral). Evaluation used 5-fold cross-validation with F1, Precision, and Recall metrics.

3. Key Contributions

1. The MFRC Dataset: A large-scale, multi-label corpus of 16k+ Reddit comments annotated under the revised MFT framework, specifically distinguishing between Equality and Proportionality.
2. Annotator Metadata: Release of detailed demographic and psychological data (via request) for the annotators, enabling future research into how annotator bias impacts subjective moral labeling.
3. Comprehensive Benchmarking: A rigorous evaluation of state-of-the-art LLMs against fine-tuned encoder models on a subjective moral task, establishing MFRC as a benchmark for AI alignment and moral reasoning evaluation.
4. Cross-Corpus Analysis: Preliminary investigation into the transferability of models between the Twitter-based MFTC and the Reddit-based MFRC.

4. Results

Model Performance

• Fine-Tuned Encoders vs. LLMs: Fine-tuned BERT models significantly outperformed LLMs (Llama and Ministral) across all moral categories in terms of F1 scores.
  • Example: BERT (single-label) achieved an F1 of 0.62 for Care, whereas Llama (zero-shot) achieved 0.40.
• Prompting Limitations: Zero-shot and few-shot prompting yielded poor results, particularly for minority classes (e.g., Purity, Loyalty).
• PEFT vs. Full Fine-Tuning: While PEFT (LoRA) improved Llama performance over zero-shot, it still lagged behind fully fine-tuned BERT.
• Single vs. Multi-Label: Interestingly, multi-label BERT performed slightly worse than single-label BERT, whereas multi-label Llama performed comparably to its single-label counterpart.

Cross-Corpus Transfer

• Models trained on MFTC (Twitter) and tested on MFRC (Reddit) showed lower performance than models trained and tested within the same corpus.
• This suggests that moral language varies significantly by platform (e.g., character limits, anonymity, community norms), limiting the generalizability of models trained on a single platform.

Inter-Annotator Agreement

• Raw Fleiss' Kappa scores were low due to the low prevalence of moral rhetoric (class imbalance).
• However, PABAK (Prevalence-and Bias-Adjusted Kappa), which corrects for these factors, showed medium to high agreement, validating the reliability of the annotations despite the subjective nature of the task.

5. Significance and Implications

• AI Alignment: The results underscore that for highly subjective tasks like moral sentiment analysis, human-annotated corpora remain essential. LLMs, even with PEFT, struggle to match the nuance of fine-tuned encoders trained on high-quality human data.
• Theoretical Advancement: By splitting Fairness into Equality and Proportionality, the MFRC allows researchers to investigate nuanced moral conflicts (e.g., income inequality vs. meritocracy) that were previously conflated.
• Platform Dynamics: The study highlights that moral rhetoric is not universal; it is shaped by the specific linguistic and social structures of the platform (Reddit vs. Twitter). This is crucial for understanding polarization and group dynamics in different digital ecosystems.
• Resource for NLP: MFRC serves as a critical infrastructure for developing, evaluating, and auditing models for moral reasoning, hate speech detection, and political discourse analysis.

In conclusion, the MFRC addresses critical gaps in moral NLP by providing a platform-diverse, theoretically updated, and richly annotated resource, while demonstrating that current generative AI models still require significant human supervision to accurately interpret complex moral sentiments.