The Moralization Corpus: Frame-Based Annotation and Analysis of Moralizing Speech Acts across Diverse Text Genres

1. Problem Statement

While Natural Language Processing (NLP) has made strides in modeling moral values (e.g., predicting moral foundations from text) and analyzing moral biases in Large Language Models (LLMs), the specific pragmatic phenomenon of moralization remains underexplored.

• Definition: Moralization is defined not merely as the presence of moral vocabulary, but as a persuasive strategy where moral values are strategically invoked to justify demands, stances, or actions.
• The Gap: Existing computational approaches often rely on simplified categorical frameworks (e.g., one moral value per sentence) or focus on explicit moral terms. They fail to capture the complex, often implicit, interplay between moral values, demands (explicit or implicit), and discourse protagonists (agents, beneficiaries, culprits) within argumentative contexts.
• Challenge: Moralization is highly subjective, context-sensitive, and pragmatically complex, making it difficult for both human annotators and automated systems to detect and analyze consistently.

2. Methodology

A. Dataset Creation (The Moralization Corpus)

The authors constructed a novel, multi-genre dataset comprising 11,503 text instances in German, drawn from seven distinct genres:

• Political debates (parliamentary plenary minutes)
• News articles (court reports, commentaries/opinion pieces, letters to the editor)
• Online discussions (Wikipedia talk pages)
• Non-fiction books
• Interviews

Data Collection Pipeline:

1. Dictionary Creation (DIMI): A multilingual dictionary of morality-indicating words was developed, starting with a 130-word seed list and expanded via co-occurrence profiles to 3,000 entries.
2. Retrieval: Text snippets (5 sentences) containing dictionary entries were retrieved from large corpora (e.g., DEREKO) and web sources.
3. Filtering: A crucial step involved distinguishing Moralizations (persuasive arguments) from Neutral Value-Referring Instances (NVIs) (descriptive uses of moral terms).

B. Annotation Framework

The authors propose a frame-based annotation scheme that captures three constitutive layers:

1. Moral Values: Mapped to the six Moral Foundations Theory (MFT) pairs (Care/Harm, Fairness/Cheating, Loyalty/Betrayal, Authority/Subversion, Purity/Degradation, Liberty/Oppression). Multi-labeling is permitted.
2. Demands: Annotators identify explicit demands or reconstruct implicit demands into simple sentences (e.g., inferring "Women should be paid equally" from "Women earn less").
3. Protagonists: Entities are labeled by Group Type (Individuals, Generic, Institutions, Social Groups) and Discourse Role (Demander, Addressee, Beneficiary, Maleficiary).

Annotation Procedure:

• Multi-step Process: Included identification (binary classification of Moralization vs. NVI), pilot training, full annotation, review, re-annotation of borderline NVIs, and formal validation.
• Inter-annotator Agreement: Achieved a Cohen's Kappa of 0.71 for the initial binary classification. Disagreements were adjudicated by experts.
• Scale: The final test set contains 1,584 instances, with a high proportion of moralizations (44.7%) due to refinement processes, while the full dataset shows NVIs outnumbering moralizations.

C. Experimental Evaluation

The study evaluated the performance of five state-of-the-art LLMs (LLaMA-4, Cohere Command, Mistral, GPT-5, Claude-3.5) on the task of moralization detection and component extraction.

• Prompt Engineering: Seven prompt configurations were tested, varying in detail, reasoning requirements (Chain-of-Thought), few-shot examples (0-shot vs. 10-shot), and explanatory steps.
• Metrics:
  • Binary Detection: Accuracy, Precision, Recall, F1-score.
  • Component Extraction: NER-style strict and partial matching for values and protagonists; BLEU, ROUGE, and BERTScore for demand generation.
  • Agreement Analysis: Fleiss' Kappa and PABAK to compare Human-Human, Model-Model, and Human-Model agreement.

3. Key Contributions

1. The Moralization Corpus: A publicly released, multi-genre dataset with fine-grained, frame-based annotations linking moral values, demands, and protagonists.
2. Novel Annotation Framework: A holistic, pragmatic approach that moves beyond simple keyword detection to model the argumentative structure of moralizing speech.
3. Comprehensive LLM Evaluation: A systematic study comparing human and model performance across diverse prompting strategies, revealing the specific limitations of current LLMs in handling implicit pragmatic reasoning.
4. Resource Release: All data, annotation guidelines, and code are open-sourced to foster interdisciplinary research.

4. Key Results

A. Corpus Statistics & Linguistic Patterns

• Implicitness: 47% of demands are implicit, requiring contextual inference.
• Protagonists: Moralizations frequently invoke collective actors (Institutions, Social Groups) rather than individuals. Beneficiaries and Addressees are the most common roles.
• Moral Foundations: The Care/Harm and Fairness/Cheating foundations dominate, though distribution varies by genre (e.g., Fairness is high in Wikipedia discussions; Liberty in court reports).

B. Model Performance

• Prompting Impact: Detailed prompt instructions (defining the criteria clearly) had the most significant positive effect on performance.
• Ineffectiveness of Few-Shot/Explanation: Contrary to expectations, adding few-shot examples or forcing Chain-of-Thought explanations did not consistently improve performance. In some cases, they degraded precision.
• Model Rankings: Cohere and GPT-5 generally outperformed others. The ensemble model (majority vote) achieved the best overall F1 scores.
• Component Extraction:
  • Moral Values: Low F1 scores (strict match ≤ 0.20), indicating difficulty in boundary detection and context sensitivity.
  • Protagonists: Moderate performance (F1 0.20–0.28).
  • Demands: High semantic similarity (BERTScore ~73–78), but explicit demands were easier to extract than implicit ones.

C. Human-Model Agreement & Disagreement

• Subjectivity: Human experts agreed more with each other than with students or models. Experts detected more moralizations (38%) than students (23%), suggesting that operational definitions broaden detection compared to everyday intuition.
• Model Bias: Models tended to be more liberal (labeling 59% as moralizations), often relying on surface cues.
• Error Analysis:
  • False Positives (Models): Caused by neutral uses of moral vocabulary (e.g., historical descriptions), missing context, and borderline cases. Stronger models (LLaMA, Cohere) had more "borderline" FPs, suggesting higher sensitivity.
  • False Negatives (Models): Primarily caused by missed implicit demands. Models struggled to infer demands when no explicit call to action was present.

5. Significance and Implications

• Methodological Foundation: The paper establishes a rigorous framework for analyzing moral rhetoric that goes beyond sentiment analysis or simple value classification. It highlights that moralization is a pragmatic act requiring the linking of values to specific demands.
• NLP Challenges: The results demonstrate that current LLMs, despite their reasoning capabilities, struggle with the implicitness and subjectivity inherent in moral discourse. They rely heavily on surface lexical cues (moral words, modal verbs) rather than deep pragmatic inference.
• Future Directions: The study suggests that future models need:
  • Fine-tuning on specific moral frame tasks.
  • Better handling of context (longer windows) to resolve implicit demands.
  • Evaluation methods that account for the inherent subjectivity of moral reasoning (e.g., focusing on variance rather than just binary accuracy).
• Interdisciplinary Impact: The corpus provides a valuable resource for linguists, political scientists, and social scientists to study how moral framing shapes public discourse, policy debates, and social movements.

In conclusion, the paper argues that while LLMs can detect obvious moralizing, the nuanced, implicit, and context-dependent nature of moralization remains a significant challenge, requiring more sophisticated modeling of argumentative structures and pragmatic reasoning.