Categorical Emotions or Appraisals - Which Emotion Model Explains Argument Convincingness Better?

This paper demonstrates that appraisal theories, which capture the subjective cognitive evaluations of an argument's impact, are more effective than categorical emotion models in predicting argument convincingness.

The Gist

Imagine you are sitting in a town hall meeting. Someone stands up and gives a speech trying to convince you to support a new law.

Some people in the room nod along, thinking, "Wow, that makes total sense!" Others shake their heads, thinking, "That's terrible!"

For decades, scientists studying arguments have tried to figure out why some speeches win people over and others don't. They usually looked at two things:

1. The Logic: Is the math right? Are the facts true? (The "Brain" part).
2. The Speaker: Do they look trustworthy? (The "Credibility" part).

But this paper argues there's a third, crucial ingredient: The Feeling. How does the speech make the listener feel?

The Big Question: "What" vs. "Why"

The researchers wanted to know: To predict if an argument is convincing, is it better to know what emotion the listener feels, or why they feel that way?

They compared two ways of looking at emotions:

1. The "Label" Approach (Categorical Emotions)

This is like looking at a weather report that just says: "It's Raining."

• How it works: You tell the computer, "The listener feels Anger."
• The problem: "Anger" is a broad label. One person might be angry because they feel insulted; another might be angry because they feel scared. The label "Anger" doesn't tell you why the anger happened, so it's hard to know if that anger will make them agree or disagree with the speaker.

2. The "Reasoning" Approach (Appraisal Theory)

This is like looking at a detailed weather report that says: "It's raining heavily, it's cold, and you forgot your umbrella, so you feel helpless."

• How it works: Instead of just saying "Anger," the computer analyzes the thoughts behind the feeling. It asks:
  • "Is this situation pleasant or unpleasant?"
  • "Did the speaker break a rule?"
  • "Can the listener fix this problem?"
  • "Is this important to them?"
• The benefit: This gives the computer a map of the listener's mind. It understands the story behind the emotion.

The Experiment: The "Magic Box" Test

The researchers used a "Magic Box" (which is actually a fancy AI called a Large Language Model) to test these two approaches. They gave the AI thousands of arguments and asked it to guess how convincing they would be to a human.

They ran three types of tests:

1. The Solo Test: The AI just reads the argument and guesses the score.
2. The "Label" Test: The AI reads the argument and is told, "The listener feels Sadness."
3. The "Reasoning" Test: The AI reads the argument and is told, "The listener thinks this is unpleasant, important, and violates a rule."

The Results: The Detective Wins

Here is what they found:

• Knowing the "Label" helped a little. Telling the AI "The listener is Sad" made it slightly better at guessing the score.
• Knowing the "Reasoning" helped a lot. Telling the AI why the listener felt that way (the Appraisal) made the AI significantly smarter. It was like giving the detective a full case file instead of just a suspect's name.
• The "All-in-One" approach failed. When they tried to make the AI guess the emotion and the score at the same time, it got confused and performed worse. It's like trying to juggle while solving a math problem; you drop the balls.

The Takeaway: Don't Just Read the Emotion, Read the Mind

The main lesson of this paper is that emotions are subjective.

If you tell a computer "This argument makes people angry," the computer doesn't know if that anger will make them agree (because they are angry at the problem) or disagree (because they are angry at the speaker).

But if you tell the computer, "This argument makes people angry because they feel it violates their moral standards," the computer understands the context. It knows that moral outrage often makes people more convinced of a cause.

In a Nutshell

Think of an argument like a movie.

• Categorical Emotions are like telling someone, "The movie is a Thriller." (You know the genre, but not the plot).
• Appraisal Theory is like telling someone, "The movie is a Thriller because the hero is trapped in a burning building and the villain is cutting the oxygen." (You understand the stakes and the reason for the fear).

The researchers found that to predict if an audience will love (be convinced by) the movie, you need to understand the plot and the stakes (Appraisals), not just the genre label.

Conclusion: To build better AI that understands human persuasion, we shouldn't just teach it to recognize feelings; we need to teach it to understand the thoughts that create those feelings.

Technical Summary

Here is a detailed technical summary of the paper "Categorical Emotions or Appraisals – Which Emotion Model Explains Argument Convincingness Better?"

1. Problem Statement

The convincingness of an argument is inherently subjective, depending not only on logical structure (logos) and the speaker's credibility (ethos) but also on the emotional response it evokes in the recipient (pathos). While previous research in computational argumentation has utilized categorical emotion models (e.g., Ekman's basic emotions like anger, joy, fear) to assess argument quality, these models often fail to capture the nuance of why an argument is convincing.

The authors hypothesize that the emotional response to an argument is driven by a subjective cognitive evaluation of the event by the recipient, based on their goals, prior knowledge, and standards. Appraisal theories (which model these cognitive evaluations) have been used in general emotion analysis but remain unexplored for assessing argument convincingness. The core problem is determining whether appraisal dimensions (cognitive evaluations) or categorical emotion labels better explain and predict the subjective convincingness of an argument.

2. Methodology

Data Source

The study utilizes the ContArgA corpus (Greschner et al., 2026), a dataset containing 800 arguments annotated by five participants each. The annotations include:

• Convincingness (CVC): Rated on a 1–5 scale.
• Discrete Emotions: Categorical labels (e.g., anger, trust, relief).
• Appraisal Dimensions: 15 specific cognitive variables (e.g., Pleasantness, Suddenness, Consequential Importance, Internal/External Checks) rated on a 1–5 scale.
• Context: Data was collected via a role-playing setup simulating a town-hall meeting to capture realistic recipient perspectives.

Experimental Setup

The authors conducted zero-shot prompting experiments using three Large Language Models (LLMs):

1. Mistral-Small-2407 (22B parameters)
2. LLaMA3.3:70B (70B parameters)
3. Gemma-3-27B-IT (27B parameters)

They compared four distinct configurations to answer two Research Questions (RQs):

• Baseline (CVC): Predict convincingness directly from the argument text.
• Pipeline Models (Unidirectional):
  • Emo→CVC: The model predicts convincingness given the argument text and the gold-standard dominant emotion category.
  • Appr→CVC: The model predicts convincingness given the argument text and the gold-standard 15 appraisal dimensions.
• Joint Models (Bidirectional):
  • Emo↔CVC: The model jointly predicts the emotion category and convincingness.
  • Appr↔CVC: The model jointly predicts the appraisal dimensions and convincingness.

Evaluation Metrics

• Convincingness: Spearman's $\rho$ correlation between predicted and gold-standard ratings.
• Emotion/Appraisal Prediction: Precision, Recall, and F1 scores (macro-average).

3. Key Contributions

1. First Systematic Comparison: This work presents the first direct comparison between categorical emotion models and appraisal theories for the specific task of predicting argument convincingness.
2. Validation of Appraisal Theory: It demonstrates that cognitive appraisal dimensions are superior to discrete emotion labels in modeling the subjective nature of argument evaluation.
3. Pipeline vs. Joint Analysis: It investigates whether joint prediction (simultaneously inferring emotion/appraisal and convincingness) improves performance over pipeline approaches, finding that pipeline methods using gold-standard inputs generally outperform joint modeling in this zero-shot setting.
4. Resource Utilization: It leverages the novel ContArgA corpus to bridge the gap between psychological theory and computational argument mining.

4. Results

RQ1: Which emotion model improves convincingness predictions?

• Categorical Emotions: Providing gold-standard emotion categories improved convincingness prediction for all three models compared to the baseline.
  • Best Improvement: Llama (+0.09), Gemma (+0.08).
• Appraisal Dimensions: Providing gold-standard appraisal dimensions yielded stronger improvements for Mistral and Llama, though Gemma performed worse than its baseline.
  • Best Improvement: Llama (+0.15), Mistral (+0.08).
• Conclusion: While both models help, appraisal dimensions generally provide a more significant boost to convincingness prediction than categorical emotions, supporting the hypothesis that cognitive evaluation is a stronger predictor of persuasiveness.

RQ2: Does joint prediction improve performance?

• Performance Drop: Jointly predicting emotions/appraisals and convincingness consistently underperformed compared to the pipeline approaches (where gold labels were provided).
  • Only Llama showed a marginal improvement in the joint setting for appraisals (+0.05 over baseline), but it still lagged behind the pipeline approach using gold labels.
  • Mistral and Gemma performed worse in the joint setting than in the baseline.
• Emotion Classification Difficulty: The models struggled significantly with emotion classification (Average F1 $\approx$ 0.12–0.14) and appraisal prediction (Average correlation $\approx$ 0.11), indicating that zero-shot prompting is insufficient for accurately inferring these subjective states from text alone.

5. Significance and Implications

• Theoretical Insight: The results validate that the subjective convincingness of an argument is better explained by the recipient's cognitive appraisal (e.g., "Is this pleasant?", "Does this violate my norms?") rather than just the resulting emotional label (e.g., "Anger"). This aligns with psychological appraisal theories in NLP.
• Practical Application: For computational argumentation systems aiming to assess or generate persuasive arguments, incorporating appraisal-based features is more effective than relying solely on sentiment or emotion detection.
• Limitations & Future Work:
  • The study relied on zero-shot prompting; the low performance in joint prediction suggests a need for fine-tuning or multi-task learning strategies to better capture the bidirectional relationship between emotion and convincingness.
  • The dataset is limited to short, isolated arguments in English; future work should explore cross-lingual contexts and longer discourse structures.
  • Ethical Considerations: The authors highlight the risk of these models being used to craft manipulative arguments that exploit cognitive biases and emotional vulnerabilities, urging for robust safety measures in downstream applications.

In summary, the paper argues that to truly understand why an argument convinces a specific audience, AI systems must model the cognitive appraisal process rather than just the resulting emotional state.