From Prediction to Justification: Aligning Sentiment Reasoning with Human Rationale via Reinforcement Learning

1. Problem Statement

Aspect-Based Sentiment Analysis (ABSA) is a fine-grained task aimed at identifying sentiment polarities toward specific aspects in text (e.g., determining that "battery life" is positive while "screen" is negative in a review). While current models achieve high accuracy, they operate as "black boxes," lacking the explicit reasoning capabilities found in human affective cognition.

• The Gap: Humans do not merely categorize sentiment; they construct causal explanations (System 2 thinking) before reaching a conclusion. Existing Large Language Models (LLMs) often output labels without providing human-intelligible rationales, limiting interpretability and trust.
• The Challenge: How to endow LLMs with the ability to explicitly learn and generate the "reasoning path" (the why) that justifies the sentiment prediction (the what), particularly for complex tasks like Aspect-Opinion-Sentiment Triplet Extraction (AOSTE).

2. Methodology: ABSA-R1

The authors propose ABSA-R1, a Reinforcement Learning (RL) framework designed to mimic a "reason-before-predict" cognitive paradigm. The model generates a natural language reasoning path ($p$) before outputting the final sentiment decision ($\hat{y}$).

Core Components:

1. Reasoning-First Generation:

   • Given an input $x$, the model generates a composite output $o = (p, \hat{y})$.
   • $p$ is a structured chain-of-thought explaining the causal logic (e.g., identifying keywords, analyzing context, verifying against counter-evidence).
   • $\hat{y}$ is the final label or extracted triplet.

2. Sentiment-Aware Reward Model (SAR):

   • To prevent "reward hacking" (where models exploit shallow patterns), the authors designed a rule-based, interpretable reward function:
     $$R(o, y) = \lambda \cdot R_f(p) + (1 - \lambda) \cdot R_a(\hat{y}, y)$$
   • Format Reward ($R_f$): Ensures structural coherence, including the use of specific tags (e.g., <thought>, <answer>) and logical transition phrases (e.g., "First," "Therefore").
   • Accuracy Reward ($R_a$): Evaluates the correctness of the final prediction. For AOSTE, it uses a soft F1-based reward with penalties for hallucinations (false positives) or omissions (false negatives).
   • Hyper-parameter ($\lambda$): Balances the trade-off between reasoning structure and prediction accuracy. Experiments show $\lambda=0.2$ is optimal, prioritizing semantic correctness while maintaining structural discipline.

3. Performance-Driven Rejection Sampling (PRS):

   • Inspired by metacognitive monitoring, this strategy acts as a filter during the RL training process.
   • Mechanism: For each input, the model generates $G$ candidate trajectories. If a trajectory yields a correct prediction, it is discarded. If the trajectory is incorrect or incomplete, it is retained for reinforcement learning.
   • Goal: This forces the model to focus exclusively on "hard cases" and its own errors, accelerating convergence on ambiguous or complex semantic scenarios (e.g., negation, implicit sentiment).

4. Training Algorithm:

   • The model is trained using an improved variant of Generalized Reward Policy Optimization (GRPO).
   • The objective function optimizes the policy $\pi_\theta$ based on the advantage of generated paths relative to the mean reward of the batch, mitigating issues like entropy collapse and premature overfitting.

3. Key Contributions

1. ABSA-R1 Framework: The first RL-based LLM framework explicitly designed to mimic human-like sentiment reasoning in ABSA, generating interpretable justifications that ground predictions.
2. Cognition-Aligned Reward & Sampling: Introduction of a dual-mechanism approach:
   • A Sentiment-Aware Reward Model that synchronizes prediction accuracy with logical reasoning quality.
   • A Performance-Driven Rejection Sampling strategy that enhances robustness by prioritizing learning from model failures (error-driven learning).
3. State-of-the-Art (SOTA) Performance: Demonstrated that explicit reasoning not only improves interpretability but also significantly boosts performance in both sentiment classification and triplet extraction compared to non-reasoning baselines.

4. Experimental Results

The model was evaluated on four SemEval benchmarks (Lap14, Rest14, Rest15, Rest16) for two tasks: Aspect-Opinion-Sentiment Triplet Extraction (AOSTE) and Aspect-Based Sentiment Classification (ABSC).

• AOSTE Performance:

  • ABSA-R1 achieved a new SOTA with an average F1 score of 80.04.
  • It outperformed the strongest structured baseline (TAGS, 71.02) by 9.02 points and the best general LLM baseline (Qwen2.5-7B, 75.78) by 4.26 points.
  • The improvement was most significant on complex datasets like Rest15 (84.81 F1).

• ABSC Performance:

  • Achieved an average Accuracy of 89.95 and F1 of 80.88.
  • Outperformed the instruction-tuned baseline (InstructSFT) by 5.59 F1 points, proving that reasoning traces benefit even standard classification tasks.

• Ablation Studies:

  • Without PRS: Performance dropped drastically (e.g., AOSTE F1 on Lap14 fell from 69.37 to 57.17), confirming that learning from errors is critical.
  • Without SAR: Performance declined (e.g., Rest14 AOSTE F1 dropped by 9.25 points), showing that the reward model is essential for generating valid, non-hallucinated reasoning.

• Human Evaluation:

  • In a qualitative study of 50 samples, 94.0% of the generated reasoning traces were judged as logically correct and factually consistent with the input text.

5. Significance

This work bridges the gap between machine prediction and human understanding in affective computing. By shifting from "black box" classification to "white box" reasoning, ABSA-R1:

• Enhances Trust: Provides transparent, human-readable justifications for sentiment decisions.
• Improves Robustness: The error-driven learning strategy (PRS) makes the model more resilient to complex linguistic phenomena like negation and implicit sentiment.
• Advances Cognitive Modeling: Successfully translates the psychological concept of "System 2" thinking (deliberate, step-by-step reasoning) into a practical NLP architecture, setting a new standard for interpretable AI in sentiment analysis.