The Overlooked Repetitive Lengthening Form in Sentiment Analysis

1. Problem Statement

The paper addresses a significant gap in Natural Language Processing (NLP) regarding Repetitive Lengthening Form (RLF), an informal linguistic style prevalent in online communication (e.g., "loooove," "amazing!!!!"). While Language Models (LMs) are widely used for Sentiment Analysis (SA), their ability to understand and explain the sentiment conveyed by RLF remains under-investigated.

The authors identify two core research questions:

1. Importance: Is RLF a critical feature for document-level sentiment analysis?
2. Understanding: Can current Pre-trained Language Models (PLMs) and Large Language Models (LLMs) effectively understand and explain RLF?

The lack of a specialized, large-scale dataset for RLF has hindered the evaluation and improvement of LMs on this specific task.

2. Methodology

A. The Lengthening Dataset

To address the data scarcity, the authors curated Lengthening, the first multi-domain dataset focused on RLF for SA.

• Scale: 850,000 samples.
• Sources: Derived from four public datasets covering five domains: Amazon Reviews (Books, Electronics), Yelp (Restaurants), TripAdvisor (Hotels), and Twitter.
• Extraction Pipeline:
  1. Identification: Used a regular expression (RLFsearch) to detect repeated letters (e.g., "soooo") and repeated punctuation (e.g., "!!!").
  2. Extraction: Segmented documents into sentences, filtering for those containing RLF words.
  3. Normalization: Identified root words (e.g., "loooove" $\to$ "love") to maintain semantic grounding.
• Statistics: RLF appears in approximately 5.8% of documents on average, with some domains (like Twitter) exceeding 13%.

B. ExpInstruct Framework

The authors propose ExpInstruct, a two-stage instruction tuning framework designed to improve both the performance and explainability of open-source LLMs for RLF tasks.

1. Word Importance Scoring (WIS) Generation:
   • GPT-4 is prompted with Chain-of-Thought (CoT) to decompose RLF sentences and assign sentiment importance scores (1–5) to each word.
   • This generates high-quality, explainable training data without human annotation costs.
2. Instruction Tuning:
   • The framework uses a unified prompt template for two tasks:
     • Task 1 (Explainability): Predicting WIS for a given sentence.
     • Task 2 (SA): Predicting the document-level sentiment label (0 or 1) based on the sentence.
   • LLM Training: Open-source LLaMA2 (13B) is fine-tuned using LoRA (Low-Rank Adaptation) and Supervised Fine-Tuning (SFT) on a small subset (1,600 samples) derived from the Lengthening dataset.

C. Unified Explainability Evaluation

To compare explainability across different model architectures (PLMs vs. LLMs), the authors propose a unified metric:

• For LLMs: Uses prompt-based WIS generation.
• For PLMs: Uses a saliency-based method (occlusion) to measure the change in loss when a word is removed.
• Normalization: Applies min-max and L1 normalization to WIS values to create a comparable score ($S_{exp}$).
• Metric: $S_{exp}$ measures the average normalized importance assigned to RLF words. A higher score indicates the model better understands the expressive value of RLF.

3. Key Contributions

1. Dataset Creation: Introduction of Lengthening, a large-scale (850k), multi-domain dataset specifically for RLF in sentiment analysis.
2. Framework Proposal: Development of ExpInstruct, a cost-effective method that enables open-source LLMs to match zero-shot GPT-4 in both performance and explainability using limited data.
3. Unified Evaluation: A novel approach to quantify and compare the explainability of PLMs and LLMs regarding informal expressions.
4. Empirical Insights: Demonstration that RLF sentences act as "signatures" for document-level sentiment and that fine-tuned PLMs can outperform zero-shot GPT-4 in accuracy but often fail in explainability.

4. Experimental Results

Performance (Accuracy & F1)

• RLF Importance: Sentences containing RLF consistently yield higher accuracy and F1 scores than non-RLF sentences across all models (both zero-shot and fine-tuned).
• Fine-tuning vs. Zero-shot:
  • Fine-tuned PLMs (RoBERTa, GPT-2, T5) on the Lengthening dataset surpassed zero-shot GPT-4 in accuracy (e.g., RoBERTa achieved ~91.6% vs. GPT-4's ~86.3%).
  • However, fine-tuned PLMs lagged significantly behind GPT-4 in explainability ($S_{exp}$), suggesting they might be "right for the wrong reasons."
• ExpInstruct Success: The ExpInstruct-tuned LLaMA2 achieved performance and explainability scores comparable to zero-shot GPT-4 using only 1,600 training samples.

Explainability Analysis

• Human Evaluation: An inter-rater agreement (IAA) of 0.44 was found between human annotators and model-generated explanations.
• Correlation: There is a strong correlation (0.91) between the human reliability scores and the automated $S_{exp}$ metric, validating the unified evaluation approach.
• Gap: Fine-tuned PLMs showed lower $S_{exp}$ scores than GPT-4, indicating a lack of deep understanding of RLF nuances despite high accuracy.

Transferability

• Fine-tuning models on RLF sentences improved document-level sentiment analysis performance significantly (average +7.6% Accuracy, +4.5% F1), proving that RLF features are transferable and critical for holistic document understanding.

5. Significance and Conclusion

• Informal Style Recognition: The paper highlights that RLF is not just noise but a crucial, expressive signal in social media sentiment analysis. Ignoring it leads to suboptimal performance.
• Efficiency: The study demonstrates that open-source models can rival proprietary giants (GPT-4) in handling nuanced informal language if trained with the right instruction strategies (ExpInstruct) and high-quality synthetic data.
• Explainability Gap: It reveals a critical limitation in current PLMs: they can predict sentiment accurately but often fail to explain why, specifically regarding informal linguistic features.
• Future Directions: The methodology is language-agnostic and can be extended to other languages where RLF is common (e.g., Spanish, Romanian, Chinese), offering a pathway for more robust global social media analysis.

In summary, this work bridges the gap between linguistic nuance and machine learning by providing the data, the framework, and the evaluation metrics necessary to make LMs truly "understand" the emotional weight of repetitive lengthening in digital communication.