Creating and Evaluating Figurative Language Dataset for Sindhi

Technical Summary: Creating and Evaluating Figurative Language Dataset for Sindhi (SiNFluD)

Problem Statement
Non-literal language expressions, including idioms, metaphors, sarcasm, and proverbs, are fundamental to human communication but pose significant challenges for Natural Language Processing (NLP). While extensive resources exist for high-resource languages like English and Chinese, low-resource languages such as Sindhi lack the necessary annotated datasets to train models for figurative language understanding. This scarcity hinders the performance of downstream NLP tasks such as sentiment analysis, machine translation, and conversational agents, where misinterpreting non-literal intent can lead to reduced model robustness and bias. Despite Sindhi being spoken by over 35 million people with a rich literary and cultural history, no existing research or resource specifically addresses the classification of non-literal expressions in the Persio-Arabic script used in Pakistan and India.

Methodology
The authors present SiNFluD, a novel benchmark dataset designed to bridge this gap. The methodology encompasses data collection, annotation, preprocessing, and experimental evaluation:

• Data Collection: Raw text was aggregated from diverse sources, including literary books, Wikisource, Quora blogs, and dedicated Sindhi proverb blogs. The initial collection focused on expressions ranging from 2 to 19 tokens.
• Annotation Process: Two native Sindhi annotators labeled the data using the Doccano tool under the supervision of a linguistic expert. The dataset was structured into a binary classification task: Literal (label 0) and Non-literal (label 1). The non-literal category was further subdivided into four specific figurative forms: Idioms, Proverbs, Metaphors, and Similes.
• Preprocessing and Filtering: To ensure quality, the dataset underwent cleaning, deduplication, and formatting corrections. Expressions with token lengths greater than 13 were filtered out due to low frequency, resulting in a balanced dataset.
• Inter-annotator Agreement: Reliability was assessed using Cohen's Kappa coefficient, yielding a score of 0.81, indicating substantial agreement between annotators.
• Experimental Setup: The final dataset comprises 4,451 instances (2,038 literal and 2,413 non-literal). The authors established baselines using 5-fold and 10-fold cross-validation. They evaluated four models:
  • mBERT (Multilingual BERT)
  • XLM-RoBERTa
  • XLM-RoBERTa-XL (a larger-scale variant)
  • SetFit (a parameter-efficient few-shot learning framework using an MPNet backbone).
    All transformer models were fine-tuned for sequence classification over five epochs.

Key Contributions

1. SiNFluD Dataset: The release of the first dedicated benchmark dataset for figurative language classification in Sindhi, covering idioms, proverbs, metaphors, and similes.
2. High-Quality Annotation: A manually curated corpus with a high inter-annotator agreement (0.81), validated by linguistic experts.
3. Comprehensive Evaluation: A systematic evaluation of state-of-the-art multilingual pre-trained language models (PLMs) and few-shot learning approaches on Sindhi figurative language, establishing robust baselines for future research.

Results
The experimental results demonstrate that modern multilingual architectures are effective for this task:

• Baseline Stability: Cross-validation showed consistent performance, with 10-fold cross-validation achieving an average accuracy of approximately 90.34% and 5-fold achieving 90.56%, indicating the dataset is well-balanced and stable across splits.
• Model Performance:
  • XLM-RoBERTa-XL achieved the highest accuracy at 92.27%, suggesting that larger model capacity and extensive multilingual pretraining better capture the semantic nuances of Sindhi figurative language.
  • XLM-RoBERTa followed with 91.38%.
  • mBERT recorded 90.57%.
  • SetFit (few-shot approach) achieved 90.81%, demonstrating competitive performance despite its parameter efficiency.
• The narrow performance range across models suggests the dataset is sufficiently informative to allow different architectures to learn discriminative features effectively.

Significance
The paper claims that SiNFluD provides a foundational resource for advancing NLP research in Sindhi, specifically for tasks requiring the understanding of non-literal expressions. By addressing the scarcity of annotated data, the work enables the development of more robust models for sentiment analysis, machine translation, and dialogue systems in low-resource settings. Furthermore, the study offers insights into the efficacy of cross-lingual transfer learning and few-shot learning strategies for South Asian languages, highlighting that advanced multilingual pretraining significantly aids in capturing the cultural and semantic drift inherent in figurative language.