Technical Summary: Comparative Evaluation of Machine Learning and Deep Learning Models for Targeted Metaphor Detection

Problem Definition

The paper addresses the challenge of targeted metaphor detection, a specific natural language processing (NLP) task where the objective is to classify whether a pre-defined candidate word (e.g., "road," "light," "train") is used metaphorically or literally within a given textual context. Unlike general metaphor detection, this task focuses on specific lexical items, requiring models to discern semantic nuances and context-dependent meanings that distinguish figurative usage from literal interpretation. The authors note that this is a difficult classification problem due to the nuance, cultural sensitivity, and discourse-level shaping of figurative language, which often renders simple keyword matching insufficient.

Methodology

The study evaluates and compares six distinct modeling approaches using a labeled dataset comprising 1,870 training samples and 800 test samples. The dataset covers seven target words: road, candle, light, spice, ride, train, and boat. The models are categorized as follows:

1. TF-IDF-based Logistic Regression:

   • Approach: Utilizes standard text preprocessing (tokenization, lemmatization) and TF-IDF vectorization (unigrams and bigrams, capped at 5,000 features).
   • Mechanism: A logistic regression classifier is trained on these features, with class imbalance adjustments applied during training.

2. Hybrid LDA-SBERT-Random Forest:

   • Approach: A hybrid architecture combining topic modeling and deep semantic embeddings.
   • Mechanism: Text is preprocessed to map keywords to metaphor IDs. Latent Dirichlet Allocation (LDA) extracts topic distributions to provide thematic structure. Simultaneously, Sentence-BERT (SBERT) generates dense sentence embeddings to capture deep semantic meaning. These two feature sets are concatenated and fed into a Random Forest classifier.

3. LSTM with Hybrid Attention:

   • Approach: A deep learning sequence model.
   • Mechanism: Employs a Long Short-Term Memory (LSTM) network to process word embeddings and capture temporal dependencies. It is augmented with a Hybrid Attention mechanism (combining additive and dot-product attention) to dynamically prioritize tokens relevant to metaphor identification.

4. LSTM with BERT Embeddings:

   • Approach: Integration of pre-trained transformer representations with recurrent networks.
   • Mechanism: Text is processed by BERT to generate high-dimensional contextual embeddings. These embeddings are then fed into an LSTM layer to capture both short-term and long-term dependencies within the sequence.

5. XGBoost with Word2Vec:

   • Approach: Gradient boosting with distributed word representations.
   • Mechanism: Word2Vec is used to generate dense word embeddings capturing semantic and syntactic relationships. These embeddings serve as input features for an XGBoost classifier, which optimizes decision trees sequentially to minimize error.

6. Majority-Vote Consensus Model:

   • Approach: An ensemble method.
   • Mechanism: Aggregates predictions from all five individual models above. The final prediction is determined by the mode (majority vote) of the individual classifiers' outputs.

Evaluation Strategy: The dataset was split 80/20 (train/test). Models were evaluated using precision, recall, weighted F1-score, macro F1-score, and accuracy.

Key Results

The experimental results indicate that hybrid feature representations outperformed pure deep learning or traditional machine learning approaches in this specific context:

• Best Performer: The LDA-SBERT-Random Forest model achieved the highest overall performance with 84% accuracy and a weighted F1-score of 0.82. It demonstrated high precision for literal labels and excellent recall for metaphorical labels.
• Runner-up: Logistic Regression (TF-IDF) performed competitively with 80% accuracy and a weighted F1-score of 0.81.
• Consensus Model: The majority-vote ensemble achieved 82% accuracy and a weighted F1-score of 0.80. While it provided balanced predictions across classes, it did not outperform the best individual model (LDA-SBERT-RF).
• Underperformers:
  • LSTM with Custom Attention: Struggled significantly, achieving only 49% accuracy and a weighted F1-score of 0.52, failing to balance precision and recall.
  • LSTM with BERT Embeddings: Achieved 73% accuracy and a 0.72 weighted F1-score, showing better capability in capturing metaphorical language but lower performance on literal labels.
  • XGBoost with Word2Vec: Scored 74% accuracy and a 0.64 weighted F1-score, excelling in recall for metaphors but underperforming on literal labels.

Significance and Claims

The paper modestly claims that its primary contribution lies in demonstrating the efficacy of hybrid feature representations for metaphor detection, particularly within the constraints of small labeled datasets.

• Hybrid Effectiveness: The results suggest that combining contextual sentence embeddings (from SBERT) with topic-level information (from LDA) creates a robust feature set that outperforms models relying solely on deep learning sequences (LSTM) or static embeddings (Word2Vec) for this specific task.
• Ensemble Limitations: The study notes that while ensemble voting can improve prediction stability and balance, it does not necessarily enhance overall performance if the constituent models vary significantly in strength or if the ensemble includes weaker models that dilute the performance of the strongest individual classifier.
• Contextual Nuance: The success of the LDA-SBERT-RF model reinforces the necessity of capturing both thematic structure and deep semantic meaning to resolve the ambiguity inherent in metaphor detection.

The authors conclude that future work should focus on utilizing larger and more diverse datasets, conducting detailed error analysis by target word, and exploring advanced transformer-based or confidence-weighted ensemble methods.