Classification of Public Opinion on the Free Nutritional Meal Program on YouTube Media Using the LSTM Method

1. Problem Statement

The study addresses the challenge of efficiently analyzing public perception regarding the Free Nutritious Meal Program (MBG), a government initiative in Indonesia aimed at improving child nutrition. While social media platforms like YouTube generate vast amounts of real-time textual data reflecting public opinion, manually analyzing these comments is infeasible due to volume and unstructured nature.

• Core Issue: There is a need for an automated, data-driven approach to classify sentiments (Positive, Negative, Neutral) to evaluate policy acceptance.
• Specific Challenge: The dataset exhibits a severe class imbalance, with negative sentiments dominating the data (87.7%), which typically biases machine learning models and hinders the detection of minority classes (positive sentiments).

2. Methodology

The research employs a quantitative approach using Natural Language Processing (NLP) and Deep Learning. The workflow consists of the following stages:

A. Data Collection

• Source: Comments scraped from two YouTube channels discussing the MBG program.
• Volume: A total of 7,733 comments were collected.
• Distribution: 5,629 negative comments (87.7%) and 790 positive comments (12.3%). Note: The study focuses on binary classification (Positive vs. Negative).

B. Preprocessing Pipeline

The raw text underwent a rigorous cleaning process to prepare it for modeling:

1. Case Folding: Conversion to lowercase.
2. Cleaning: Removal of URLs, mentions, hashtags, numbers, and special characters.
3. Normalization: Standardization of slang and non-standard Indonesian words.
4. Tokenization: Splitting text into individual words.
5. Stopword Removal: Elimination of non-significant words using the NLTK library.
6. Stemming: Reducing words to their root forms using the Sastrawi library (specific for Indonesian).

C. Feature Representation

• Tokenization & Padding: Text was converted to integer indices. Sequences were padded to ensure uniform length for batch processing.
• Embedding: Words were mapped to 128-dimensional vectors.

D. Model Architecture

The study utilized Long Short-Term Memory (LSTM) networks, chosen for their ability to capture long-term dependencies and sequential context in text, outperforming traditional RNNs.

• Architecture:
  • Embedding Layer: 128 dimensions.
  • LSTM Layer: Hidden dimension of 128 with a dropout rate of 0.3.
  • Dropout Layer: 0.5 rate to prevent overfitting.
  • Fully Connected (FC) Layer: 2 output neurons.
  • Output Layer: Softmax activation for binary classification.
• Hyperparameters:
  • Optimizer: Adam (Learning Rate: 0.0005).
  • Loss Function: CrossEntropyLoss.
  • Training: 20 Epochs, Batch Size 16.
  • Total Parameters: ~2.23 million.
• Baseline Comparison: The LSTM model was compared against traditional machine learning algorithms: SVM, Logistic Regression, and Naive Bayes.

E. Evaluation Metrics

Performance was assessed using a confusion matrix and the following metrics:

• Accuracy: Overall correctness.
• Precision, Recall, and F1-Score: Specifically analyzed to address the class imbalance issue.

3. Key Results

The LSTM model demonstrated strong overall performance but revealed significant disparities between classes due to data imbalance.

• Overall Accuracy: 89% on the test set (963 samples).
• Negative Class Performance (Majority Class):
  • F1-Score: 0.94
  • Precision: 0.94
  • Recall: 0.93
  • Interpretation: The model is highly effective at identifying negative sentiments.
• Positive Class Performance (Minority Class):
  • F1-Score: 0.55
  • Precision: 0.54
  • Recall: 0.57
  • Interpretation: The model struggles to detect positive sentiments, often misclassifying them as negative (False Negatives).
• Training Dynamics: The model achieved a training accuracy of 98.26% and validation accuracy of 88.99%, with loss decreasing steadily from 0.68 to 0.1225 over 20 epochs.

4. Key Contributions

1. Policy Evaluation Framework: Provides a scalable, automated framework for the Indonesian government to evaluate public reception of the MBG program using social media data.
2. LSTM Effectiveness on Indonesian Text: Validates that LSTM architectures are highly effective for sentiment analysis in the Indonesian language, particularly for capturing context in informal social media comments.
3. Empirical Evidence of Imbalance Impact: The study explicitly quantifies how severe class imbalance (87.7% negative) degrades the performance of deep learning models on minority classes, offering a cautionary case study for future sentiment analysis projects.
4. Comparative Analysis: Confirms that deep learning (LSTM) outperforms traditional methods (SVM, Naive Bayes) in handling the sequential nature of social media text, despite the data imbalance.

5. Significance and Future Directions

• Significance: The research highlights that while deep learning is powerful, it is not a "silver bullet" for imbalanced datasets. The high accuracy (89%) masks the model's inability to hear the "positive voice" of the public, which is critical for a balanced policy assessment.
• Limitations: The primary limitation is the data imbalance, leading to a bias toward the negative class.
• Future Recommendations:
  • Data Balancing: Apply techniques like SMOTE (Synthetic Minority Over-sampling Technique) or class weighting during training.
  • Data Collection: Actively seek more positive sentiment data to balance the dataset.
  • Architecture Upgrade: Explore advanced architectures like Bi-LSTM (Bidirectional LSTM) or hybrid CNN-LSTM models to further improve feature extraction and minority class detection.

In conclusion, the paper successfully demonstrates the utility of LSTM for Indonesian sentiment analysis but underscores the critical necessity of addressing data imbalance to ensure fair and comprehensive public opinion monitoring.