Strengthening False Information Propagation Detection: Leveraging SVM and Sophisticated Text Vectorization Techniques in comparison to BERT

Technical Summary: Strengthening False Information Propagation Detection via SVM and Text Vectorization

Problem Statement
The rapid dissemination of misinformation through online platforms poses a significant threat to public opinion, societal stability, and democratic frameworks. The volume of daily data renders conventional manual fact-checking insufficient. While deep learning models have shown promise, there is a need to evaluate the efficacy of classical machine learning approaches, specifically Support Vector Machines (SVM), combined with various text vectorization techniques, to determine if they can offer competitive performance with lower computational overhead compared to large transformer models.

Methodology
This study utilizes the ISOT "Fake and Real News Dataset," comprising approximately 21,477 real news articles (sourced from Reuters) and 23,421 fake news articles (sourced from Politifact and Wikipedia). The research employs a rigorous preprocessing pipeline involving tokenization, stopword removal, and the elimination of punctuation and short words using NLTK and Gensim libraries.

The core methodology involves a comparative analysis of two classification approaches:

1. SVM Classifiers: The study implements SVMs with both Linear and Radial Basis Function (RBF) kernels. These are tested against three distinct text vectorization techniques:
   • Bag of Words (BoW): A baseline method converting text into token count matrices.
   • TF-IDF (Term Frequency-Inverse Document Frequency): A method that weighs terms based on their frequency in a document relative to the corpus.
   • Word2Vec: A dense vector representation capturing semantic relationships using the Continuous Bag-of-Words (CBOW) architecture.
2. BERT Base Model: A transformer-based Large Language Model (BERT-base-uncased) is trained on the same dataset to serve as a high-performance benchmark.

The dataset is split into training and testing sets, and models are evaluated using accuracy, precision, recall, and F1-scores. Visualization techniques, including 3D t-SNE plots and ROC curves, are used to analyze data separability and model performance.

Key Contributions

• Comparative Vectorization Analysis: The paper provides a detailed comparison of BoW, TF-IDF, and Word2Vec when paired with SVM classifiers for fake news detection, identifying the most efficient text representation methods for this specific task.
• Kernel Evaluation: It analyzes the marginal benefits of using an RBF kernel versus a linear kernel in SVMs for this domain.
• Resource-Constrained Benchmarking: The study contrasts the performance of lightweight SVM models against the computationally intensive BERT model, highlighting the trade-offs between accuracy and resource requirements.
• Dataset Utilization: The research utilizes a preprocessed version of the ISOT dataset, ensuring the findings are grounded in a diverse spectrum of political and world news.

Results
The experimental results demonstrate that while the BERT base model achieves the highest performance with an accuracy of 99.98% and an F1-score of 0.9998 (trained in 3 epochs on a GPU), the SVM models offer highly competitive results with significantly lower computational costs.

• SVM Performance: The SVM classifier using a Linear Kernel and BoW vectorization achieved the best results among the SVM configurations, with 99.81% accuracy and an F1-score of 0.9980.
• TF-IDF Performance: The SVM with a Linear Kernel and TF-IDF vectorization followed closely with 99.52% accuracy and an F1-score of 0.9949.
• Word2Vec Performance: Word2Vec yielded lower performance (96.54% accuracy with Linear Kernel), suggesting that for this specific dataset, frequency-based methods outperformed semantic embedding methods.
• Kernel Comparison: The RBF kernel provided only marginal improvements over the Linear Kernel (e.g., BoW + RBF reached 99.62% vs. 99.81% for Linear), indicating that the added computational complexity of the RBF kernel may not be justified for this specific application.
• Computational Efficiency: A critical finding is the disparity in training time and hardware. The BERT model required an NVIDIA Tesla T4 GPU and approximately 1 hour 45 minutes for training, whereas the SVM models trained on a standard CPU (Intel Core i7) in under 3 minutes.

Significance and Claims
The paper claims that despite the superior accuracy of transformer models like BERT, SVM models combined with simple vectorization techniques (specifically BoW and TF-IDF) offer a highly effective, resource-efficient alternative for fake news detection. The study suggests that in scenarios with limited computational resources or real-time constraints, these classical machine learning approaches are viable and robust. The authors conclude that the high performance of BoW and TF-IDF with SVMs underscores the importance of word frequency in distinguishing fake news, and that sophisticated semantic embeddings (like Word2Vec) may not always be necessary for this specific classification task. The work aims to contribute to the preservation of information integrity by demonstrating that high-accuracy detection systems can be built without the heavy computational overhead of large language models.