Sentiment Classification of Saudi Stock Market Tweets: Lexicon-Based Weak Supervision, Classical Machine Learning, and Zero-Shot Open-Source Large Language Models
This study evaluates sentiment classification of Saudi stock market tweets by comparing classical machine learning models trained on lexicon-derived labels with a zero-shot open-source LLM, revealing that while classical models merely reproduce potentially flawed rule-based labels, the LLM offers a distinct, domain-aware perspective that highlights the critical need for human-annotated ground truth to validate financial sentiment analysis.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern financial world, prices are not set solely by balance sheets and economic reports. They are also shaped by the collective mood of investors, a force that can shift markets in real time. For decades, researchers have tried to measure this mood by reading news and social media posts, looking for words that signal optimism or fear. When a company announces a profit, the tone is usually positive; when it reports a loss, the tone is negative. This task is straightforward in English, where dictionaries of emotional words exist and computers can easily count them. However, the challenge becomes far more complex when the language is Arabic, specifically the Arabic used in the bustling financial markets of Saudi Arabia. The language is rich with variations in spelling, dialect, and grammar, and financial news often mixes factual announcements with genuine opinions, making it difficult for standard computer programs to tell the difference.
This complexity led a team of researchers at King Khalid University to investigate how well computers can understand the sentiment of tweets about the Saudi stock market. They focused on a specific question: do the traditional methods used to label these tweets actually work, or do they simply trick the computer into memorizing a set of rules? To find out, they gathered one thousand tweets about companies listed on the Saudi Exchange, known as Tadawul. These posts came from a mix of financial news outlets, market analysts, and individual traders. The researchers wanted to see if they could teach a computer to recognize whether a tweet was positive, negative, or neutral, and to compare two very different approaches: one that relies on a pre-made list of words, and another that uses a powerful, modern artificial intelligence system capable of understanding context without being explicitly taught.
The researchers began with the traditional approach, which is common in many studies of this kind. They created a list of 180 financial terms, dividing them into positive words like "profit" and "growth," negative words like "loss" and "decline," and neutral words like "analysis." They then used this list to label the one thousand tweets automatically. If a tweet contained a positive word, it was marked as positive; if it had no matching words, it was marked as neutral. Once the tweets were labeled, they trained four different types of classical computer models to learn from these labels. The goal was to see if these models could reproduce the labels the list had assigned. The results were strikingly high. A linear support vector machine, one of the models tested, correctly matched the list's labels in 86.1 percent of cases. This seemed like a success, suggesting the computer had learned to understand the market's mood.
However, the researchers then introduced a second, more advanced tool: a large language model called Qwen2.5. Unlike the classical models, this system was not trained on the labeled tweets. Instead, it was asked to read the tweets and decide the sentiment on its own, using only a simple instruction to act as a financial analyst. When the researchers compared the large language model's decisions to the original list-based labels, the agreement was surprisingly low. The model agreed with the list on only 37.6 percent of the tweets. The discrepancy was not random. The large language model tended to label factual announcements, such as dividend declarations or earnings reports, as positive. The original list, which relied strictly on matching specific words, had labeled these same announcements as neutral because they lacked emotional keywords.
This disagreement revealed a critical flaw in the original method. The researchers realized that the list of words and the tweets had not been prepared in the same way. The tweets had been cleaned and standardized to handle different spellings, but the list of words had not. As a result, many words in the tweets did not match the words in the list simply because of minor spelling differences, not because the sentiment was different. When the researchers fixed this mismatch by applying the same cleaning rules to the list, the situation changed dramatically. The number of tweets that matched the list jumped from 42.4 percent to 67.8 percent. More importantly, the agreement between the large language model and the corrected list rose to 57.5 percent. This showed that a large part of the initial disagreement was not due to the artificial intelligence being wrong, but rather due to the reference labels being flawed.
The study concludes that high accuracy scores in these types of research can be misleading. When a computer model scores 86 percent accuracy against a set of labels, it often means the model has simply learned to follow the same rigid rules used to create those labels, rather than truly understanding the sentiment. The large language model, despite its lower agreement with the flawed labels, appeared to capture a more nuanced view of the market, recognizing that factual news about profits is often a positive signal for investors. The researchers emphasize that to truly know which method is correct, a human must step in to verify the labels, as no computer program can currently be trusted to validate its own training data without a human reference. This work serves as a reminder that in the rush to automate financial analysis, the quality of the data used to teach the machines is just as important as the machines themselves.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.