A Multi-Model, Multi-Domain Benchmark of Large Language Model Agreement with Humans and with Each Other in Sentiment Classification
This paper presents a multi-domain benchmark evaluating five large language models, a rule-based lexicon, and a supervised transformer against human annotations, revealing that while top-tier models like Claude Opus 4.7 and GPT-5.5 achieve strong alignment with humans, significant disagreement persists across models and with human labels—particularly on informal text—demonstrating that model selection materially impacts sentiment classification outcomes.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of computers that read and understand language, there is a task as old as the internet itself: figuring out if a piece of text is happy, sad, or just stating a fact. This is called sentiment analysis. For years, people have used these tools to guess how a stock market will react to a news report, to see if customers are angry at a product, or to track public mood during a health crisis. Recently, a new generation of these computer programs, known as large language models, has arrived. These are the same types of systems that can write stories, answer questions, and hold conversations. Because they are so good at understanding human language, many people have started using them to do the job of sorting text by emotion, hoping to replace the slow, expensive work of human reviewers with a single computer command.
But there is a problem with this new approach. Just as different people might disagree on whether a movie review is sarcastic or sincere, these different computer programs might not agree with each other. One program might call a sentence "positive," while another calls the exact same sentence "negative." If a bank or a hospital relies on these tools to make important decisions, it matters deeply which one they choose. Until now, no one had systematically checked how well these different programs agree with each other, or how often they agree with actual human judges, when they are all looking at the same text.
A researcher named Aneesh K Sajan decided to settle this question by running a massive, controlled experiment. He gathered five of the most advanced language models currently available from four different technology companies, along with two older, simpler tools that serve as a baseline. He fed them a total of 8,872 sentences drawn from five different sets of data. These sentences came from a mix of sources: formal financial reports, casual social media posts, movie reviews, and earnings calls. The goal was to see if the computers could correctly identify the emotion in the text compared to how humans had labeled it, and more importantly, to see how often the computers agreed with one another.
The results revealed a clear split in performance. Two models, one from a company called Anthropic and another from OpenAI, stood out as the most accurate. When these two looked at formal financial text, they matched human judgments almost perfectly. They also agreed with each other on nearly 90 percent of the sentences they analyzed. A third model from OpenAI was very close behind, forming a tight group of high-performing tools that generally saw the world in the same way. However, a model from Google, which was designed to "think" through problems before answering, performed shockingly poorly. It got the sentiment wrong so often that its results were barely better than random guessing. The researcher tested a specific theory to see if this failure was because the Google model was spending too much time thinking about simple questions, a phenomenon known as "overthinking." He forced the model to answer with a much shorter thinking time, but the poor performance remained. This ruled out the idea that the model was simply over-analyzing the task; the failure was inherent to how that specific model was behaving in this setup.
Another open-source model, which can be run on a researcher's own computer rather than a cloud server, performed moderately well. It was not as accurate as the top commercial models, but it was significantly better than the older, rule-based tools that had been used for years. The study also found that the type of text mattered greatly. All the models were much better at understanding formal, professional writing than they were at understanding casual, informal language like tweets or text messages. On informal text, even the best models dropped to a level of agreement that was only "moderate," meaning they still disagreed with humans and with each other on a large number of sentences.
Perhaps the most striking finding was how little the models actually agreed with one another. When the researcher looked at the entire group of six tools, they all gave the exact same answer on only about 14 to 24 percent of the sentences. On the remaining sentences, there was no clear majority. This means that for a huge portion of the text people analyze every day, the final result depends entirely on which computer program you happen to be using. If a financial analyst uses one model to read a report and another analyst uses a different model, they could reach opposite conclusions about the same news.
The study concludes that while these new tools are powerful, they are not interchangeable. The choice of model is not a minor technical detail; it is a decision that changes the data itself. For tasks involving formal financial documents, the top models are reliable and consistent. But for informal text, or for tasks where the difference between positive and negative is subtle, the disagreement between models is a sign of genuine uncertainty. In these cases, the researcher suggests that relying on a single computer is risky. Instead, the best approach is to use multiple models together or to have a human review the sentences where the computers disagree, ensuring that the final decision is not just a product of which software was chosen.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.