Sentiment Analysis of Government Mobile App Reviews: Validating Zero-Shot Classification with a Large Language Model

Every day, millions of people interact with their governments through the small screens of their smartphones. They use apps to pay for prescriptions, renew identity documents, or access public records. While these tools are meant to make life easier, they also generate a constant stream of feedback in the form of user reviews. These short, informal texts are a goldmine for understanding how citizens feel about digital public services, but they are also a challenge for researchers. Unlike a simple star rating, which offers a single number, a review contains words that carry nuance, irony, and complex emotions. In countries where the primary language is not English, analyzing this feedback becomes even harder because the sophisticated computer programs usually used for this task are often trained on English data and struggle with the specific grammar and vocabulary of other tongues.

For a long time, the solution to this problem has been to either translate the reviews into English or to build custom software for each specific language, a process that requires vast amounts of labeled data and technical expertise. However, a new type of artificial intelligence, known as a large language model, has emerged. These models are trained on enormous amounts of text from across the internet and can understand context and instruction without needing to be retrained for every new job. The question facing researchers is whether these powerful tools can accurately read and categorize the feelings in short, non-English reviews of government apps without any special tuning, and if they can do so consistently enough to be trusted by public administrators.

A team of researchers from the Jan Evangelista Purkyně University in Ústí nad Labem set out to answer this question using the Czech language as their test case. They gathered a massive collection of over ten thousand reviews from six different government mobile applications used by citizens in the Czech Republic. These apps cover a wide range of services, from electronic prescriptions to digital identity cards. The researchers wanted to see if a standardized process using a large language model could sort these reviews into three simple categories: positive, negative, or neutral. To ensure the results were reliable, they did not simply trust the computer's output. Instead, they created a "ground truth" by having human experts read and agree on the sentiment of a carefully selected sample of five hundred reviews. This human-verified sample served as the benchmark against which the computer's performance was measured.

The study compared the large language model against several other methods, including simple word-counting tools and older types of AI that are commonly used for text analysis. The word-counting tools, which rely on dictionaries of positive and negative words, performed poorly. They often failed to understand the context of a sentence, missing the fact that a word like "but" could completely flip the meaning of a review. The older AI models, while better than the dictionaries, still struggled to achieve high accuracy without specific training on Czech government data. In contrast, the large language model, when given clear instructions on how to classify the text, demonstrated a remarkable ability to understand the nuances of the Czech language. It correctly identified the sentiment in the vast majority of the test cases, matching the human experts' judgments with a high degree of agreement.

One of the most critical findings of the research was the stability of the results. Because large language models are complex systems, there was a concern that running the same analysis twice might produce different answers. The researchers tested this by running the classification process five separate times under identical conditions. They found that the newer version of the model was incredibly consistent, producing nearly the same results every time. This consistency is vital for real-world use, as it means the tool can be relied upon to give the same answer to the same problem, regardless of when it is used. The older version of the model, however, showed much more variation between runs, suggesting that the specific version of the software matters greatly.

When the researchers applied the most stable and accurate model to the entire collection of ten thousand reviews, a clear picture of citizen sentiment emerged. The analysis revealed that negative feedback was the most common, with nearly two-thirds of all reviews expressing dissatisfaction. Positive reviews made up about a quarter of the total, while truly neutral comments were the rarest, accounting for less than eight percent. Interestingly, the text of the reviews did not always perfectly match the star ratings users gave the apps. While a one-star rating almost always corresponded to negative text, and a five-star rating usually matched positive text, the middle ground was messy. Many three-star reviews contained negative text, and some five-star reviews included complaints, showing that users often mix their feelings in ways that a simple star rating cannot capture.

The study also highlighted where the technology still faces challenges. The most difficult category for the computer to get right was the "neutral" sentiment. This category includes factual descriptions, questions, or mixed feelings where no strong emotion is present. The model sometimes struggled to distinguish between a neutral comment and one that was slightly negative or positive. However, even with this difficulty, the model outperformed all other available methods that did not require custom training. The researchers concluded that a standardized procedure using a large language model can effectively analyze short, non-English citizen feedback, provided that the process is carefully documented and validated against human judgment.

This work does not suggest that the technology is perfect or that it can replace human insight entirely. The researchers emphasized that the model's performance depends on the specific instructions given to it and the quality of the validation process. They also noted that the results are specific to this particular set of government apps and the Czech language, and that different applications or languages might yield different outcomes. Nevertheless, the study provides strong evidence that public administrators can now use these tools to listen to their citizens at scale, even in languages where specialized software was previously unavailable. By turning thousands of unstructured comments into clear, categorized data, these tools offer a new way to understand the daily experiences of people using digital public services, helping governments improve the tools that serve them.