Machine Learning versus Deep Learning for Public Financial Report Classification and Popularity Prediction: A Feasibility Study on Ghana's Controller and Accountant-General's Department
This feasibility study evaluates the performance of classical machine learning and deep learning models on Ghana's Controller and Accountant-General's Department financial reports, revealing that while fine-tuned XGBoost achieved perfect category classification, the subsequent ablation study demonstrated that keyword features significantly inflated these results, underscoring the critical need for rigorous feature validation before deploying such tools in public sector contexts.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Governments around the world are increasingly putting their financial records online. This shift is meant to bring transparency, allowing citizens and auditors to see how public money is spent without needing to visit a physical office. However, as these digital archives grow, they become difficult to manage. A single government department might publish hundreds of documents over a few years, ranging from annual budget statements to payroll circulars. Sorting these files by hand is slow and prone to error, while figuring out which reports people actually want to read is even harder. To solve this, agencies are looking to computers to do the sorting and predicting for them. This is where machine learning comes in. These are computer programs that learn to recognize patterns in data, such as the words in a document title, to make decisions like "this is a payroll report" or "this report will be downloaded often." But there is a catch: these programs usually need thousands of examples to learn well. When a government department only has a few hundred documents, it is unclear if these smart tools can actually work, or if they are just guessing based on lucky shortcuts.
In Ghana, the Controller and Accountant-General's Department holds a vast collection of public financial reports. Two researchers, Emmanuel Osei-Dwomoh and Gabriel Osei Forkuo, decided to test whether automated tools could organize this specific collection and predict its popularity. They gathered 131 official reports published between 2019 and 2024. These documents included everything from chart-of-accounts editions to international accounting standards materials. The researchers set out to build two different types of computer models to handle the work. The first type relied on "classical" machine learning, where a human analyst carefully creates a list of specific clues, such as counting how many times the word "budget" appears or noting the length of the file. The second type used "deep learning," a more complex approach where the computer reads the raw text of the titles and descriptions character by character, trying to find patterns on its own without human help. They tested these models on three specific jobs: sorting the reports into the correct category, guessing if a report would be downloaded more than the average, and predicting the exact number of downloads.
The results were surprising and revealed a hidden trap in how these systems learn. When the researchers first ran the tests, the classical machine learning model seemed perfect. It correctly sorted every single report in the test group into its right category, achieving a flawless score. The deep learning model also performed very well on this task. However, the researchers suspected that this perfect score was not because the computers truly understood the content of the reports. They noticed that the clues they had given the classical model included specific keywords that were almost identical to the category names. For instance, if a report was titled "2024 Payroll Circular," the model was given a flag that said "the word payroll is present." Since the category was "Payroll," the model was essentially being told the answer before it even started.
To prove this, the researchers removed those specific keyword clues and ran the test again. The perfect score vanished. The model's accuracy dropped significantly, settling at a level that was still quite good but far from perfect. This confirmed that the initial success was largely due to a shortcut: the reports often contained their own category names in their titles. The deep learning models, which read the text directly, also found this shortcut and achieved similar high scores, proving that the raw text itself contained the answer, not necessarily a deep understanding of the document's structure. When the researchers looked at the other tasks, the picture changed. Predicting which reports would be popular was much harder. The classical models performed reasonably well, correctly identifying popular reports about three-quarters of the time, while the deep learning models struggled more. The deep learning models also showed a strange behavior: when the researchers tried to "fine-tune" them by giving them more training time, their performance actually got worse. This suggests that with such a small collection of documents, the complex deep learning models were too sensitive and started to memorize the training data rather than learning general rules.
The study concludes that for small government archives like this one, simple, classical machine learning tools are a better starting point than the more complex deep learning systems. These simpler tools are easier to understand and less likely to break when the data is limited. However, the researchers warn that anyone using these tools must be careful not to rely on shortcuts. If a model is trained to recognize a category because the word "Payroll" appears in the title, it might fail completely when a new report uses different wording. The most honest takeaway from the work is that while computers can help organize these financial records, they are not yet magic. They work best when the data is clear and the clues are genuine, but on small, imperfect datasets, their impressive scores can sometimes hide a lack of true understanding. The researchers suggest that before any government portal deploys such a system, it must be tested rigorously to ensure it is learning the right things, not just copying the labels.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.