Where A Small Language Model Helps in Invoice Categorisation, Understood Through Embedding Geometry

Every business that buys or sells goods must keep a precise record of every transaction. When a company receives an invoice, a human accountant must decide which specific category that expense belongs to, a task known as assigning a General Ledger code. This is not a simple sorting job; it requires professional judgment to understand whether a purchase is for daily operations, a capital investment, or a personal expense for a business owner. Getting this right is the foundation of accurate financial reporting and tax compliance. If these codes are wrong, small businesses face unnecessary fines, and governments lose billions in tax revenue. For years, the hope has been that artificial intelligence could automate this difficult work, but the technology has often struggled to match the nuance of a trained human accountant.

Researchers at the University of Bristol and System Holdings Limited recently explored a different path. Instead of using massive, complex AI systems that require vast amounts of computing power and raise concerns about data privacy, they tested whether smaller, specialized language models could learn to do this job effectively. These smaller models can be run on a single computer, keeping sensitive financial data secure within a company's own walls. The team wanted to understand not just if these models worked, but how they thought. They looked at the mathematical "shape" of the information inside the computer, examining how the model grouped similar invoices together in its internal memory. They discovered that the way these smaller models organize information is surprisingly efficient and that this organization holds the key to their success.

The researchers worked with a real-world dataset of over two thousand invoices from seven different clients of a UK accounting firm. Each invoice had already been correctly categorized by professional accountants. The team fed this data into two different types of AI models: a smaller, efficient model called SBERT and a larger, more complex one called DeBERTa. They tested the models with different versions of the invoice text. One version included only the vendor name, date, and price. Another included the full list of items purchased. A third version added explicit labels, such as "Vendor Name:" or "Price:", to make the text look more structured for a human reader.

The results revealed a counterintuitive truth about how machines read. The version of the text that was easiest for a human to read, with its clear labels and structured format, actually made it harder for the AI to distinguish between different types of invoices. When the computer saw the words "Vendor Name" and "Price" repeated in every single document, the internal mathematical representation of those documents became too similar, blurring the lines between them. The model performed best when it was given the raw, unstructured text of the invoice, just as a human accountant might glance at a messy receipt. The smaller model, SBERT, proved to be the superior choice. When trained on just a few hundred examples, it achieved a high level of accuracy, correctly categorizing invoices far better than a much larger model or a standard "zero-shot" AI that had not been trained on this specific task.

A deeper look into the computer's internal memory explained why the smaller model succeeded where the larger one struggled. The researchers found that the larger model's internal representation of the data was "collapsed." As the text grew longer and more detailed, the model's ability to distinguish between different meanings faded, and it began to rely mostly on the length of the text rather than its content. In contrast, the smaller model maintained a rich, multi-dimensional space where different types of invoices occupied distinct regions. Even before being trained on the specific task, the model naturally grouped invoices by the company that issued them, showing that it understood the source of the document. More importantly, it could separate different financial categories even when the text was similar, provided it had seen enough examples.

This ability to generalize was critical when the team tested the models on a completely new client whose data had never been seen before. The larger model struggled significantly, requiring hundreds of new examples to start working correctly. The smaller model, however, adapted much faster, reaching a high level of accuracy with only about one hundred new invoices. This suggests that the smaller model had learned a more robust way of understanding the meaning of the text, rather than just memorizing patterns. It could transfer what it learned from one business to another, recognizing that a "motor repair" for one client was the same category as a "motor repair" for a new client, regardless of the specific vendor.

The study concludes that for tasks like invoice categorization, a smaller, in-house model is often a better tool than a massive, generic one. It offers a practical balance of cost, security, and performance. By running on a single computer, it keeps sensitive financial data private, and by being smaller, it is cheaper to operate. The research also showed that adding extra structure to the text does not help the AI; in fact, the raw text is often more effective. This finding simplifies the process for businesses, as they do not need to spend time and money reformatting their invoices before feeding them to the system. The work demonstrates that with the right approach, artificial intelligence can handle the nuanced, skilled judgment required in accounting, provided the system is designed to understand the geometry of the data rather than just the volume of the text.