Development of an Agentic Retrieval-Augmented Generation System with Hybrid Retrieval for Budget Planning and Validation
This study proposes an Agentic Retrieval-Augmented Generation system with hybrid retrieval for SME budget planning, demonstrating through comparative evaluation that a Routing-Based Agentic workflow offers the optimal balance of retrieval quality and factual grounding for evidence-based validation.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the quiet, often tedious world of organizational management, the budget is more than a simple list of numbers; it is a promise of what an organization intends to do and a record of what it actually spent. For small and medium-sized businesses, keeping this promise requires a rigorous process of checking every proposed cost against official rules, past spending habits, and market realities. This task, known as budget validation, is notoriously difficult because the necessary information is scattered. A manager might need to cross-reference a formal government regulation on a specific item's price, a spreadsheet of historical project costs, and a narrative memo explaining why a particular expense is justified. These documents exist in different formats: some are rigid tables of numbers, while others are flowing text full of exceptions and conditions. When a human tries to hunt through this maze of data to prove that a budget is reasonable, the process is slow, prone to error, and often leaves gaps in the evidence.
To solve this, researchers have turned to a type of artificial intelligence called Retrieval-Augmented Generation. Imagine a system that does not just guess an answer based on its training but first goes out to find the specific facts it needs from a library of documents before speaking. However, standard versions of this technology struggle with financial data because they treat numbers and text as the same thing, often missing the precise values required for accounting. To bridge this gap, a team of researchers from Indonesia and Malaysia developed a specialized system designed to handle both the hard numbers and the soft context of budget planning. They built a tool that can look up exact cost figures in a database while simultaneously reading through narrative guidelines to understand the rules behind those numbers.
The researchers set out to test three different ways of organizing this intelligent system, treating the computer program like a team of workers rather than a single brain. The first approach used a single agent, a lone worker responsible for finding all the information and writing the answer. The second approach introduced a router, a traffic controller that first listened to the user's question to decide which type of problem it was—whether it was a simple question about rules, a request to check a specific budget, or a need for a recommendation. Based on that decision, the router sent the task down a specific path where specialized tools could be used. The third approach was the most complex, using an orchestrator to break a single question into smaller parts, assigning different workers to find different pieces of evidence, and then having a final worker stitch all those pieces together into one answer.
The team tested these three methods using thirty different scenarios that mimicked real-world budgeting tasks. They measured how well each system found the right information, how faithfully it stuck to that information without making things up, and how long it took to give an answer. The results revealed that the single worker was the fastest, completing a task in about twenty-seven seconds, but it was less thorough in finding the necessary evidence. The most complex team of workers took the longest, over eighty-six seconds, and while it found good information, the extra time spent coordinating did not always lead to a better final answer. The middle path, the one with the traffic controller, proved to be the most effective. It took about seventy-eight seconds to respond, but it was the most accurate at finding the right evidence and the most reliable in ensuring that every claim it made was backed by a real document.
This middle approach, which the researchers call Routing-Based Agentic RAG, emerged as the best choice for the job. It achieved the highest scores in finding relevant context and in staying true to the source material, with a faithfulness score of 0.878, meaning it rarely invented facts. It also shared the highest precision score of 0.731 with the most complex system, indicating that the information it retrieved was highly relevant. The researchers found that by separating the tasks based on what the user actually needed, the system could apply the right strategy for each situation. When a user asked for a rule, the system knew to look strictly at the official guidelines. When a user asked to validate a budget item, the system knew to pull up historical data for comparison. When a user asked for a recommendation, the system knew to blend rules with past experiences.
The study concludes that for the critical task of budget validation, where every number must be traceable to a source, the ability to retrieve the correct evidence is more important than speed. While a simple, fast system can answer questions fluently, it risks missing the specific details that prove a budget is reasonable. The routing system, by directing questions to the most appropriate path, ensures that the answers are not just smooth but grounded in fact. The researchers noted that this system is designed to support human decision-makers, not replace them. It provides the evidence and the comparison, but the final judgment on whether a cost is reasonable still rests with the human manager. The work suggests that for organizations dealing with complex financial records, a system that can intelligently sort and retrieve both numbers and text offers a powerful way to bring clarity and accountability to the budgeting process.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.