Evaluating Retrieval Accuracy and Latency for Enterprise RAG Systems on FiQA
This study evaluates the retrieval accuracy and latency of TF-IDF, BM25, a quantized MiniLM dense retriever, and their reciprocal rank fusion on the FiQA benchmark, finding that while the dense model significantly outperforms sparse methods, the fusion approach yields only a marginal and statistically insignificant improvement over MiniLM alone.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern world of artificial intelligence, a common challenge is helping a computer answer questions using a vast library of documents it has never seen before. Imagine a system that can read a million pages of text and then discuss them with you. For this to work well, the system needs a librarian that can find the right pages instantly. If the librarian brings back the wrong pages, the computer will give a wrong answer. If the librarian takes too long to find the right pages, the conversation stalls and becomes useless. This balance between finding the correct information quickly and finding the correct information accurately is the central problem researchers face when building these intelligent systems. They must decide how to search through the library: should they look for exact word matches, like a traditional index, or should they look for the general meaning of the words, like a human understanding the context?
A recent study set out to test these different search methods on a specific collection of financial forum questions and answers. The researchers wanted to see which approach worked best for an enterprise system, where speed and accuracy are both critical. They compared four different ways of searching. The first two methods relied on matching words directly. One was a basic count of how often words appeared, while the other was a more refined version that understood how common or rare a word was across the entire collection. The third method used a modern technique that turned every question and every document into a mathematical representation of its meaning, allowing the system to find answers even if the words were different. The fourth method was a hybrid approach that tried to combine the results of the word-matching search and the meaning-based search to get the best of both worlds.
The team ran these four methods against a standard test set containing nearly sixty thousand documents and six hundred and forty-eight questions. They measured two things: how often the correct answer appeared near the top of the list, and how long the computer took to produce that list. The results showed a clear trade-off. The method that relied purely on matching words was the fastest, finding a relevant document in the top five results about forty percent of the time, but it took less than six milliseconds to do so. The method that understood meaning was significantly more accurate, finding a relevant document in the top five results about fifty-four percent of the time, but it took nearly eighteen milliseconds to run. This slower speed is because the computer had to translate the question into its complex mathematical form before it could search.
When the researchers combined the two methods, hoping to get the high accuracy of the meaning-based search with the speed of the word-matching search, the results were surprising. The combined method did not become significantly more accurate than the meaning-based search alone. In fact, the improvement in accuracy was so small that it could easily have been due to random chance. However, the combined method did take longer, requiring nearly twenty-four milliseconds to complete a search. This happened because the system had to run both searches separately and then merge the lists, effectively doubling the work for a gain that was barely measurable. The study also found that the two methods often found different correct answers. The word-matching search found relevant documents that the meaning-based search missed, and vice versa. While this showed that the two methods complemented each other, the simple way they were combined did not successfully keep the best answers from both lists in the final top five.
The researchers emphasized that these findings apply specifically to the speed of the search itself, not to the final answer the computer generates. A fast search that finds the right document is only half the battle; the computer must still read that document and write a good answer. The study also noted that the test data came from public financial forums, which might not perfectly represent the complex, restricted documents found inside a real company. Furthermore, the computer used for the test was a standard processor, not a specialized high-speed machine, so the times measured reflect a specific setup rather than a universal limit.
Ultimately, the work suggests that for a system needing to be both fast and accurate, simply adding a second search method does not automatically improve the result. The most effective approach depends on the specific needs of the user. If speed is the absolute priority, a simpler word-matching search might be sufficient. If accuracy is more important, the more complex meaning-based search is worth the extra time. But combining them requires careful tuning, as the extra time spent merging lists may not be worth the tiny gain in finding the right document. The study provides a clear, measured look at these choices, showing that in the world of artificial intelligence, adding more complexity does not always lead to a better outcome.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.