Comparing RAG and GraphRAG for Page-Level Retrieval Question Answering on a Math Textbook
This paper evaluates RAG and GraphRAG for page-level question answering on a mathematics textbook using a 477-item dataset, finding that embedding-based RAG (particularly with voyage-3-large) significantly outperforms GraphRAG in retrieval accuracy and answer quality while being more efficient, with BM25 serving as a strong baseline and RAG providing proportionally greater benefits for weaker, cost-effective local LLMs.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to solve a tricky math problem, but your brain is a bit like a super-smart robot that has read the entire internet. It knows a lot, but sometimes it makes things up or forgets exactly which page in your specific textbook holds the answer you need. This is the world of "Large Language Models" (LLMs)—AI that can chat and solve problems but sometimes gets lost in its own vast knowledge. To fix this, scientists invented a trick called "Retrieval-Augmented Generation" (RAG). Think of RAG as giving the robot a library card and a librarian. Instead of guessing from memory, the robot asks the librarian, "Hey, which page in this specific book talks about this?" The librarian finds the right page, hands it to the robot, and the robot uses that fresh information to give you a correct answer. But what if the librarian is too eager? What if they grab the whole book, or even the whole library, just to find one sentence? That's where a newer, flashier idea called "GraphRAG" comes in. It tries to map out how every idea connects to every other idea, like a giant spiderweb of knowledge, hoping to find the answer by following the threads. The big question for students and teachers is: When you need to find a specific page in a math book to study for a test, which librarian is better? The one who grabs the exact page quickly, or the one who builds a giant web of connections?
A team of researchers from Carnegie Mellon University and The University of Hong Kong decided to put these two librarians to the test using a real undergraduate math textbook. They created a dataset of 477 questions, each tied to a specific page in the book, and asked different AI systems to find the right page and then answer the question. They compared five different "embedding" models (which are like smart search engines that understand the meaning of words), a classic search method called BM25 (which just looks for matching words, like a very old-school index), and the fancy GraphRAG system.
The results were a bit of a surprise for the tech enthusiasts who love complex webs. The researchers found that the simple, direct approach of embedding-based RAG was the clear winner for finding specific pages. The best model, called "voyage-3-large," managed to find the correct page 99.4% of the time when it was allowed to look at the top 10 results. Even the classic, word-matching BM25 method did a great job, beating out several of the more complex neural models. In contrast, GraphRAG struggled to be precise. While it was good at finding some relevant information, it often grabbed way too much context—about 47,000 words of text compared to the 3,700 words the other methods used. This "information overload" confused the AI, making its final answers slightly worse. The study suggests that for tasks where you need a specific page reference, like studying for a math test, the simple, focused librarian is much better than the one who tries to map the entire universe of connections.
The team also dug into the mistakes. When the best AI model got the wrong page, it was usually a "near miss." About 63% of the time, it grabbed a page from the same chapter, just a few pages away. This is actually helpful! If a student is looking for a proof but gets the page with the theorem statement instead, they are still learning the right material, just in a slightly different order. It's like a tutor saying, "You're looking for the solution, but check the page before it first; it explains the rule you need."
Finally, the researchers tested if these findings held up with cheaper, open-source AI models that don't require expensive cloud servers. They found that while these smaller models were worse at answering questions on their own, they improved dramatically when given the right textbook pages to read. The open-source model saw a 39% jump in quality with the help of retrieval, compared to only a 16% jump for the powerful commercial model. This suggests that for schools or students who can't afford expensive AI services, a simple system that points to the right page in a book can make a local, free AI tutor surprisingly effective.
In the end, the paper concludes that for page-level retrieval in education, less is often more. The fancy, complex GraphRAG system wasn't the right tool for this specific job; it brought too much noise and not enough precision. Instead, a straightforward search that finds the top few pages and lets the AI read them is the most reliable way to build a trustworthy AI tutor that students can actually use to study.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.