Evidence-Aware Abstention in Low-Resource Scientific RAG: A Persian Retrieval-Stage Study

In the modern world of artificial intelligence, computers are increasingly asked to answer complex questions by reading vast libraries of documents. This process, known as retrieval-augmented generation, works by having the computer first search for relevant text and then use that text to formulate an answer. The goal is to stop the machine from relying solely on its internal memory, which can be outdated or incorrect, and instead ground its responses in real, cited evidence. However, a critical problem remains: just because a computer finds a document does not mean that document actually contains the answer. If the evidence is weak, missing, or irrelevant, the system should ideally know to stop and admit it does not know, rather than guessing and providing a false or unsupported response. This is especially difficult in fields like science, where precision is vital, and in languages with fewer digital resources, where the tools to check the quality of information are often less developed.

Researchers Asghar Hussain from The Islamia University of Bahawalpur set out to solve this specific problem for the Persian language within the realm of scientific question-answering. The team focused on a dataset called PersianSciQA, which contains thousands of scientific questions paired with abstracts from research papers. Their work was not about building a full, end-to-end system that writes long essays, but rather about the crucial first step: deciding whether the computer should answer a question at all. They asked a simple but profound question: can a system learn to recognize when the evidence it has found is strong enough to support an answer, and when it is too weak, prompting the system to abstain? By treating this as a two-step process—first finding the information, then deciding if it is good enough—the researchers created a safety mechanism to prevent the computer from confidently stating things it cannot prove.

To begin, the team had to ensure their foundation was solid. They audited the available data, checking for errors and inconsistencies that could skew the results. They discovered that while the data was largely clean, there were significant discrepancies between different versions of the dataset, with thousands of records assigned to different training or testing groups than their source documents suggested. They also found a small number of records where the computer-generated answers were marked as errors. By carefully cleaning and organizing this data, they established a reliable baseline. They then tested a standard search method, known as BM25, which is a classic algorithm for finding text matches. The results showed that the search method worked exceptionally well for questions where the connection to the source text was very strong, successfully finding the correct document in the top ten results for nearly 98 percent of those cases. However, for questions with weaker connections, the success rate dropped to about 78 percent, highlighting the gap between finding a related document and finding the right one.

With the search baseline established, the researchers moved to the core of their study: building a lightweight decision-maker. This system was designed to look at the search results and decide whether to answer or to stay silent. Instead of relying on complex, heavy-duty models that require massive computing power, they used a simple statistical model that analyzed specific clues. These clues included the scores the search engine gave to the documents, the number of words shared between the question and the found text, and the language composition of the documents. The goal was to see if adding these extra layers of information could help the system distinguish between a solid answer and a shaky guess. They tested this on a set of questions where the answerability was known, measuring how often the system would incorrectly try to answer a question that lacked sufficient evidence.

The findings revealed that the system could indeed learn to be more cautious and accurate. When the researchers used only the raw search scores to make decisions, the system still attempted to answer questions it shouldn't, resulting in an unsupported answer rate of about 40 percent when the system was set to answer roughly one-quarter of the questions. However, when they added the lexical evidence features—essentially looking at how well the words in the question matched the words in the found text—the performance improved significantly. With this enhanced model, the rate of unsupported answers dropped to about 31 percent at the same level of activity. This improvement was statistically significant, meaning it was unlikely to be a random fluke. The researchers found that the word-matching features were the primary driver of this success, while other language-based features provided only a minor boost.

Crucially, the study also clarified what it did not achieve. The researchers were transparent that their work was limited to the retrieval stage; they did not have access to the large language models required to generate full, human-like answers in this specific environment. Therefore, their results measure the quality of the decision to answer, not the factual accuracy of a generated paragraph. They also noted that their language analysis was not a true test of multilingual capabilities, as the data was overwhelmingly in Persian, with very little presence of other languages. Despite these boundaries, the work provides a clear, reproducible framework for the next stage of research. It demonstrates that even in low-resource scientific settings, a system can be taught to recognize the limits of its own evidence. By learning to abstain when the evidence is weak, the system becomes more reliable, ensuring that when it does speak, it is backed by the documents it has found. This approach offers a practical path forward for building more trustworthy artificial intelligence in languages and fields where data is scarce.