GreekBarRetrieval: A Benchmark for Greek Statutory Retrieval
This paper introduces GreekBarRetrieval, a new benchmark for Greek statutory retrieval derived from bar-exam questions, and demonstrates that a ten-round LLM-based query reformulation loop significantly enhances the performance of BM25, enabling it to outperform both vanilla dense retrievers and other advanced retrieval strategies.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of law, finding the right rule to apply to a specific situation is the foundation of justice. When a lawyer or a judge needs to answer a complex question about a crime or a contract, they cannot rely on memory alone; they must locate the exact text of the law that governs the situation. This process, known as statutory retrieval, involves searching through thousands of legal articles to find the few that are truly relevant. The challenge is that people describe real-life events in ordinary, everyday language, while the laws that govern them are written in a specialized, abstract code. A person might say, "He refused to give the medicine," but the law might phrase the same concept as "omission of a duty to act." Bridging this gap between how humans speak and how laws are written is difficult, especially for languages like Greek, which have complex word structures that make simple word-matching unreliable. Without the ability to accurately retrieve these legal texts, artificial intelligence systems cannot provide trustworthy answers or explain their reasoning, which is a critical requirement in legal settings.
Researchers in Greece have tackled this problem by creating a new testing ground called GreekBarRetrieval. They built this benchmark using questions from the actual Greek bar exams, which are the rigorous tests lawyers must pass to practice. In these exams, candidates are presented with a detailed story of a legal case and asked to identify which specific articles of the penal code or other laws apply. The researchers took 283 of these exam questions, along with the full stories behind them, and paired them with a massive library of 6,308 candidate legal articles. The goal was to see if computer systems could look at a question and its story, then successfully find the correct legal articles hidden within that large library. This setup is unique because it tests the system's ability to find the source material before it even attempts to write an answer, separating the skill of searching from the skill of reasoning.
When the researchers first tested standard computer search methods, the results showed a clear divide between two different approaches. One approach, known as sparse retrieval, works by looking for exact word matches between the question and the legal text. The other, called dense retrieval, uses advanced AI to understand the meaning behind the words, even if the specific vocabulary is different. In their initial tests, the meaning-based systems performed far better, successfully finding the relevant articles in the top hundred results about 77 percent of the time. The word-matching systems, by contrast, only succeeded about 36 percent of the time. This confirmed that simply matching words is often not enough when the language of the question and the language of the law are so different.
However, the researchers discovered that the performance of the word-matching systems could be dramatically improved by using a large language model to rewrite the search questions before the computer looked for answers. Instead of searching with the original, messy story full of irrelevant details, the AI was asked to translate the question into the precise, formal language of the law, stripping away the narrative fluff. This simple step of reformulating the query boosted the word-matching system's success rate from 36 percent to 60 percent, effectively closing much of the gap with the more complex meaning-based systems. The study found that this method of rewriting the question was far more effective than other common tricks, such as trying to translate the Greek text into English first or tweaking the internal settings of the search engine.
To push the performance even further, the team introduced a method where the computer searches, reviews what it found, and then searches again based on those results. This iterative process, which mimics how a human lawyer might refine their search strategy after reading a few initial documents, allowed the word-matching system to reach a success rate of 67 percent. While this did not quite surpass the best meaning-based system, which reached 73 percent, it produced a much more organized list of results, placing the most important articles at the very top of the list. This is particularly valuable because legal systems often need to work with a limited number of documents. The researchers noted, however, that this iterative approach comes with a heavy price: it requires the computer to perform many more calculations and take significantly longer to complete the search compared to a single, direct search.
The study concludes that while advanced AI systems that understand meaning are powerful, they are not the only path forward. By using a large language model to translate everyday questions into the formal language of the law, even simpler and faster search tools can become highly effective. This suggests that the key to better legal AI is not just building bigger models, but ensuring that the questions asked are clear and use the correct terminology. The researchers made their data and code public, providing a new standard for testing how well computers can navigate the complex landscape of Greek law, and offering a blueprint for improving legal search tools in other languages as well.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.