Annotated Surrogate Retrieval for Polish Statutory Law
This paper introduces and evaluates three retrieval methods for Polish statutory law that utilize language-model-generated document surrogates, demonstrating that the ASCR-H cascade with reranking significantly outperforms existing baselines in top-rank accuracy on legal examination questions, while a more cost-efficient alternative (DTF) achieves comparable performance at the head of the ranking with substantially lower latency and cost.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the vast library of a nation's laws, finding the single paragraph that answers a specific legal question is a task of extreme precision. Imagine a library containing over eighty thousand distinct articles, each a tiny piece of a massive legal puzzle. When a lawyer or a student asks a question, the answer is not a general summary or a collection of related ideas; it is one specific sentence buried somewhere in that mountain of text. If a computer system retrieves the correct article but places it tenth on a list of suggestions, a human reader scanning the top results might miss it entirely, rendering the search useless. This is the core challenge of statutory retrieval: the system must not just find the right document, it must place it at the very top of the list. This problem is made even harder by the nature of the language itself. In Polish, words change their form depending on their role in a sentence, meaning a question and its legal answer might share the same meaning but look completely different on the surface.
Researchers set out to solve this specific problem for Polish law by testing three different strategies to help computers find the right legal article. They built their test on real questions from the 2024 and 2025 entrance examinations for legal trainees in Poland. These are high-stakes questions where the correct answer is a single article of law. The team created a system that does not just read the raw text of the laws but also attaches a "surrogate" to each article. Think of this surrogate as a set of helpful notes written by a smart assistant for every single law article before the search even begins. These notes include a summary, a list of themes, and a set of hypothetical questions that the article could answer. When a user asks a question, the computer can match it against these pre-written notes, which often bridge the gap between how a person asks a question and how a law is written.
The study tested three distinct approaches to using these notes. The first approach, which the researchers call ASCR-H, acts like a careful editor. It first uses the notes to narrow down the search to the most promising laws, then uses a dense search method to find other possibilities, and finally, it uses a powerful language model to re-rank the top candidates. This method is slow and expensive because it asks the computer to think deeply about the list multiple times. The second approach, DTF, is much faster and cheaper. It skips the deep thinking and re-ranking entirely. Instead, it combines the results from three different search methods—one that looks at the notes, one that looks at the raw text, and one that looks at the structure of the law—and merges them into a single list using a fixed, mathematical rule. The third approach is a middle ground that drops the re-ranking step but keeps the initial filtering.
The results revealed a clear trade-off between speed and precision at the very top of the list. The careful editor approach, ASCR-H, was the most successful at placing the correct article at the number one spot. It succeeded 72.3% of the time, significantly outperforming all other methods that did not already know the answer in advance. The fast approach, DTF, placed the correct article at the top only 51.9% of the time, a gap of more than twenty points. However, the story changes if you look further down the list. While the careful editor was better at the very top, the fast approach caught up quickly. By the time you look at the top twenty results, the two methods were statistically indistinguishable, and the fast method actually started to perform slightly better at deeper levels. This suggests that if a human is willing to scan a longer list, the cheaper, faster system is just as effective.
The researchers also discovered what did not work. They tested several common tricks used in other search systems, such as rewriting the user's question to make it clearer or using a technique called pseudo-relevance feedback, which tries to improve the search by pretending the top results are correct and using them to refine the query. None of these tricks helped; in fact, some made the results worse. They also found that simply using a dictionary to strip away word endings, a common fix for languages with complex grammar, provided no benefit because the exam questions were already written in language very close to the laws themselves.
Perhaps the most surprising finding was that getting the correct article to the top of the list did not necessarily make the final answer better. The researchers tested whether a computer program, acting as a judge, could choose the right multiple-choice answer based on the retrieved articles. They found that the computer was already so good at answering these questions from its own internal knowledge that it did not matter much whether the correct article was ranked first or fifth. The system was saturated; it knew the answer without needing the text. This means that for this specific type of exam, the massive effort required to get the perfect ranking at the very top might not be worth the cost, because the final result is the same. The study concludes that for legal search, the best tool depends entirely on the goal: if you need the absolute best chance of seeing the right law immediately, the expensive, careful method is superior. But if you need to cover a wide range of possibilities quickly and cheaply, the fast, simple method is just as good, provided you are willing to look a little deeper into the results.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.