AIL: Integrating LLM Capacity Awareness, Legal Intuition, and Lexis Understanding into Legal Information Extraction

In the world of law, a court judgment is more than just a final decision; it is a complex story of facts, arguments, and reasoning that explains why a specific outcome was reached. For decades, computers have struggled to read these documents, often getting lost in the dense, specialized language that lawyers and judges use. While artificial intelligence has become very good at reading ordinary text, legal writing presents a unique challenge. It relies on centuries of tradition, specific terms that mean one thing in a courtroom and something else entirely in daily conversation, and a logic that shifts from case to case. To build useful tools that can help lawyers search for precedents or predict outcomes, researchers need to turn these unstructured stories into organized, machine-readable data. The question has always been how to do this without losing the nuance that makes the law work.

A team of researchers has tackled this problem by developing a new way to work with artificial intelligence, one that treats human legal experts not as mere supervisors, but as essential partners in the process. Instead of trying to force a computer to do everything alone, they created a system where human intuition and deep knowledge of legal language guide the machine at every step. They tested this approach on the court judgments of Macao, a region with a unique legal system that blends Chinese and Portuguese traditions, resulting in documents that are notoriously difficult to process. The result is a highly accurate method for turning messy court records into clean, structured information, proving that the best way to teach a computer about the law is to let a human lawyer show it the way.

The researchers began by recognizing a fundamental limitation in current artificial intelligence. While large language models can read vast amounts of text, they often miss the subtle, hidden meanings that a human lawyer would catch immediately. A computer might see a list of facts, but it struggles to understand which facts actually matter to the final verdict or how a judge's reasoning connects to a specific law. The team argued that trying to automate the entire extraction process, letting the computer guess what to pull out of a document, leads to errors. Instead, they proposed a framework called AIL, which stands for Awareness of LLM capacity, Legal intuition, and Legal lexis understanding. This approach acknowledges that computers have limits, that human experience is needed to spot what is important, and that the specialized vocabulary of the law must be explained clearly to the machine.

To put this idea into practice, the team focused on the court judgments of Macao. These documents are a perfect test case because they are long, often lack a standard format, and use a mix of languages and legal concepts that are unique to the region. Some judgments run to over a thousand pages, and the writing style can vary wildly from one judge to another. The researchers gathered thousands of these public documents and set out to build a system that could read them and pull out specific details, such as the date of the case, the charges involved, the arguments made by both sides, and the final ruling. They did not simply feed the documents to a computer and hope for the best. Instead, they designed a step-by-step workflow where human experts and the computer worked together in a cycle of testing and refining.

In the first stage, the human experts defined what information needed to be extracted, creating a list of fifteen specific fields to fill for every case. They then wrote instructions, known as prompts, to tell the computer how to find this information. However, they knew the computer would make mistakes, so they built in a rigorous checking process. A group of legal experts, including judges and prosecutors, reviewed the computer's first attempts. They looked for errors, such as missing details, made-up facts, or information that was pulled from the wrong part of the text. When the computer failed, the humans did not just blame the machine; they adjusted the instructions. They explained the confusing legal terms more clearly, gave the computer examples of what to look for, and told it exactly where in the document to search. This cycle of testing, finding errors, and refining the instructions happened over and over until the computer's performance improved dramatically.

The results of this collaboration were striking. After refining the system through this human-led process, the researchers achieved an overall accuracy rate of 98.6 percent in extracting information from the Macao judgments. This means that in nearly every single case, the computer correctly identified the key details the humans were looking for. Even for the most difficult fields, such as identifying the specific procedural measures taken during a trial or summarizing the complex facts of a case, the accuracy remained well above 90 percent. The team also found that the structured data they created was immediately useful. They used the new dataset to train other artificial intelligence models to predict the outcome of future cases and to classify the type of legal dispute. These downstream tasks showed that the information extracted was not just accurate, but also rich enough to support advanced legal analysis.

The study also revealed that the specialized legal models, which are often trained on data from mainland China, did not perform as well as general-purpose models when applied to Macao's unique legal system. This suggests that a model trained on one legal culture does not automatically understand another, even if they share a language. The success of the project came from the human experts who understood the specific quirks of Macao's laws and could teach the computer how to navigate them. By combining the computer's speed and ability to process large amounts of text with the human expert's deep understanding of legal logic and language, the researchers created a system that is both efficient and reliable.

This work offers a new path forward for legal technology. It suggests that the future of legal artificial intelligence does not lie in trying to build a machine that knows everything on its own, but in building systems where humans and machines work together. The human provides the intuition and the specialized knowledge, while the machine handles the heavy lifting of reading and organizing the data. By following this collaborative approach, researchers can create high-quality datasets for any legal system, no matter how complex or unique. The study demonstrates that when we respect the limits of artificial intelligence and bring human expertise into the loop, we can unlock the vast, untapped information hidden within court judgments, making the law more accessible and understandable for everyone.