On Measuring Semantic Preservation in Legal Ontology Learning
This paper proposes and validates a novel evaluation framework that quantifies semantic loss in legal ontology learning by comparing LLM task performance on source documents versus structured representations, revealing that semantic preservation varies significantly depending on the specific pairing of language models and ontology learning methods.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to teach a robot to understand a complex legal contract. You have two choices: you can feed the robot the original, messy, human-written text, or you can translate that text into a super-organized, structured map called an "ontology." Think of an ontology like a giant, perfectly labeled filing cabinet where every concept is a drawer and every relationship is a clear label. The idea is that this structured map should help the robot reason better than it could with the messy text. But here's the catch: when you take a juicy, nuanced story and force it into a rigid filing cabinet, you might accidentally throw away the most important flavors. This is the problem of "semantic loss"—losing the meaning while keeping the structure. Scientists have been great at checking if the filing cabinet is built correctly (are the drawers labeled right?), but they haven't had a good way to check if the contents of the drawers are still tasty enough to solve the actual problem. This paper steps into that gap, asking a simple but vital question: "Does turning our text into a structured map actually help the robot, or does it just make the robot dumber by stripping away the details it needs?"
The authors of this paper, Albert Sadowski and Jarosław A. Chudziak, decided to test this idea using a very specific and tricky playground: legal merger agreements. These are the massive, complicated contracts companies sign when they buy each other, filled with fine print and subtle legal conditions. They wanted to see if converting these contracts into ontologies helped Large Language Models (LLMs)—the super-smart AI chatbots we use today—answer questions about them.
To do this, they set up a clever experiment. First, they let six different AI models read the original, messy contracts and answer 34 different types of legal questions. This gave them a "baseline" score, representing the maximum amount of information the AI could understand from the original text. Then, they took those same contracts and ran them through three different "ontology learning" methods (LLMs4OL, NeOn-GPT, and NeOn-CoT) to turn them into structured maps. Finally, they asked the same AI models to answer the same questions, but this time, they could only look at the structured maps, not the original text.
The results were a bit of a shock. The paper found that in almost every case, the AI models got worse when they had to use the structured maps instead of the original text. It's like taking a delicious, complex stew, draining all the broth and spices to make a neat list of ingredients, and then asking a chef to recreate the taste of the stew using only that list. The "semantic loss" was real and measurable. On average, the AI models lost between 8.4 and 27.4 percentage points of accuracy depending on which method was used. The method called LLMs4OL was the worst offender, causing the most information to vanish, while NeOn-GPT was the "least bad," preserving the most meaning.
However, the story gets even more interesting when you look at which AI model was doing the reading. The paper discovered that the loss wasn't the same for everyone; it depended heavily on the "personality" of the AI model and which method was used to build the map. For example, one model called Gemini worked surprisingly well with a specific method called NeOn-CoT, keeping about 86-88% of the original meaning for difficult tasks. But if you paired that same method with a different model, the performance would tank. This suggests there is no single "best" way to build these legal maps; instead, you have to find the perfect match between the AI brain and the map-making tool.
The authors also noticed that the type of question mattered a lot. If the question was simple, like "Is this a yes or no?" the structured maps worked almost perfectly, losing almost no information. But if the question required understanding subtle legal nuances—like the difference between "more likely than not" and "reasonably likely"—the structured maps failed miserably, often losing up to 65% of the meaning. The rigid structure of the ontology couldn't capture the delicate shades of probability that lawyers rely on.
In the end, the paper doesn't say we should stop using ontologies. Instead, it warns us not to assume that making things structured automatically makes them better. The authors suggest that before we trust a structured map to help an AI make legal decisions, we need to test it first to see if it actually preserves the meaning we care about. They found that for complex legal reasoning, the process of turning text into a structured map often strips away the very details that make the answer correct. So, while ontologies are still useful for organizing data, we need to be careful not to throw the baby out with the bathwater, especially when the "baby" is a subtle legal argument that could change the outcome of a billion-dollar deal.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.