NormDef-FR: an annotated corpus of normative definitions for the automatic processing of French legal codes

Law is often imagined as a vast library of rules, but for a computer, it is a chaotic sea of text. In France, the legal system is built on codes—massive collections of laws where every word carries specific weight. Within these thousands of pages, lawmakers frequently pause to define their own terms. They might write, "By personal data, we mean any information relating to an identified person," or "For the purpose of this article, a vehicle includes a bicycle." These are not just helpful explanations; they are the legal engine that determines how rights and obligations work. Without these precise definitions, the rest of the law cannot function correctly. However, finding these definitions is difficult because they are scattered across more than 161,000 active articles, buried in different codes ranging from health to taxation, with no organized map to guide a search.

This lack of structure creates a significant problem for technology. If a lawyer or a digital system wants to know exactly what a specific term means in the eyes of the law, they often have to read through countless pages manually, with no guarantee they have found every instance. Natural language processing, the field of computer science that teaches machines to understand human language, has made great strides in analyzing legal texts, but it has struggled with this specific task. Most existing tools are designed for English or for general definitions found in dictionaries, not for the strict, binding definitions found in French legislation. To bridge this gap, researchers have created a new resource called NormDef-FR, a carefully curated collection of these legal definitions designed to teach computers how to find and understand them automatically.

The researchers, working at Abdelmalek Essaadi University in Morocco, approached the problem by building a bridge between human expertise and machine automation. They started by gathering the current, active laws from the official French legal database. Instead of trying to teach a computer to read the entire library at once, they first used simple, rule-based patterns to spot likely candidates. They looked for specific phrases that French law uses to introduce definitions, such as "is defined as" or "is considered to be," or the use of a colon to separate a term from its meaning. This initial sweep generated a list of potential definitions, but it was far from perfect. Many of these candidates were false alarms, capturing sentences that looked like definitions but were actually something else, like a list of documents to submit or a description of a legal effect.

To turn this rough list into a reliable training tool, the team brought in human experts. They manually reviewed thousands of these candidates, deciding which ones were true, usable legal definitions and which ones should be discarded. They also corrected the boundaries of the text, ensuring that the "term" being defined and the "definition" itself were cut out exactly right. This process was rigorous. The researchers had two independent experts label a sample of the data to ensure they agreed on what counted as a definition. The result was a high level of agreement, confirming that the rules for identifying these definitions were clear and consistent. The final product, version 1.0.0 of NormDef-FR, contains 693 verified definitions extracted from 363 different articles across 50 distinct legal codes. Each entry links the defined term to its meaning, includes metadata about where it came from, and notes the specific type of definition it is.

With this gold-standard collection in hand, the researchers tested how well computers could learn from it. They set up three different challenges to see where the machines succeeded and where they stumbled. The first task was simple extraction: could a system spot the definition and pull out the term and the meaning? The second task was broader: could a system look at an entire article and decide if it contained a definition at all? The third task was the most difficult: could a system take a candidate definition generated by a computer and decide if it was good enough to keep, or if it was a mistake that should be thrown away?

The results revealed a clear hierarchy of difficulty. The machines were surprisingly good at the second task. When asked to identify which articles contained definitions, the models performed with near-perfect accuracy, especially when they were trained on a set of "true negatives"—articles that had been manually checked to ensure they contained no definitions at all. This finding was crucial because it showed that the earlier method of assuming any article not in the definition list was a "negative" was flawed; many of those articles actually contained hidden definitions. Once the researchers cleaned up the negative examples, the computers could easily tell the difference between a definition-heavy article and a regular one.

However, the third task proved to be the real bottleneck. When the computers had to validate specific term-definition pairs, their performance dropped significantly. While they could easily spot the general area where a definition might be, they struggled to determine if the specific pair they had pulled out was legally valid. They often mistook a list of requirements or a procedural instruction for a definition. The best models managed to get about 71 percent of these difficult decisions right, a far cry from the near-perfect scores seen in the article detection task. This suggests that while computers can find the right neighborhood, they still need human help to verify the exact house.

The study also highlighted a surprising limitation of advanced artificial intelligence in this specific context. The researchers tested a sophisticated language model, similar to those used for general conversation, against a simpler, more traditional method based on word patterns. For the difficult task of validating candidates, the complex model did not outperform the simpler one. In fact, the simpler method was more stable and reliable. This indicates that for highly specialized legal texts, where the rules are rigid and the patterns are specific, a straightforward approach that relies on clear linguistic markers can be more effective than a complex system trying to "understand" the text in a general sense.

The creation of NormDef-FR is more than just a dataset; it is a demonstration of how to build reliable resources for legal technology. The researchers showed that simply assuming an absence of data is not enough; you must actively verify what is not there. By manually checking the negative examples, they prevented the system from learning the wrong lessons. This approach ensures that the resource is robust and trustworthy for future developers. The dataset is now available to the public, complete with documentation and scripts that allow others to reproduce the work. It serves as a foundation for building better tools for legal research, helping to automate the tedious work of finding definitions so that lawyers and systems can focus on interpreting the law rather than hunting for its meaning. While the work is not finished—future versions will need to cover more types of laws and track how definitions change over time—it provides a solid starting point for bringing clarity to the complex world of French legal codes.