When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning
This paper introduces a benchmark for temporal applicable-law determination and reveals that large language models exhibit a strong bias toward applying the most recent law due to reinforcement learning-induced reasoning convergence, leading to a counterintuitive inverse relationship where models with stronger general reasoning capabilities perform worse on temporally grounded legal tasks.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The law is not a static monument; it is a living landscape that changes over time. New rules are written, old ones are rewritten, and the specific version of a law that applies to a situation depends entirely on when that situation happened. A contract signed today might be governed by one set of rules, while a similar contract signed five years ago falls under a different set. This principle, known as non-retroactivity, is a cornerstone of legal fairness: people should be judged by the laws that existed when they acted, not by laws created later. For artificial intelligence to be useful in the legal world, it must understand this timeline. It cannot simply know what the law says; it must know which version of the law was in force at the exact moment a legal event occurred. If a computer system gets the timing wrong, it applies the wrong rules, leading to fundamentally incorrect conclusions, regardless of how smart its analysis of the facts might be.
Researchers have long hoped that large language models, the powerful AI systems capable of answering complex questions and writing text, could automate parts of this legal reasoning. These models have shown promise in predicting court outcomes and analyzing contracts. However, a critical question remained unanswered: can these models reliably figure out which version of a law applies to a specific case? A new study by a team of researchers from institutions in Japan and China investigates this precise problem. They built a specialized test to see if these advanced AI systems could correctly identify the right version of a law based on the dates of the events in a case. The results revealed a surprising and systematic flaw. The most advanced models, despite their ability to reason through complex problems, consistently failed when the correct answer was an older version of the law. Instead, they overwhelmingly chose the newest, most recently enacted version, even when the facts of the case clearly pointed to the past.
To understand the depth of this issue, the researchers constructed a benchmark using thousands of real civil court judgments from China. They selected cases where the relevant events happened before a major legal update, specifically before the implementation of the Civil Code in 2021. In these cases, the correct legal answer required citing the older, standalone laws that were in effect at the time, such as the old Property Law or Contract Law. The team then asked various state-of-the-art AI models to identify the correct version of the law for each case. The results were starkly uneven. When the case involved the current law, the models performed quite well, often getting the answer right more than 70 percent of the time. But when the case required applying an older law, their performance collapsed. On these older cases, most models scored below 15 percent, with some dropping to near zero. The models were not randomly guessing; they were consistently making the same specific mistake. They identified the correct type of law but applied the wrong version, always reaching for the most recent one.
The researchers then set out to diagnose why this was happening. They first considered whether the models simply lacked the necessary knowledge. Perhaps the AI had never "read" the old laws or did not understand the rules about how laws change over time. To test this, they asked the models to recite the text of the old laws and to answer multiple-choice questions about the rules of legal timing. The models performed exceptionally well on these tests, proving they possessed the facts and the rules. They knew the old laws existed and understood the principles of non-retroactivity. The failure was not a lack of memory or a gap in knowledge. The problem lay in how the models used that knowledge when faced with a real case.
Further investigation revealed that the issue was tied to the way these models are trained to reason. Modern AI systems are often fine-tuned using a process that encourages them to think through problems step-by-step, producing a chain of reasoning before giving an answer. This training is designed to make the models better at solving difficult logic puzzles and math problems. However, the researchers found that this very strength became a weakness in the legal context. The models with the strongest general reasoning abilities were actually the ones that performed the worst on the timing task. The training process seemed to have narrowed the models' thinking, causing them to converge on a single, dominant path: applying the current law. It was as if the models had learned that the most recent information is the most important, and they became so focused on this pattern that they stopped exploring other possibilities, even when the facts of the case demanded it.
The study suggests that the models were not failing because they were too dumb to understand the timeline, but because they were too efficient at following a specific, learned habit. When the researchers provided the models with a simple hint reminding them to consider the date of the event and the specific rules of legal timing, the performance improved dramatically. This indicated that the models had the capacity to get it right, but their default mode of operation was misaligned with the specific requirements of legal reasoning. Without a nudge to look at the timeline, their powerful reasoning engines would simply reinforce the bias toward the newest law.
This discovery highlights a subtle but dangerous limitation in current artificial intelligence. It shows that making a model smarter at general reasoning does not automatically make it better at every task. In fact, for tasks that require careful attention to specific constraints, like the timing of a law, the very mechanisms that make these models powerful can cause them to ignore crucial details. The researchers found that when the models were forced to explore different reasoning paths rather than sticking to their default habit, they could successfully identify the correct, older version of the law. The study concludes that for AI to be truly reliable in the legal field, it must be trained not just to reason well, but to respect the specific, often counter-intuitive rules of the domain, such as the fact that the newest law is not always the right one.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.