💬 NLP

By Their Fruits You Will Know Them: Comparing Formalizations of Law by the Decisions They Encode

This paper introduces a method for systematically comparing legal formalizations by enumerating and verbalizing edge cases where they disagree, revealing that structural similarity does not guarantee behavioral consistency and that LLM-generated formalizations can produce distinct, legally significant divergences.

Original authors: Julius Vernie, Matthias Grabmair

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Julius Vernie, Matthias Grabmair

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very complicated recipe for a legal rule, written in a language that is slightly vague. You ask nine different expert chefs (the AI models) to translate this recipe into a strict, step-by-step flowchart that a robot can follow.

The problem is: even if the chefs use the same ingredients and write the steps in a similar order, they might secretly interpret a few key instructions differently. One chef might think "add salt" means "a pinch," while another thinks it means "a cup." If you just look at their flowcharts side-by-side, they might look 90% identical. But if you feed the robot a specific, tricky ingredient combination, one chef's robot might say "Serve this dish," while the other's says "Throw this away."

This paper is about a new way to find those hidden disagreements before anyone gets hurt.

The Core Problem: "By Their Fruits You Will Know Them"

The title comes from an old saying: you can't just judge a tree by how it looks; you have to taste the fruit. In the world of AI and law, the "tree" is the formal rule (the flowchart), and the "fruit" is the decision it makes on a specific case.

The authors argue that simply comparing the structure of two AI-generated legal rules isn't enough. Two rules can look almost identical on paper but produce completely opposite results in real-life scenarios. They wanted a way to systematically find the "edge cases"—the weird, specific situations where two different AI interpretations of the same law clash.

The Method: A Three-Step Detective Process

The researchers built a pipeline to catch these disagreements, which they call "By Their Fruits." Here is how it works, using a simple analogy:

1. The Translation (Formalization)
First, they took 10 different EU laws (like the GDPR or the AI Act) and asked 9 different top-tier AI models to turn each law into a "decision tree." Think of this as asking nine different architects to draw a blueprint for the same house based on a vague description.

2. The Map-Making (Matching)
Next, they had to compare the blueprints. But the architects used different labels for the same rooms. One called it "The Kitchen," another called it "The Cooking Area." The researchers used an AI to act as a translator, matching up the nodes (the steps) in the different trees that represent the same legal concept. They built a "shared map" of where the trees overlap.

3. The Stress Test (SAT Solver & Edge Cases)
This is the magic part. Once they had the shared map, they used a mathematical tool (a SAT solver) to act like a stress-test engineer. They asked: "What is the absolute smallest, weirdest set of facts that would make these two blueprints give different answers?"

The tool found these "edge cases"—the specific combinations of facts where the rules diverge. For example, it might find that the rules only disagree if a robot is used by the police and the data was collected legally but not for a specific medical purpose.

4. The Storytelling (Verbalization)
Finally, the raw math (which looks like a list of "True/False" variables) is boring and hard for humans to understand. So, they used another AI to turn these math problems into short, realistic stories.

Instead of: "Input A=True, Input B=False, Input C=True."
They get: "A company sells a facial recognition system to the police to identify race. The data was collected legally, but the system is not used for filtering."

What They Found

When they ran this experiment on 10 EU laws and 9 AI models, they discovered some surprising things:

Looks Can Be Deceiving: There was almost no connection between how similar the rules looked structurally and how often they disagreed on the outcome. Two models could have 95% matching structures but still disagree on 50% of the edge cases.
Real Legal Drama: The "fruits" (the edge cases) revealed genuine, deep-seated disagreements in how the law should be interpreted.
- Example: On the topic of AI that categorizes people by race for police work, the AI models split evenly. Half said it's strictly banned; the other half said it's allowed if the data was collected legally. This wasn't a computer error; it mirrored a real, ongoing debate among human legal experts about what the law actually means.
Hidden Errors: The method also caught clear mistakes. One AI model accidentally added an extra rule that made a legal condition much harder to meet than the law intended. The "fruit" (the edge case) exposed this error immediately.

The Bottom Line

The paper concludes that to truly understand if an AI is interpreting the law correctly, you can't just look at its code or its flowchart. You have to test it against specific, tricky scenarios.

Their method acts like a spotlight, shining a light on the exact moments where different AI interpretations of the law collide. It turns abstract mathematical differences into concrete stories that human lawyers can read, understand, and debate. It doesn't tell you which AI is "right" (that's a job for human experts), but it ensures you know exactly where and why they are disagreeing.

Technical Summary: By Their Fruits You Will Know Them

Problem Statement

The formalization of legal provisions into machine-readable structures promises automated legal reasoning and machine-accessible law. With the advent of Large Language Models (LLMs), it has become feasible to generate these formalizations directly from statutory text. However, any formalization inherently involves implicit interpretive choices. When an LLM acts as the author, these choices are opaque, and the consequences of divergent interpretations are difficult to anticipate. A formalization that appears structurally sound may yield unjust or incorrect verdicts in edge cases not envisioned by its drafter.

The core challenge addressed is how to systematically compare multiple formalizations of the same legal provision to identify where they diverge in their logical outcomes, rather than merely comparing their structural similarity. Existing methods often rely on structural matching or comparison against a "gold standard," but the authors argue that comparing formalizations against one another without assuming a ground truth is necessary to uncover genuine interpretive ambiguities and errors.

Methodology

The authors propose a four-stage pipeline to compare LLM-generated formalizations of legal provisions:

Formalization:
Nine frontier LLMs are instructed to generate formalizations for ten distinct EU legal provisions (spanning GDPR, AI Act, DMA, etc.). Each formalization is represented as a rooted, labeled tree where leaf nodes are input variables (facts) and internal nodes are logical operators (AND, OR, NAND, NOR). The root represents the binary legal consequence (e.g., "prohibited" or "permitted").
Matching (Equivalence Class Graph Construction):
To compare trees with potentially different structures, the method identifies nodes across different formalizations that represent the same legal concept. An LLM (specifically Claude Opus 4.6, selected for consistency) groups nodes from all formalizations into Equivalence Classes (ECs). This creates a shared vocabulary (interface) for comparison. The matching process ensures that each formalization contributes at most one node to any given EC, forming a directed acyclic graph (DAG) of concepts.
Edge Case Analysis (SAT-Based Enumeration):
For every pair of formalizations, the method defines a shared interface based on the deepest ECs common to both. It translates the sub-trees above this interface into Boolean functions ( $f_A$ and $f_B$ ).
- The system computes the disagreement function $f_A \oplus f_B$ .
- Using a SAT solver (Z3), it enumerates prime implicants of this disagreement function. A prime implicant represents a minimal set of input conditions (a partial assignment) that forces the two formalizations to disagree, regardless of the values of other unassigned variables.
- These prime implicants are treated as "edge cases."
Verbalization:
Raw prime implicants are not human-interpretable. The authors apply a selection pipeline to reduce the set of edge cases to a representative subset (grouping by "root causes" of disagreement). These selected cases are then verbalized by an LLM into concrete, realistic factual scenarios. The verbalization includes the scenario, the specific legal question, and an analysis of the interpretive divergence, allowing legal experts to review the specific point of contention.

Key Contributions

The paper makes three primary contributions:

A Framework for Comparative Formalization: A method to systematically compare different formalizations of the same legal provision by enumerating the specific input cases on which they disagree, rather than relying on aggregate structural similarity.
Verbalization of Edge Cases: A technique to transform SAT-derived logical edge cases (prime implicants) into concrete factual scenarios that legal experts can evaluate, bridging the gap between formal verification artifacts and human legal reasoning.
Empirical Study of LLM Formalizations: An application of this method to nine frontier LLMs formalizing ten EU provisions. The study reveals that behavioral divergence is largely uncorrelated with structural agreement and that verbalized cases expose qualitatively distinct types of disagreement, including those mirroring genuine controversies in legal commentary.

Results

The study generated 90 formalizations (9 models $\times$ 10 provisions) and analyzed 360 pairs.

Structural vs. Behavioral Divergence: The authors found that high structural agreement (measured by the coverage of the shared interface) does not guarantee logical equivalence. Even when formalizations matched well on concepts (high coverage), they frequently disagreed on outcomes. Conversely, low-coverage pairs were often trivially equivalent simply because they shared no structure to disagree over.
Model Variance: Disagreement rates varied significantly across models (ranging from ~32% to ~71% of pairs being non-equivalent), but no single model was uniformly "edgy." Behavior was highly provision-dependent, with some models swinging from 0% to 100% disagreement across different provisions.
Qualitative Findings:
- Interpretive Ambiguity: In the case of Article 5 of the AI Act (regarding biometric categorization for law enforcement), the models split evenly (4 vs. 4) on whether a specific carve-out applied. This divergence mirrored a real, ongoing controversy in legal commentary regarding the scope of the exception.
- Formalization Errors: In the case of Article 3 of the Unfair Contract Terms Directive (UCTD), one model (Kimi K2.5) incorrectly encoded a procedural burden-of-proof rule as an additional substantive requirement (conjunction instead of disjunction), leading to an over-constrained definition.
- Limitations: The pipeline only surfaces disagreements; it cannot detect errors shared by all models (e.g., if all models miss a cross-reference in the law).

Significance and Claims

The paper claims that "to know the law is to know its verdicts." Its significance lies in shifting the evaluation of legal formalizations from static structural analysis to dynamic, outcome-based comparison.

Audit Aid, Not Automation: The authors explicitly state that their pipeline is intended as a "human-in-the-loop audit aid," not a component of an automated legal decision system. The formalizations are not validated for autonomous use.
Uncovering Hidden Divergence: The method demonstrates that purely structural comparison is insufficient. Two formalizations can look similar but produce radically different legal outcomes in specific edge cases.
Mirroring Legal Discourse: The verbalized edge cases successfully surfaced disagreements that correspond to genuine, unresolved debates in legal scholarship, suggesting that LLMs are not just hallucinating randomly but are grappling with the same ambiguities that human experts face.

The authors conclude that while their method has limitations (dependence on LLM matching, lack of external legal context, and the inability to verify correctness without a gold standard), it provides a necessary mechanism for "knowing" the law through the decisions its formalizations encode.