💬 NLP

From Statute to Control Flow: Span-Grounded Deontic Trees for Defeasible Scope Parsing

This paper introduces NormBench, a multilingual benchmark featuring Span-Grounded Deontic Trees (SG-DT) to diagnose and mitigate Silent Scope Omission in rule-following agents by shifting focus from end-task outcomes to the precise structural parsing of defeasible legal scopes, revealing that constrained intermediate representations significantly improve exception handling in complex regulatory contexts.

Original authors: Jian Chen, Siyuan Li, Chucheng Wan, Zixuan Yuan

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jian Chen, Siyuan Li, Chucheng Wan, Zixuan Yuan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Silent Skip"

Imagine you are hiring a very smart, fluent robot assistant to enforce the rules of a game. You give it the rulebook.

Rule 1: "If you touch the ball, you get a penalty."
Rule 2: "Unless you are the goalie, then you can touch it."
Rule 3: "But if the goalie is wearing a red hat, they still get a penalty."

A human reads this and understands the layers. But the paper argues that current AI models often suffer from Silent Scope Omission (SSO).

The AI sees Rule 1, applies it, and confidently says, "Penalty!" It completely silently skips Rule 2 and Rule 3. The AI's answer sounds reasonable and grammatically perfect, but it misses the crucial "unless" and "except" clauses that actually change the outcome. It's like a chef who follows a recipe perfectly but silently forgets the instruction to "remove the salt if the customer is on a diet," resulting in a dish that looks right but tastes wrong.

The Solution: NormBench and the "Tree Map"

To fix this, the authors built a new testing ground called NormBench. Think of this as a specialized gym for AI, filled with 2,290 different legal rules (from Chinese laws, US tax codes, and corporate policies) designed specifically to trick AI with these "silent skips."

Instead of just asking the AI, "What's the penalty?", they force the AI to draw a Span-Grounded Deontic Tree (SG-DT).

The Analogy: The Construction Blueprint
Imagine the law is a messy pile of lumber.

Old Way: The AI tries to build a house by guessing which piece of wood goes where. It might build a house that looks okay from the outside, but the roof is on the wrong floor.
New Way (SG-DT): The AI must first draw a blueprint.
- Every single branch of the blueprint must be tied to a specific piece of wood (a "span" of text) from the original rulebook.
- The blueprint must explicitly show the "exclusion guards." For example, it must draw a line saying, "This branch only exists IF the red hat condition is FALSE."
- This turns the messy text into a rigid, logical flowchart that can be audited. If a branch is missing, the blueprint is invalid.

What They Found: The AI's "Brain Farts"

The authors tested top-tier AI models (like GPT-5, Claude, and DeepSeek) on this new test and found three major issues:

1. The "Recursion Decay" (The Tower of Logic)

The Analogy: Imagine stacking blocks.
- Layer 1: "Do X." (Easy)
- Layer 2: "Do X, unless Y." (Okay)
- Layer 3: "Do X, unless Y, unless Z." (The AI collapses).
The Finding: As the rules get deeper (nested exceptions inside exceptions), the AI's performance crashes. It's not that the AI ran out of memory; it's that its "reasoning muscles" get weak when the logic gets too deep. It can find the rules, but it can't stack them correctly.

2. The "Auditability Trap" (The Confident Liar)

The Analogy: A student who memorizes the textbook quotes perfectly but doesn't understand the math.
The Finding: The AI is great at finding the right sentences in the text (high "faithfulness"). It will quote the rule about the "red hat" perfectly. But when asked to connect that quote to the right part of the logic tree, it fails. It looks like it understands the law, but it's actually just "quoting" without "reasoning."

3. The "Domain Paradox" (Generalists vs. Specialists)

The Analogy: A general doctor vs. a specialist who only reads medical journals but has never treated a patient.
The Finding: Surprisingly, general-purpose AI models (trained on everything) did a much better job at understanding these legal structures than "Legal AI" models (trained only on law). The legal models were too focused on sounding like a lawyer (using the right tone) and failed at the rigorous logic required to map the rules.

The Cross-Language Glitch

The authors also tested if an AI understands the same rule in Chinese and English.

The Finding: The AI could build a good tree for the Chinese version and a good tree for the English version. But the two trees often didn't match each other!
The Analogy: It's like translating a recipe. The Chinese version says "Add salt unless the soup is salty." The English version says "Add salt, but if the soup is salty, don't." The AI might build the logic correctly for each language individually, but the "unless" logic gets flipped between the two, leading to different results for the same rule.

Does This Actually Help?

The paper tested if forcing the AI to draw this "blueprint" (SG-DT) before answering a question actually improved the final answer.

The Result: It depends.
- Yes: When the case is tricky and relies on a specific exception (the "active" exception), the blueprint helps the AI avoid the "Silent Skip."
- No: When the case is simple, or if the AI is already very good, forcing it to draw the blueprint sometimes confuses it or slows it down, leading to worse answers.

Summary

The paper argues that to make AI reliable for laws and rules, we can't just let it chat. We need to force it to build a rigid, auditable map of the rules, where every exception is explicitly tied to the text. This helps catch the "Silent Skips" where the AI ignores important exceptions, but it's not a magic bullet that fixes everything instantly.

Technical Summary: From Statute to Control Flow: Span-Grounded Deontic Trees for Defeasible Scope Parsing

1. Problem Definition: Silent Scope Omission (SSO)

Rule-following agents designed to execute policies and regulations frequently fail via Silent Scope Omission (SSO). In this failure mode, a model applies a general rule but silently drops nested exceptions, counter-exceptions, or overriding clauses. The resulting output appears compliant and passes lightweight checks but fails on critical edge cases where the "defeater" clauses (e.g., "unless," "except," "provided that") determine the final outcome.

While often framed as an agentic-systems issue, the authors identify the underlying bottleneck as statutory and policy understanding, specifically the ability to recover the non-monotonic structure of legal texts where later clauses restrict earlier ones. Existing legal NLP benchmarks (e.g., LexGLUE, LegalBench) typically emphasize end-task outcomes (QA, entailment, judgment prediction). These metrics can reward a correct final decision even when the model fails to recover the internal scope/override structure, thereby masking the structural omissions that cause SSO.

2. Methodology: NormBench and SG-DT

2.1 NormBench Benchmark

To diagnose and mitigate SSO, the authors introduce NormBench, a benchmark comprising 2,290 provision-language items across three domains:

NormBench-ZH: Codified Chinese statutory provisions (national laws, administrative regulations, local policies).
NormBench-EN: English rules from U.S. tax law, EU GDPR, and corporate policies.
Cross-lingual: Official ZH–EN translation pairs to test structural invariance.

The dataset is organized from controlled (codified statutes) to in-the-wild (operational policies) and includes 1,790 unique underlying provisions. It is annotated with a specific intermediate representation designed for defeasible scope parsing.

2.2 Span-Grounded Deontic Trees (SG-DT)

The core methodological contribution is SG-DT, a compiler-style intermediate representation (IR) that anchors every logical branch to immutable source text spans. Unlike standard free-form generation, SG-DT enforces four design constraints:

Span-grounded Auditability: Every node is a pointer to a specific text span, enabling deterministic mapping to downstream signatures.
Decidability under Defeasibility: Exceptions are compiled into Explicit Exclusion Guards. A general rule branch explicitly includes an exclusion leaf pointing to the exception trigger, ensuring the rule applies only if the exception is not met. This prevents silent dropping of tail-end scope.
Deterministic Evaluation: The structure supports exact structural scoring and compilation into executable decision procedures.
Learnability: The representation is dense and seq2seq-friendly, avoiding expert-only ontologies.

Schema Details:

Branches ( $B_i$ ): Defined by an Anchor (trigger span), Modality (e.g., Obligation, Prohibition), Condition Tree ( $C$ ), and Effects ( $E$ ).
Defeasibility: Handled via an Explicit Exclusion Policy. If a rule has an exception trigger $T$ , the general branch includes an exclusion leaf ( $\neg T$ ), and the exception branch includes $T$ as a positive precondition. This ensures logical disjointness ( $B_{gen} \land B_{exc} \equiv \text{False}$ ) without heuristic priority rules.
Recursion Depth ( $D$ ): Defined as the length of the exclusion-induced defeater chain (e.g., $D \ge 2$ for nested exceptions).

3. Key Contributions

NormBench & SG-DT: A standardized benchmark and IR for evaluating structural scope recovery in legal texts, covering 2,290 items across controlled and in-the-wild domains.
Pathology Analysis: Identification of two key barriers to reliable legal agents:
- The Recursion Bottleneck: Performance degradation driven by logical depth (nested exceptions) rather than context length.
- The Auditability Trap: Models retrieve relevant spans with high fidelity but fail to assemble the correct control flow (attachment of exceptions).
Mechanism-Specific Intervention: Demonstration that SG-DT utility is conditional. It significantly improves outcomes in exception-active/SSO-prone cases where the result depends on a defeater, but aggregate gains can be mixed or negative when the added structure is unnecessary or parser fidelity is low.

4. Experimental Results

4.1 Intrinsic Structure Recovery (SG-DT Parsing)

Evaluations of frontier LLMs (e.g., GPT-5.2, DeepSeek-V3.2, Claude-Opus-4.5) reveal a Structure-Grounding Gap:

High Faithfulness, Low Logic: Models achieve high SpanFaith (~~0.77) and low hallucination but low Edge-F1 (~~0.24). They can "quote" the rule but fail to wire the exceptions correctly.
The Domain Paradox: General-purpose Frontier LLMs significantly outperform specialized Legal LLMs (e.g., ChatLaw, LawGPT). Legal-domain models suffer from catastrophic grounding failures (Edge-F1 ~0.04), suggesting their training prioritizes conversational fluency over rigorous, span-grounded pointer discipline.
Recursion Decay: Performance drops sharply when defeater depth increases ( $D \ge 2$ ). For instance, GPT-5.2's Edge-F1 falls from 0.22 ( $D=0$ ) to 0.07 ( $D \ge 2$ ).

4.2 Cross-Lingual Isomorphism

On bilingual (ZH–EN) pairs, models often produce structurally different logic trees for the same provision despite similar monolingual quality.

Iso-F1 < 0.40: Even when Tree-ES (edit similarity) is moderate, the ZH and EN trees are rarely isomorphic.
Translation Ambiguity: Small differences in how scope cues (e.g., "unless" vs. "provided that") are rendered can flip which defeater a condition attaches to, creating a "divergent control flow" risk.

4.3 Downstream Utility (Statutory Reasoning)

Using SG-DT as a constrained intermediate output improves whole-tree fidelity and defeater recovery, but the benefits are mechanism-specific:

Exception-Active Cases: SG-DT provides clear gains (e.g., +4.2% on LegalBench for DeepSeek-V3.2) when the case facts trigger an active exception.
Simple Cases: In cases without active defeaters, adding SG-DT structure can introduce overhead or degrade performance (e.g., GPT-4o-mini drops in accuracy).
Recursive-CoT: Combining SG-DT with Recursive Chain-of-Thought does not consistently improve accuracy and can sometimes negate gains by drifting from the crisp decision boundary.

5. Significance and Claims

The paper claims that the primary failure mode in legal agents is not a lack of retrieval or fluency, but a failure to materialize nested exceptions within the control flow.

Diagnostic Value: NormBench and SG-DT provide a necessary diagnostic layer that separates retrieval failures from reasoning failures, specifically isolating the "Silent Scope Omission" pathology.
Auditability: By grounding every logical branch in immutable text spans and enforcing explicit exclusion guards, SG-DT enables deterministic compilation and audit, addressing the "black box" nature of current LLM legal reasoning.
Pragmatic Utility: The authors modestly claim that SG-DT is not a universal drop-in solution. Its value is highest in high-stakes, exception-heavy scenarios where the cost of SSO is high. For simpler rules, the structural overhead may not be justified.

The work positions SG-DT as an agent-facing intermediate representation that bridges the gap between raw statutory text and executable, auditable decision procedures, moving beyond surface component extraction to explicit defeasible scope modeling.