Technical Summary: Confidently Wrong: Exception Chain Collapse in Frontier LLM Rule Evaluation

1. Problem Statement

The paper identifies a specific, systematic failure mode in frontier Large Language Models (LLMs) termed "exception chain collapse." This occurs during eligibility evaluation tasks involving nested conditional rules of the form: "A is required UNLESS B applies, UNLESS C overrides B."

While frontier LLMs perform well on straightforward multi-route logic (e.g., simple OR-branching), they degrade significantly when required to evaluate multi-level exception chains (specifically three levels deep). The paper documents two distinct failure patterns:

1. Exemption Anchoring: The model treats exemptions as secondary to the primary route. If the primary route fails, the model "anchors" on that failure and fails to independently evaluate valid alternative exemption routes.
2. Exception Chain Collapse: The model fails to correctly nest multi-level UNLESS logic, often conflating independent exemption pathways (e.g., treating a "veteran" exemption as dependent on an "age" exemption).

The Core Instability: A critical finding is that frontier model accuracy on these tasks is a "moving compliance boundary." Between March and April 2026, specific failure cells in benchmarks closed silently under the same model alias (e.g., GPT-5.4 and Claude Opus 4.6) without version bumps. This makes benchmark-time accuracy claims unreliable for regulated workflows where consistency is mandatory.

2. Methodology

The authors propose and evaluate the Aethis Eligibility Module, a neuro-symbolic architecture designed to separate rule authoring from rule execution.

The Architecture

• Phase 1: Automated Rule Authoring (LLM): An LLM reads authoritative legal sources (legislation, policy guidance) and generates rules as structured code in a constrained Domain Specific Language (DSL). This phase includes a provenance chain where every generated rule is linked to specific source citations (document ID, section path, direct quote).
• Phase 2: The Eligibility Module (Deterministic Engine): The generated DSL is compiled into formal Satisfiability Modulo Theories (SMT) constraints. An SMT solver then evaluates these constraints against structured applicant data.
  • Key Mechanism: The engine treats every eligibility route as a formally independent boolean branch combined via disjunction (OR). It evaluates these branches deterministically, independent of model drift, reasoning effort, or prompt format.

Benchmark Design

The authors constructed a benchmark of 225 scenarios across four domains:

1. Life in the UK: Real UK immigration legislation (British Nationality Act 1981).
2. English Language Proficiency: Real UK immigration guidance.
3. Spacecraft Certification: A synthetic statute modeled on UK legislative structure with three-level exception chains.
4. Construction Insurance: Synthetic policy wording modeled on London market DE3/DE5 clauses with five-level exception chains.

The benchmark was evaluated against eight LLMs (four frontier, four production-tier) from Anthropic and OpenAI, comparing them against the deterministic Eligibility Module.

3. Key Contributions

1. Failure Pattern Taxonomy

The paper formally characterizes "exception chain collapse" and "exemption anchoring" as systematic errors in nested exception-chain evaluation. These failures are shown to be:

• Compositional: They arise from the depth of the logic (three levels) rather than a lack of legal knowledge.
• Robust to Prompting: Enhanced prompting (e.g., "think step by step," "evaluate exemptions independently") fails to fix the issue; instead, it trades false negatives for false positives, reducing net accuracy.
• Unstable: The specific failure points shift silently across model updates, making frontier models unreliable for high-stakes, audit-critical decisions.

2. The Aethis Eligibility Module

The paper presents a neuro-symbolic system that relocates uncertainty from the inference boundary (where it is silent and continuous in LLMs) to the specification boundary (where it is deliberate and audited).

• Guarantee: The execution layer provides mathematically defined semantics. If the rule bundle is correctly formalized, the execution is 100% consistent with the specification, regardless of model version or configuration.
• Auditability: Every determination includes a provenance chain linking the outcome directly to the specific legislative clause and the logical path taken.

3. Empirical Evidence

• Benchmark Results: The Eligibility Module achieved 100% accuracy across all 225 scenarios.
• LLM Performance: Frontier models showed significant degradation on exception-chain tasks. For example, in the March 2026 snapshot, Claude Opus 4.6 scored 89.7% on the spacecraft section (61/68), with 7 specific scenarios failing 0/3 times across independent runs.
• Adversarial Extension: In a v3.8 adversarial extension (20 new construction insurance scenarios), the deterministic engine scored 20/20. While some frontier models reached 100% on the original suite, they failed on the deeper adversarial cases (e.g., Claude Opus 4.7 failed 2/20, GPT-5.4 default failed 1/20).
• External Validation: On 949 held-out cases from nine LegalBench tasks, the Eligibility Module was significantly more accurate than three frontier models (combined McNemar's $p \le 0.003$), with the largest margins (+41 percentage points) on multi-prong rule-application tasks.

4. Significance and Claims

The paper does not claim that LLMs are generally unreliable or that they cannot be used for legal reasoning. Instead, it makes a modest, specific claim:

• Relocation of Uncertainty: The primary contribution is architectural. By using LLMs for authoring (where their fluency is an asset) and SMT solvers for execution (where determinism is required), the system makes the uncertainty tractable. The uncertainty is moved to the rule formalization step (Level 2), which can be managed via test-driven validation and SME review, rather than remaining in the inference step (Level 3), where it is silent and unobservable.
• Regulatory Defensibility: For high-stakes domains (immigration, insurance, safety certification) where false negatives deny rights and explainability is mandatory, the deterministic execution layer offers a property that frontier LLMs cannot: invariance. A compiled rule bundle yields the same result today as it will in six months, regardless of silent changes to the underlying model weights.
• Limitations: The authors explicitly state that the system guarantees correct execution of a specification, not the correctness of the specification itself. The quality of the rule bundle depends on the LLM's ability to formalize the source text (Level 2), which is mitigated but not eliminated by test-driven validation. The system currently requires structured inputs and does not handle unstructured document extraction.

In summary, the paper argues that for nested exception-chain evaluation in regulated workflows, deterministic formal execution is a necessary condition for trust, and that neuro-symbolic architectures provide a viable path to achieving this without sacrificing the utility of LLMs for rule extraction.