Technical Summary: Deontic Reasoning with Agentic Harnesses (DAR)

Problem Statement

Deontic reasoning involves answering questions by applying explicit rules and policies to case-specific facts, a capability critical for high-stakes domains such as tax computation, immigration appeals, and policy compliance. The primary technical challenge in this domain is the nature of the rulesets: statutes are often lengthy, heavily cross-referenced, and contain provisions where most are irrelevant to a specific case, while obligations are qualified by definitions and exceptions located elsewhere in the text.

Current evaluation paradigms typically place the entire statute, case facts, and question into a single prompt, requiring the model to identify and apply relevant rules in a single pass. However, recent work on agentic search suggests that models handle long corpora more effectively when they can search them using general-purpose tools (e.g., grep, sed, shell commands) rather than receiving them as static context. It remains an open question whether this "agentic search" advantage extends to deontic reasoning, where the task is not merely retrieval but reasoning grounded in retrieved rules.

Methodology

The authors introduce Deontic Agentic Reasoning (DAR), a framework where the statute is not provided as static context but is instead placed as a file (statute.txt) within a harness environment. The model interacts with this file on demand using general-purpose tools.

Experimental Setup

• Datasets: Evaluation is conducted on the hard subsets of DeonticBench, covering four tasks:
  • SARA (Numeric): Numerical tax-liability computation (U.S. federal tax).
  • SARA (Binary): Binary statutory-entailment classification.
  • Airline: Application of baggage fee policies.
  • USCIS-AAO: Immigration-appeal outcome prediction.
• Models: The study evaluates nine models, including frontier proprietary models (GPT-5.1, GPT-5.2, Claude Sonnet 4.5) and open-source models (Qwen3.5 family, Qwen3-Coder, Kimi K2).
• Harnesses: The authors compare Direct Solving (static context) against two agentic harnesses orchestrated by the Harbor framework:
  • Terminus-2: A terminal-based agent harness where the model operates autonomously in a sandbox via an interactive tmux session.
  • Terminus-KIRA: An enhanced harness built on Terminus-2, designed to mitigate specific failure modes observed in frontier models, such as premature submission, false completion signals, and brittle plan adjustment.

Key Results

The experiments reveal a stark divergence in performance based on model capability:

1. Frontier Models Benefit: Proprietary models (GPT-5.2, Claude Sonnet 4.5, GPT-5.1) show significant improvements under agentic harnesses, particularly on numerical tasks.

   • On SARA-Numeric, GPT-5.2 improved from 30% (Direct) to 60% (Terminus-KIRA).
   • Claude Sonnet 4.5 rose from 36% to 54%.
   • These models leverage the harness to perform self-directed retrieval and recover from intermediate errors, effectively unlocking latent statute-reading abilities.

2. Open-Source Models Degrade: The same scaffolding that aids frontier models causes significant performance degradation in open-source models, especially on numerical tasks.

   • Qwen3.5-35B dropped from 34% to 11% on SARA-Numeric under Terminus-KIRA.
   • Qwen3.5-122B fell from 37% to 20%.
   • On the Airline task, every open-source model collapsed to near-zero accuracy under Terminus-2 and Terminus-KIRA, despite having non-trivial baselines in direct solving.

3. Token Consumption and Error Modes:

   • Agentic harnesses significantly increase token consumption. Open-source models under Terminus-2 consumed up to 4× more tokens per trial than frontier models (e.g., Qwen3.5-122B averaged 401k tokens vs. ~100k for frontier models).
   • Instead of intelligent stopping, weaker models appear to use the additional turns to generate longer, more confident, but incorrect answers.
   • Error analysis shows open-source models suffer primarily from timeouts (exceeding the 10-minute budget) and parsing failures, whereas frontier models remain largely error-free across harnesses.

Contributions

The paper makes three primary contributions:

1. DAR Framework: The introduction of Deontic Agentic Reasoning, a setup where agents access statutes on demand via a harness rather than static context.
2. Systematic Comparison: A comprehensive evaluation of DAR against direct prompting across a spectrum of frontier and open-source models on DeonticBench.
3. Capability-Dependent Effects: The demonstration that the efficacy of agentic harnesses is not uniform; they act as capability amplifiers for frontier models but can degrade the performance of weaker models while inflating computational costs.

Significance and Claims

The paper claims that agentic harnesses push the frontier on the hardest deontic reasoning tasks but are not a universal fix. The findings support the "Mismanaged Geniuses Hypothesis," suggesting that capable models possess latent abilities to navigate complex rules that are only unlocked when given interactive tool access. Conversely, for weaker models, the harness acts as a "confidence amplifier," leading to more verbose but less accurate outputs.

The authors caution against the assumption that providing tool access uniformly improves performance. They highlight that for weaker models, harnesses can produce confident errors while consuming substantially more compute. Consequently, they advise against deploying current LLMs, with or without agentic harnesses, as autonomous decision-makers in high-stakes legal or tax contexts, noting that even frontier models achieve only partial accuracy on these tasks.

The study is limited to the DeonticBench domain (U.S. tax, immigration, airline policies) and specific harness implementations. The authors note that for substantially longer rulesets (e.g., the full U.S. Internal Revenue Code), the current file-based approach may not scale efficiently without pairing DAR with more advanced retrieval systems.