💻 computer science

LEDGER: Deterministic Checks and Auditable Evaluation for Business AI Systems

The paper introduces LEDGER, an evaluation platform that enables companies to create deterministic, auditable, and cost-aware tests for their business AI systems using proprietary data, demonstrating how a self-hosted open-weight model can achieve frontier-level accuracy at a fraction of the cost of proprietary alternatives.

Original authors: Ben Sirb, Peter A. Devine, Nick Ryan, Alex Chiocchi

Published 2026-11-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ben Sirb, Peter A. Devine, Nick Ryan, Alex Chiocchi

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern workplace, companies are increasingly relying on artificial intelligence to answer questions about their own internal data. An engineering manager might ask how many code changes were reviewed last week, or a financial analyst might request the average time it took to resolve a specific type of error. These systems are not just a single computer program thinking in a vacuum; they are complex assemblies where a language model is connected to the company's private databases, guided by strict rules about what it can and cannot reveal, and managed by software that decides which tools to use. The challenge for any business is that these systems are fragile. A small change to the model, the rules, or the way the software talks to the database can suddenly make the answers wrong. While public tests exist to rank how smart different artificial intelligence models are on general tasks, those scores do not predict how well a specific company's system will work with its own unique data and rules. Companies need a way to test their own systems repeatedly after every single change, ensuring that the answers remain correct and that the cost of getting those answers stays manageable.

A team of researchers at Interval, a company that builds business intelligence tools, has created a new method called LEDGER to solve this problem. Instead of relying on public scores, LEDGER turns a company's own data, questions, and rules into a repeatable, rigorous test. The system works by creating specific scenarios: a question is asked, a specific correct answer is defined in advance, and a set of evidence is prepared to prove whether the answer is right. This evidence might be a frozen snapshot of a database query, a specific policy rule, or a recorded change in a software dashboard. When the artificial intelligence system attempts to answer the question, the LEDGER platform checks the result. If the answer is a simple number or a clear fact, a deterministic check—a computer program that follows strict logic without guessing—scores it immediately. If the answer is written in complex prose that the computer cannot easily parse, a separate artificial intelligence judge reviews it, but only after a deterministic check confirms that the evidence supports the claim. Every single score is tied back to the specific evidence that produced it, creating a clear, auditable trail that shows exactly why an answer was marked correct or incorrect.

The researchers used this platform to build and test a business intelligence assistant powered by DeepSeek V4 Flash, a powerful open-weight model that the company hosts on its own private servers. They compared this custom-built system against seven other leading models, including five proprietary systems from major technology companies and another open-weight model. The comparison was strict: every system was given the exact same tools, the exact same data, and the exact same set of questions. The results showed that when the open-weight model was wrapped in the company's specific tools and rules, it performed just as well as the most advanced proprietary models. On a test set of 97 carefully selected scenarios, the custom system answered 96 correctly, while the other top models answered between 94 and 97 correctly. This demonstrated that the "harness" of tools and rules surrounding the model was just as important as the model itself. Without these specific tools and definitions, the open model performed slightly worse, but with them, it reached the same level of accuracy as the most expensive, state-of-the-art systems.

The most significant finding, however, concerned the cost. While the accuracy was similar across the board, the price to get a correct answer varied wildly. The researchers calculated the cost per correct answer for each system. The proprietary models, which are accessed through expensive cloud services, cost between six and seventy-seven times more per correct answer than the self-hosted open-weight model used by Interval. Even when the researchers applied the cheapest available pricing tiers for the proprietary models, they still cost significantly more than the company's own private setup. This suggests that for businesses with specific data needs, the choice of model is less about finding the single "smartest" brain and more about finding the right balance between accuracy, control, and cost. The study argues that by using a rigorous, auditable testing system like LEDGER, companies can confidently choose cheaper, self-hosted models without sacrificing performance, provided they invest in the right tools and rules to guide them.

The researchers were careful to note that their results are specific to their own workload and the particular way they structured their tests. They did not claim that this open-weight model is universally superior to all others in every situation, nor did they suggest that the cost savings would be identical for every company. The study measured performance on a specific set of business questions and found that, for this type of work, a well-designed system around a cheaper model can match the results of the most expensive options. The key takeaway is that accuracy in business applications is not solely a property of the artificial intelligence model itself, but of the entire system built around it. By making the testing process transparent and repeatable, companies can make informed engineering decisions, moving away from blindly paying for the most expensive models and toward building systems that are both accurate and economically sustainable.

Technical Summary: LEDGER – Deterministic Checks and Auditable Evaluation for Business AI Systems

1. Problem Statement

Public benchmarks (e.g., Text-to-SQL) evaluate models on generic tasks and do not predict performance on a specific company's operational data, rules, and orchestration logic. A deployed AI assistant is a composite system comprising a model, prompts, tools, data access policies, and orchestration; changes to any component can alter correctness. Furthermore, public test items often leak into training data, inflating scores via memorization. Companies lack a practical, repeatable mechanism to retest their AI systems after every change using their own data, with scores that are auditable and traceable to specific evidence.

2. Methodology: The LEDGER Platform

The authors present LEDGER (Logged Evidence for Deterministic, Governed Evaluation and Review), an evaluation platform designed to turn a company's data, questions, and rules into a repeatable test.

Core Architecture

Scenario Definition: Each test case (scenario) records a question or interface action, an expected outcome, and the evidence source. Evidence sources include:
- Frozen SQL Queries: Executed against a frozen copy of the company's data warehouse.
- Fixtures: Hand-entered facts extracted from test documents.
- Policy Rules: Deterministic checks for refusal (e.g., PII) or clarification requirements.
- Interface Contracts: Verification of dashboard state transitions.
Deterministic Checks (Oracles): Where the outcome can be computed from evidence (e.g., a specific number or a refusal), a deterministic check scores the answer. This ensures the verdict is identical every time.
Logged LLM Review: For answers in prose that deterministic checks cannot parse, a separate LLM judge evaluates the response. Crucially, this judge is constrained by a deterministic evidence check. A verdict is only valid if the answer passes the evidence check (e.g., stating the correct value in prose) and the judge's approval.
Provenance and Freezing: Every run records the model identity, prompt hash, scenario content hash, tool schemas, and raw outputs. This allows every score to be traced back to specific evidence and enables rerunning the same scenarios against different systems.

Experimental Design

The platform was applied to Interval's business intelligence assistant, which queries an engineering and operations warehouse.

System Under Test (SUT): DS4+harness, running the privately hosted open-weight model DeepSeek V4 Flash (DS4) inside a custom task harness (prompts, rule packs, skills, sub-agents, and four governed data tools).
Baselines: Seven models were tested: five proprietary frontier models (GPT-5.6 Sol/Terra, Claude Opus/Sonnet/Fable), the open-weight GLM-5.3, and DS4 running "bare" (without the harness).
Prompt Conditions:
- Schema-only: Lists warehouse tables and columns.
- Schema-plus-registry: Adds the company's metric registry (definitions, contracts, and SQL logic).
Datasets:
- Development Set (814 scenarios): Used to iteratively build and repair the harness.
- Test Set (97 scenarios): Generated after the harness was frozen, using new values absent from the development set to prevent data leakage.

3. Key Contributions

The LEDGER Platform: A system that separates the scenario, evidence, system configuration, and scoring method. It enables deterministic scoring where possible and constrained LLM review where necessary, ensuring every score traces to evidence.
A Case Study on System vs. Model Contribution: The study isolates the value of the "harness" (orchestration and rules) from the base model. It demonstrates that a privately hosted open-weight model (DS4) can achieve frontier-level accuracy when augmented with a company-specific harness and metric registry.
Cost-Accuracy Trade-off Analysis: The paper introduces "cost per correct answer" as a primary metric, revealing significant cost disparities between proprietary APIs and self-hosted open-weight models when accuracy is normalized.

4. Results

Accuracy

On the held-out test set of 97 scenarios:

DS4+harness achieved 96/97 correct.
Bare DS4 (with registry) achieved 94/97.
Frontier Models (Proprietary + GLM-5.3) achieved 94 to 97/97.
Bare DS4 (schema-only) achieved 89/97, trailing the frontier models.
Impact of Harness: The harness lifted bare DS4 from 89 to 96 correct. The metric registry alone recovered 5 of the 7 points lost by bare DS4 without it. The frontier models performed similarly with or without the registry.

Cost Efficiency

Using the "schema-plus-registry" prompt:

Self-hosted DS4 (bare) cost an estimated $0.0010–$ 0.0018 per correct answer.
Proprietary Frontier Models cost $0.0111 to $0.0766 per correct answer.
Multiplier: Frontier models cost an estimated 6 to 77 times more per correct answer than self-hosted DS4. Even using DeepSeek's off-peak API rates, the cost was 1.6 to 3.0 times higher than self-hosting.

Development Set Observations

On the development set (814 scenarios), DS4+harness passed 808. Baselines passed between 627 and 650. However, the authors note these figures understate baseline performance because many baseline answers were "undeterminable" from saved records (e.g., missing logs) and the harness was iteratively tuned on this specific data.

5. Significance and Claims

The paper claims that LEDGER provides a defensible framework for companies to adopt open-weight, self-hosted models without sacrificing accuracy.

Measurable Trade-offs: It makes the trade-off between accuracy and cost explicit. If multiple models reach similar accuracy once company definitions are spelled out, the choice becomes an engineering decision based on cost and control rather than a blind reliance on "frontier" model rankings.
Iterative Improvement: Because every failure points to a specific scenario, expected value, and rule, the system allows for targeted repairs (often driven by coding agents) while human review validates the definition of "correct."
Auditability: The platform ensures that evaluation results are not black boxes; every score is traceable to evidence, and the review process is recorded with versioned judges.

The authors conclude that without such a test, the "safe choice" is to remain on expensive proprietary models. With LEDGER, a company can achieve frontier-level accuracy on its own workload at a fraction of the cost, making the choice of model a measurable and defensible engineering decision.

6. Limitations

The paper explicitly acknowledges several limitations:

Ceiling Effects: Systems are near the ceiling on the test set; single runs cannot definitively rank systems or establish equivalence.
Template Dependence: Scenarios share query logic, potentially overstating evidence in paired tests.
Review Variability: While LLM review is constrained, it remains non-deterministic; the paper reports the variability of the judge on reruns.
Audit Scope: Human audits were conducted by a single author and were not fully blind to the study context, limiting independent error estimation.
Scope: The platform is demonstrated on one company's tasks and tool-calling models; external reproduction is limited by confidential data.