💻 computer science

AI Risk-Management Control in Human-in-the-Loop Systems: Governing the Operating Confidence Threshold as a Delegation Boundary

This paper proposes treating the operating confidence threshold in human-in-the-loop AI systems as a formal, auditable risk-acceptance control derived from explicit cost-benefit analysis rather than a mere tuning parameter, demonstrating its effectiveness in stabilizing automated delegation and minimizing human review in regulated professional settings.

Original authors: Musarat Kabir-Chisty

Published 2026-10-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Musarat Kabir-Chisty

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern workplace, artificial intelligence is increasingly asked to make decisions that matter. It sorts documents, matches names in databases, and flags important information. But a critical question remains unanswered by most software: when should the computer decide for itself, and when should it stop and ask a human for help? This is not just a technical setting; it is a question of trust and responsibility. If a machine acts too confidently when it is wrong, it can cause serious harm that no one notices until it is too late. If it acts too cautiously, it wastes the time of expensive human experts who could be doing more important work. The point where the machine hands the decision back to a person is called a confidence threshold. Think of it as a speed limit sign on a highway: it tells the driver exactly when to slow down and take control. Until now, many organizations have set this limit by guessing or picking a round number like ninety percent, without a clear reason for why that specific number was chosen.

A new study by Musarat Kabir-Chisty argues that this limit should not be a guess. Instead, it should be a carefully calculated decision, treated with the same seriousness as any other safety rule in a regulated profession. The author, who works in legal technology, developed a method to find the perfect "handover point" by weighing the costs of two different kinds of mistakes. The first mistake happens when the computer acts on a wrong answer without a human looking at it first. The second happens when the computer stops to ask a human for help, even though it actually had the right answer. In fields like law or immigration, the first mistake is far more dangerous than the second. A wrong file sent to a court can ruin a case, while a human reviewing a correct file just takes a few extra minutes. The study proposes that organizations should set their confidence threshold based on how much they are willing to pay in human time to avoid the risk of an unreviewed error.

To test this idea, the researcher applied the method to two real-world systems used by a legal practice in London. The first system helps immigration lawyers manage thousands of client documents. Every time a new file arrives, the computer tries to sort it into the correct folder. If the computer is very sure, it files the document automatically. If it is unsure, it sends the file to a human lawyer to check. The second system deals with client records, trying to match names from a list of new clients against a database of existing cases. Again, if the match is clear, the system links the records; if it is ambiguous, a human must decide. In both cases, the team looked at the actual scores the computer gave to thousands of documents and records to see how the system performed at different confidence levels.

The results revealed something surprising about how these systems actually behave. In the document-sorting task, the computer's confidence scores were not spread out evenly. Instead, they were sharply divided into two groups: the system was either extremely confident, with scores near the top, or it was quite unsure. There were very few cases where the computer was "sort of" sure. Because of this gap, the researchers found that the exact number chosen for the threshold did not matter much, as long as it was placed in that empty space between the confident and the unsure. Whether they set the limit at ninety-two percent or ninety-five percent, the system would send the exact same documents to the human reviewers. This made the decision robust and safe. In the second task, involving client records, the pattern was even clearer. The scores for matches were either very high or very low, with almost nothing in the middle. This meant that the system could automatically handle nearly half of the cases while leaving only a tiny handful of truly difficult ones for human experts.

The study also showed that this approach changes who is responsible for the work. By treating the threshold as a formal rule rather than a software setting, the organization can clearly document why they chose to let the machine act on certain cases and not others. If a mistake happens, they can look back and see that the decision was made based on a calculated trade-off between the cost of human time and the risk of an error. This makes the system auditable. The researchers found that when they used their method, the number of mistakes the computer made without human help was extremely low, while the number of times it asked for help unnecessarily was also kept to a minimum. The system worked efficiently because it was designed to match the specific risks of the job, not just to follow a generic rule.

This work suggests that the future of artificial intelligence in high-stakes fields depends on how well we manage these handover points. It is not enough to build a smart computer; we must also build a smart process for deciding when to trust it. The study demonstrates that by looking at the actual data and understanding the real-world costs of mistakes, organizations can set these limits with confidence. They can ensure that human experts are not overwhelmed by easy tasks they do not need to do, nor are they left blind to the rare moments when the machine might be wrong. In the end, the goal is not to replace human judgment but to define exactly where it begins and ends, creating a partnership where both the machine and the human play to their strengths.

Technical Summary: AI Risk-Management Control in Human-in-the-Loop Systems

Problem Statement
In high-stakes organizational settings (e.g., legal, finance, healthcare), AI systems increasingly operate under a "confidence-gated" paradigm: the system acts automatically when a model's confidence score exceeds a threshold ( $\tau$ ) and defers to human review otherwise. While existing governance frameworks mandate human oversight, they rarely provide guidance on how to operationalize the specific decision boundary where authority transfers from human to machine.

Currently, operating thresholds are often treated as technical hyperparameters, set to arbitrary round numbers (e.g., 0.90 or 0.95) without explicit justification. This paper argues that the operating threshold is fundamentally a risk-acceptance control. It defines the level of residual automated error an organization tolerates without human review. In regulated environments like legal practice, the costs of errors are asymmetric: a false accept (acting automatically on a wrong decision) can corrupt records or breach professional obligations, whereas a false reject (escalating a correct decision) merely wastes expert time. Treating the threshold as a mere tuning parameter obscures the ethical and governance implications of delegating consequential decisions to machines.

Methodology
The paper proposes a transparent, cost-sensitive framework for governing the operating threshold as a "decision delegation boundary." The methodology consists of three core components:

Formulation as Selective Classification: The problem is framed as selective classification (classification with a reject option) under asymmetric error costs. The system minimizes a total expected cost function $J(\tau)$ :
$J(\tau) = C_{FA} \cdot FA(\tau) + C_{FR} \cdot FR(\tau)$
Where $C_{FA}$ and $C_{FR}$ are the costs of false accepts and false rejects, respectively, and $FA(\tau)$ and $FR(\tau)$ are the expected counts of these errors at threshold $\tau$ . The optimal threshold $\tau^*$ is the point where the marginal cost of admitting an additional automatic error equals the marginal cost of an unnecessary escalation.
Governance of the Cost Ratio: The ratio $C_{FA} : C_{FR}$ is not determined by the model but by organizational values and risk tolerance. The paper recommends that this ratio be owned by a risk owner or compliance function, derived from the actual consequences of errors (reversibility, regulatory exposure, population affected), and documented. Alternatively, organizations may fix a maximum tolerable selective risk and choose the lowest threshold achieving it.
Empirical Validation and Sensitivity: The method requires:
- Calibration: Verifying that confidence scores reflect observed accuracy (e.g., via reliability diagrams), as modern foundation models are often overconfident.
- Score Distribution Analysis: Examining the distribution of confidence scores to determine if the operating point sits in a "flat basin" (robust to small changes in $\tau$ ) or a "steep slope" (fragile).
- Per-Category Thresholds: Recognizing that a single global threshold may be suboptimal for heterogeneous data; per-category thresholds can be derived by applying the cost objective within specific classes.

Key Contributions

Reframing the Threshold: The primary contribution is a governance shift, treating the operating threshold not as a model parameter but as a governance artifact. It is the specific control that instantiates delegated machine authority and must be documented, justified, and periodically reviewed.
Cost-Sensitive Protocol: A reproducible protocol for deriving the threshold from an explicit expected-cost objective, linking machine learning mechanics to legal accountability and professional responsibility.
Auditability: The framework provides a mechanism to make delegated authority auditable. By documenting the cost ratio, the score distribution, and the resulting threshold, organizations can reconstruct the rationale for automated decisions in the event of an adverse outcome.

Results from Deployments
The method was demonstrated on two real-world deployments within a regulated legal practice:

Document Classification (Immigration Casework):
- Task: Auto-filing documents into case checklists.
- Data: 2,471 classifier-seen documents.
- Distribution: The confidence scores were sharply bimodal, with 99.0% of documents (2,447 of 2,471) scoring $\ge 0.98$ and a sparse corridor between 0.60 and 0.95.
- Outcome: The cost-minimizing threshold ( $\tau^*$ ) was found to be in the range [0.83, 0.85]. The deployed threshold of 0.92 was set higher as a conservative margin. The analysis showed that the operating point is robust; the cost function $J(\tau)$ is flat across a wide range, meaning small variations in the threshold do not significantly alter the expected cost. The system achieved a selective risk of 0.04% with 99.6% coverage.
Entity Resolution (Record Linkage):
- Task: Reconciling client entities against a case database.
- Data: 270 reconciled entities.
- Distribution: Sharply bimodal with a high-confidence mode ( $\ge 0.90$ ) and a low-confidence mode ( $\le 0.65$ ), leaving a completely empty gap between 0.70 and 0.90.
- Outcome: The auto-link threshold was set at 0.90. Because no entities fell in the 0.70–0.90 range, the specific choice of 0.90 was empirically robust; any threshold within the empty gap would route the exact same entities. Only 2.2% of cases required human review, concentrating oversight on genuinely ambiguous cases.

Significance and Claims
The paper modestly claims that its contribution lies in governance rather than machine learning methodology. The expected-cost criterion is classical; the novelty is the reframing of the threshold as a governed, documented, and auditable artifact.

Accountability: Treating the threshold as a governance artifact restores a chain of responsibility. It ensures that the "decision delegation boundary" is not an opaque configuration but a deliberate organizational choice subject to review.
Meaningful Oversight: By designing the threshold to concentrate human attention on genuinely uncertain cases (rather than flooding reviewers with easy cases or hiding errors), the method supports "meaningful human oversight" rather than nominal rubber-stamping.
Generalizability: While demonstrated in legal workflows, the framework applies to any high-stakes "accept-or-defer" system (e.g., financial onboarding, clinical records) where AI acts above a confidence threshold.

The paper concludes that setting the threshold is an act of institutional design that allocates moral and professional responsibility. Therefore, it warrants the same explicit justification and documentation as any other delegation of consequential authority.

Technical Summary: AI Risk-Management Control in Human-in-the-Loop Systems

More like this