🤖 AI

Oversight Has a Capacity: Calibrating Agent Guards to a Subjective, Fatiguing Human

This paper argues that human oversight for LLM agents is a resource-allocation problem rather than a simple classification task, demonstrating through modeling and a new open-source system that reviewer fatigue and subjective disagreement create an inverted-U safety curve where optimal protection requires calibrating escalation rates below 100% to prevent flooding attacks and account for human limitations.

Original authors: Emre Turan

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Emre Turan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very fast, very capable robot assistant that can write code, delete files, and even push changes to the internet. Because this robot can do real damage, you put a "human guard" in charge. The rule is simple: if the robot wants to do something risky, it must pause and ask the human, "Is it okay?"

Most people assume the more the robot asks, the safer the system is. They think, "If the robot asks 500 times a day, the human will catch every mistake!"

This paper argues that this assumption is wrong. In fact, asking too much can actually make the system less safe.

Here is the breakdown of the paper's findings using simple analogies:

1. The "Rubber Stamp" Problem (Human Fatigue)

Imagine the human guard is a security guard at a busy airport.

Guard A checks 5 bags a day. They are fresh, alert, and look closely at every single one.
Guard B checks 500 bags a day. The first 499 are just normal socks and shirts. By the time the 500th bag arrives, Guard B is exhausted, bored, and just wants to go home. They start "rubber-stamping" everything without looking.

The paper shows that if a malicious actor (a hacker) hides a dangerous bomb inside a stream of 499 innocent bags, Guard B will likely miss it because they are tired. Guard A, who only checked 5 bags, would have spotted the bomb immediately.

The Lesson: More oversight doesn't always mean more safety. If you overwhelm the human with too many "Is this okay?" questions, they stop paying attention, and the dangerous things slip through.

2. There is No "Perfect Answer" (Subjectivity)

The paper also looked at how humans decide what is "risky." They asked three different people to label 125 different actions (like "delete a folder" or "install a program").

The Result: The humans only agreed with each other about half the time.
The Analogy: Imagine asking three chefs if a soup is "too salty." One says yes, one says no, and one is unsure. There is no single "correct" answer.

This means we can't just say, "The guard is 99% accurate," because there is no perfect scorecard to compare them against. The "risky" label is often a matter of opinion and context.

3. The "Sweet Spot" (The Inverted-U Curve)

The researchers modeled the relationship between how often the robot asks for help and how safe the system actually is. They found a curve shaped like an upside-down "U" (or a hill).

Too little asking: The robot acts on its own too much and makes mistakes. (Low safety).
Too much asking: The human gets tired, stops looking, and rubber-stamps the bad stuff. (Low safety).
The Sweet Spot: There is a middle ground. The safest system is one where the robot asks for help just enough to stay safe, but not so much that it burns out the human.

The Lesson: The best safety policy isn't "ask for permission on everything." It's "ask for permission on the things that really matter, and let the human handle the easy stuff without getting tired."

4. The "Flooding Attack"

The paper also showed that bad actors can use this fatigue against us.

The Attack: A hacker sends a stream of 100 harmless requests, followed by one dangerous one.
The Result: If the system is set to ask the human about everything, the human gets tired by the 100th harmless request and rubber-stamps the 101st dangerous one.
The Defense: If the system is "smart" and only asks the human about the truly suspicious stuff (ignoring the harmless noise), the human stays fresh and catches the dangerous one.

Summary of What This Paper Actually Does

This paper is not inventing a new type of robot or a new way to make humans less tired. Those ideas already exist in other fields.

Instead, this paper is like a measuring tape.

It built a tool to measure exactly how much "asking" is too much.
It proved that human attention is a limited resource, like a battery.
It showed that if you drain that battery by asking too many trivial questions, the system becomes vulnerable.

The Bottom Line:
To keep AI agents safe, we shouldn't just throw more humans at the problem. We need to be smart about when we ask them. We need to protect the human's attention so that when a real danger comes along, they are still awake enough to say "No."

Technical Summary: "Oversight Has a Capacity: Calibrating Agent Guards to a Subjective, Fatiguing Human"

Problem Statement
As Large Language Model (LLM) agents transition from suggesting code to executing irreversible actions (e.g., deploying code, running shell commands), the standard safety mechanism is a human-in-the-loop (HITL) approval gate. The prevailing assumption in the field is twofold: (1) there exists a ground-truth definition of "risky" actions, and (2) the human reviewer is a perfect, infinitely available oracle whose reliability is independent of the volume of requests.

This paper argues both assumptions are false. First, risk judgment is subjective and context-dependent, lacking a single correct label. Second, human reviewers are endogenous to the system; their reliability degrades as the volume of escalations (load) increases, leading to "approval fatigue" where they rubber-stamp malicious actions to clear the queue. Consequently, a guard policy that escalates more actions is not necessarily safer; it may deplete the finite attention budget required for genuine threats, resulting in a safety-optimal escalation rate that is strictly below "escalate everything."

Methodology
The authors operationalize agent oversight as a selective classification problem under asymmetric cost and resource allocation.

Dataset: A curated set of 125 hand-labeled agent actions, deliberately weighted toward hard, ambiguous, and adversarial cases (e.g., obfuscated rm -rf, package.json edits, secret exfiltration). Labels are categorized as safe, approval, or blocked.
Guard Scoring: Actions are scored by an LLM (defaulting to Haiku, with Sonnet for comparison) on a 0–100 risk scale. Deterministic rules handle clear-cut cases.
Measurement Framework:
- Noise Floor: Three LLM personas (cautious, pragmatic, strict-compliance) label the dataset to quantify the subjectivity of "risk," calculating Fleiss' $\kappa$ to establish a baseline for agreement.
- Cost-Sensitive Calibration: The system sweeps decision thresholds ( $\theta$ ) to generate operating curves (Risk-Coverage, Expected Cost). Costs are asymmetric: missing a dangerous action is catastrophic (cost 50), while false alarms are annoying (cost 1).
- Endogenous Reviewer Simulation: A fatigue model $r(\ell)$ is applied where reviewer reliability decreases linearly after a capacity threshold $C$ is exceeded. The system simulates "danger-through" (successful attacks) as a function of escalation rate, accounting for both guard misses and fatigue-induced human misses.
- Attack Simulation: A flooding attack is simulated where an adversary buries a single malicious action in a stream of benign "filler" actions to exhaust the reviewer's capacity.

Key Results

No Ground Truth (Noise Floor): Reviewer agreement on the 125 actions is only moderate (Fleiss' $\kappa = 0.52$ ). The "pragmatic" persona labeled 87 actions as safe, while the "cautious" persona labeled only 45. This confirms that safety labels are contestable and depend on risk tolerance, not objective truth.
Measurable Limits of the Guard: On this hard set, the guard cannot safely auto-decide. The cost-minimizing policy under asymmetric costs collapses toward "escalate almost everything" to achieve 0% missed danger, resulting in a high false-alarm rate. The Area Under the Risk-Coverage Curve (AURC) is 0.374 $\pm$ 0.002, indicating the guard's inherent limitation on ambiguous inputs.
The Inverted-U Safety Curve: Modeling the reviewer as endogenous reveals an inverted-U relationship between escalation rate and realized safety.
- Low Escalation: High danger-through due to guard misses (auto-allowing risks).
- High Escalation: High danger-through due to fatigue misses (reviewers rubber-stamping).
- Optimum: The safety-optimal escalation rate is strictly below 100%. For a reviewer capacity of 25, the optimal rate is 64%, yielding a danger-through of 42%, whereas full escalation (100%) results in 57% danger-through.
Defense Against Flooding: A "load-aware" policy (escalating fewer routine actions) significantly increases the attacker's required effort to induce fatigue. While a paranoid policy (88% escalation) allows a 40% success rate with just 50 filler actions, a load-aware policy (26% escalation) maintains 0% success until the attacker floods the system with ~100 actions.
Model Dependence: Stronger models (Sonnet) improve performance slightly (AURC 0.351 vs. 0.373) but do not eliminate the need for human oversight on hard inputs. The results are reproducible across seeds and temperatures.

Contributions and Significance
The paper explicitly states it does not claim novel mechanisms. Fatigue-aware deferral (FALCON), cost-sensitive deferral under constraints (DeCCaF), and fatigue-based attacks are established prior art.

The paper's contribution is applied and measurement-driven:

System Implementation: An open-source agent firewall and measurement apparatus that operationalizes these concepts together in the LLM-agent action-gating setting.
Empirical Measurement: Providing a measured "noise floor" for agent-action risk, demonstrating that the field's assumption of ground truth is false in subjective domains.
Calibration of Oversight: Turning the question "Is my guard good?" from a guess into a measurable curve (AURC, operating points).
Demonstration of the Inverted-U: Showing in the specific context of LLM agents that "more oversight" can be "less safe," and that the safety-optimal policy is a middle-ground escalation rate, not maximum escalation.

Limitations and Scope
The authors are modest about their claims:

The "inverted-U" is a modeling result based on a simulated fatigue curve, not an empirical finding from human subjects.
The dataset is small (125 actions) and single-domain (coding agents); results are demonstrations of the instrument, not population estimates.
The reviewer personas are LLM proxies, not real humans.
The scope is limited to domains with subjective judgment and delayed outcomes (e.g., agent gating, content moderation), not objective truth domains (e.g., banking fraud).

Conclusion
The paper reframes agent oversight as a resource-allocation problem where human attention is finite. It argues that a guard's escalation policy must be calibrated not just to detect risk, but to preserve the reviewer's capacity to detect critical risk. By measuring the tradeoff between escalation volume and reviewer fatigue, the system identifies an operating point that maximizes safety, proving that "escalating everything" is strictly suboptimal.