Practical Judgment, Virtue, and Intuition in the Use of Opaque AI-Enabled Systems
This paper argues that the ethical and legal concerns surrounding opaque, autonomous AI systems can be effectively mitigated by prioritizing non-quantifiable human capabilities such as practical judgment, virtue, and intuition, using the military domain as a primary exemplar to demonstrate how these traits bridge the gap between technical opacity and societal norms.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you're playing a high-stakes video game where the rules are written in a language you don't speak, and the game engine is a "black box." You can see the inputs (you pressing buttons) and the outputs (the character jumping), but you have absolutely no idea how the code inside the box decided to make that jump. Now, imagine that same black box is driving a tank, flying a drone, or making life-or-death decisions on its own. This is the world of opaque AI: systems that work, but whose inner workings are hidden, unpredictable, and often impossible for humans to fully understand.
The authors of this paper, Nathan Gabriel Wood and Andrew P. Rebera, are worried. They ask: If we can't see how the machine thinks, and if the machine can act on its own, how do we stop it from making terrible mistakes?
The Problem: The "Black Box" and the "Autopilot"
Think of opacity like a magician's trick. You see the rabbit appear, but you can't see the mechanism. When a machine is "opaque," even its creators can't always explain why it chose a specific output. This gets scary when you add autonomy.
Imagine a security camera (the opaque system) that spots a person and flags them as a "threat."
- Scenario A (Human in Control): The camera flags the person. A human guard sees the flag, looks at the video, realizes it's just a guy in a swimsuit (a "naked soldier" scenario), and stops the alarm. The human catches the mistake.
- Scenario B (Autonomous): The camera is hooked up to a robot soldier. It flags the guy in the swimsuit as a threat, and without asking anyone, the robot soldier fires. Because the system is opaque, no one knew it would make that mistake, and because it's autonomous, no human was there to hit the "stop" button in time.
The paper argues that trying to fix this with just better technology (like "Explainable AI" or XAI) isn't enough. Sometimes, the math is just too complex to explain, or the system changes itself so fast that explanations become useless before they're written.
The Solution: The "Virtuous Pilot"
Instead of trying to make the black box transparent (which might be impossible), the authors suggest we lean into the one thing machines can't do: human intuition and character.
They propose that the best defense against a mysterious, autonomous machine is a human with practical judgment, virtue, and intuition.
Think of it like training a dog. You don't program a dog with a rigid rulebook like "If the ball is red, fetch; if the ball is blue, ignore." You train the dog through experience, repetition, and a bond of trust. Eventually, the dog knows what to do in a weird situation because it has "good judgment."
The authors argue that military personnel (and other professionals) should be trained similarly. They need to develop a "virtuous character"—traits like courage, honesty, and restraint. This allows them to:
- Sense the "vibe": Just as a skilled animal handler knows their animal is about to bolt by a subtle twitch of an ear, a skilled human operator can sense when an AI system is acting weird, even if they can't see the code.
- Spot the "Naked Soldier": Machines might see a person bathing and think "Target." A human with good judgment sees the context and thinks, "Wait, that's not a threat."
- Know when to say "No": The most important judgment is deciding whether to use the system at all.
Judgment is a Finite Battery
Here is a crucial twist in the paper: Judgment is a resource. It's like a battery.
Every time you make a hard decision, you use up some of your "judgment battery." If you are tired, stressed, or overwhelmed, your battery drains fast. If your battery hits zero, you start making dangerous mistakes.
The authors suggest a new rule for using opaque AI: Only turn on the black box when your judgment battery is about to die.
- If you have plenty of energy and time, use your own brain.
- If the situation gets so crazy that you are about to burn out and make a fatal error because you are trying to do everything yourself, then you can offload some tasks to the opaque AI.
This isn't about letting the AI take over because it's "smarter." It's about using the AI as an emergency backup when your human brain is maxed out. If you use the AI when you still have plenty of judgment left, you are wasting a valuable human skill and risking "de-skilling" (forgetting how to do things yourself).
What This Paper is NOT Saying
It's important to know what this paper doesn't promise:
- It doesn't say AI is safe. The authors admit these systems are risky and unpredictable.
- It doesn't say we can fix the "black box." They argue that for many modern AI systems, full transparency is impossible.
- It doesn't say rules are useless. Rules and technical fixes are still needed, but they aren't enough on their own.
- It doesn't say everyone can do this. The paper explicitly states that if a person lacks the necessary character (honesty, self-reflection, courage) or judgment, they should not be allowed to deploy these systems. In fact, their bosses should take the decision away from them.
The "Swiss Cheese" Safety Model
The authors wrap up with a visual idea called the "Swiss Cheese Model." Imagine safety is a block of Swiss cheese. Each layer (rules, technology, training) has holes in it. If the holes line up, a disaster happens.
The authors suggest that human virtue and judgment are just another slice of cheese. They won't plug every hole, and they won't stop every disaster. But when you stack them on top of technical fixes and institutional rules, the holes are less likely to line up.
In short: We can't always see inside the machine, and we can't always predict what it will do. But by training humans to be wise, honest, and self-aware, and by only letting them use these mysterious machines when they are truly overwhelmed, we can keep the risks from turning into tragedies. It's not a magic fix; it's a way to make sure the human in the loop is the smartest, most careful part of the equation.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.