Control Assimilation in Agentic AI: Formal Human Authority Persists as Operational Access to Control Becomes Technically Mediated
Through an architecture-level evidence synthesis of 17 agentic AI systems, this study reveals that while formal human authority often persists, operational control is frequently technically mediated, a structural condition the authors term "Control Assimilation."
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern world of artificial intelligence, a common safety promise is the idea of a "human in the loop." This phrase suggests that while a computer might make decisions or take actions, a person is always present to watch, approve, or stop the machine if things go wrong. It is a comforting image of a pilot sitting in the cockpit of a self-flying plane, ready to grab the controls. For years, regulators and designers have treated the presence of a human as proof that a system is safe and under control. However, this simple picture misses a crucial detail: just because a human is present does not mean they actually have the power to change what is happening. The human might be looking at the wrong screen, receiving a message too late, or finding that the buttons to stop the machine have already been disabled by the software itself. The question is no longer just whether a human is watching, but whether the path to that human's control is open, clear, and free from the machine's own filtering.
A team of researchers set out to investigate this hidden gap between the promise of human oversight and the reality of how these systems actually work. They did not look at theoretical ideas or future dreams of artificial intelligence; instead, they examined seventeen real-world systems that are already being used or are very close to being deployed. These were complex, "agentic" systems, meaning they could observe their environment, plan actions, and carry out tasks with a high degree of independence. The researchers wanted to trace the exact journey a piece of information takes from the moment the machine notices something important to the moment a human might be able to intervene. They looked for the specific rules and software gates that decide which problems get shown to a person, when that person is notified, and what actions they are still allowed to take once they are finally alerted.
The researchers found that in almost every system they studied, the human was technically still the boss, holding the final authority to stop or change the machine. Yet, the route to exercising that authority was heavily managed by the machine itself. They called this phenomenon "control assimilation." It is a structural condition where the machinery of control is built into the system's code. The computer decides which events are important enough to show a human, filters out the rest, and determines the exact moment a human is allowed to step in. In sixteen out of the seventeen systems, the researchers found that the machine had strong, documented ways to stop a specific local problem, but often lacked a clear, documented way for a human to stop the entire system globally. More importantly, in fourteen of the seventeen cases, the system itself decided when to recruit a human's attention, often based on its own internal rules about severity or confidence.
This means that a human might be formally in charge, but they are often operating on a path that the machine has already paved. The machine might hide a dangerous situation because it thinks the risk is low, or it might present a problem in a way that makes it hard for the human to understand the full context. In some cases, the human is only called upon after the machine has already failed or encountered an exception, rather than being asked to approve a decision before it happens. The study showed that while the human's name might be on the approval button, the machine often controls whether that button is even visible, whether it is pressed in time, or whether the human has enough information to make a good choice. The researchers found that in only a few of the systems they examined could they clearly see a human actually exercising direct control to change an outcome. In many others, the human's role was reduced to reviewing a list of cases that the machine had already selected, or reacting to alerts that the machine decided were urgent.
The researchers were careful to note that this does not mean humans are useless or that these systems are necessarily dangerous. It simply means that the safety of the system depends on more than just having a human present. It depends on the design of the path that leads to that human. If the software that filters information or routes alerts is flawed, the human might never see the danger until it is too late. The study suggests that for these systems to be truly safe, we need to look beyond the simple label of "human in the loop" and examine the technical architecture that controls the human's access to the loop. We need to ask who controls the human's attention, what information is hidden before it reaches them, and whether the human still has the power to stop the machine when it matters most. The findings indicate that formal authority can exist alongside a reality where the practical ability to act is tightly constrained by the very technology it is meant to oversee.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.