Modeling AI Overreliance as a Complex Adaptive System
This paper models AI overreliance as a complex adaptive system, demonstrating that while individual learning and social consensus do not inherently cause collective failure, visible unverified use triggers feedback cascades that can be prevented through strategic feedback design.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern workplace, a new kind of assistant has arrived. It is an artificial intelligence that can draft emails, write code, or summarize legal documents. We are told these tools are incredibly accurate, yet a strange problem has emerged: people do not always use them correctly. Sometimes they trust the machine too much, accepting a wrong answer as if it were fact. Other times, they ignore a helpful tool because it made a single mistake in the past. Researchers have long known that the accuracy of the software is not the only thing that matters; what matters more is how humans decide when to listen and when to double-check. For years, scientists studied this behavior by asking one person to solve a single problem in a quiet lab. They watched how that individual reacted to a suggestion. But this approach misses a crucial part of the real world. In reality, people do not work in isolation. They see their colleagues using these tools. They hear about successes and failures. They adjust their own behavior based on what they observe in others. This creates a complex web of influence where one person's trust can ripple through a whole group, potentially changing how everyone works.
A researcher at the University of Pittsburgh decided to model this social reality. Instead of studying one person at a time, they built a computer simulation of a whole population. They created a virtual world where hundreds of digital agents faced a series of tasks. In this world, each agent had three choices: solve the problem alone, accept the answer from an AI without looking, or use the AI but then verify the answer by checking sources or asking a human. The agents learned from their own experiences, updating their belief about how reliable the AI was. Crucially, they also learned from their neighbors in the simulation. They could see what their peers were doing. If they saw many colleagues accepting AI answers without checking, they felt a social pressure to do the same. The researcher ran thousands of these simulations to see how the group's behavior evolved over time, testing different levels of task difficulty, AI quality, and social influence.
The first thing the simulation revealed was that the environment sets the stage. The difficulty of the task and the actual quality of the AI determined the baseline level of trust. When tasks were easy and the AI was good, people trusted it appropriately. When tasks were hard and the AI was poor, people tended to over-rely on it, accepting wrong answers more often. However, the researcher found that the most dangerous factor was not the AI itself, but the way people watched each other. They discovered a specific mechanism they call "social proof." This is the feeling that a behavior is correct because everyone else is doing it. In the simulation, when agents saw their neighbors using the AI without checking, the utility of checking dropped. It felt unnecessary to verify if everyone else was just accepting the answer. This created a feedback loop. As more people stopped checking, it became even more normal to stop checking, until the entire group fell into a state of collective over-reliance. The population stopped verifying the AI's work entirely, not because the AI was perfect, but because the social signal said verification was no longer needed.
The researcher then tested whether simply connecting people in a network was enough to cause this collapse. They wondered if the structure of the group mattered—whether having a few popular "hubs" with many connections would change the outcome. Surprisingly, they found that the network structure itself did not drive the over-reliance. If the agents were only learning from the raw results of their peers' work, the group simply reached a consensus on the AI's quality without changing their overall behavior. The network did not make them blindly trust the machine. The collapse only happened when the visible behavior of others directly influenced the decision to check or not check. It was the sight of unverified use that mattered, not just the presence of a connected network. This distinction is vital. It means that the problem is not that people are too connected, but that they are seeing the wrong kind of behavior.
Finally, the researcher explored how to stop this collapse. They tested different ways to redesign the feedback loop. They found that simply making it cheaper or easier to check the AI's work did not solve the problem. Even if verification took less time, the social pull to stop checking remained too strong. The only way to reverse the trend was to change what people saw. When the researcher made the act of verification visible—so that agents could see their peers checking the work—the group reversed course. The population began to check again, and the collective over-reliance disappeared. Similarly, if they dampened the social signal that made unverified use look normal, the group recovered. The study suggests that to keep people from blindly trusting AI, we cannot just make the tools better or the checking process easier. We must design our interfaces so that the act of checking is visible and celebrated, breaking the cycle where silence and unchecked acceptance become the default.
The results of these simulations offer a clear picture of how human-AI interaction works in a social setting. The danger of over-reliance is not a flaw in the individual user, but a systemic failure of the feedback loop. When a community sees unchecked use as the norm, verification dies out. When verification is made visible, the community recalibrates. This does not mean the problem is solved, but it points the way forward. The researcher emphasizes that these findings come from computer models, not real-world experiments, so they represent a set of testable hypotheses rather than a final verdict. Yet, the logic is robust: the way we see our peers using technology shapes how we use it ourselves. If we want to avoid a future where entire groups stop questioning the machines they rely on, we must be careful about what behaviors we make visible. The solution lies not in the algorithm, but in the design of the social environment around it.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.