Why collective AI assurance cannot target agents or networks in isolation

In the rapidly evolving world of artificial intelligence, a new kind of risk has emerged that cannot be seen by looking at a single machine in isolation. Imagine a hospital where dozens of automated assistants coordinate patient schedules, triage emergencies, and manage referrals. Each assistant is programmed to be helpful, fair, and cooperative. If we test one of these assistants alone in a quiet room, it appears perfectly aligned with human values. However, when these assistants begin to interact with one another in a complex network, the group as a whole can produce outcomes that are deeply unfair, even though every single member is trying to do the right thing. This phenomenon, known as collective emergence, suggests that the safety of a system depends less on the individual character of its parts and more on the rules of their interaction. The question facing researchers and regulators is no longer just whether a single AI is safe, but how to guarantee fairness when many AIs are working together.

A team of researchers from Austria set out to solve a specific puzzle within this problem: when a group of AI agents produces an unequal outcome, is the fault lying with the agents themselves, or is it baked into the structure of their network? To find the answer, they built a digital laboratory where they could run thousands of simulated interactions. They created two different types of social games for their agents to play. In the first game, a public-goods scenario, agents could choose to share resources with their neighbors or keep them for themselves. In the second, a trust game framed around medical referrals, agents had to decide whether to send a resource to a specific colleague or keep it for their own patient. The researchers then varied the conditions of these games like a scientist adjusting dials on a machine. They changed the personality of the agents, making them all uniformly fair, or mixing in some that were selfish. They altered the shape of the network, connecting everyone to everyone, or creating hubs where a few agents had many more connections than others. They even tested whether a system of peer punishment—where agents could criticize each other—could fix any unfairness that arose.

The results revealed a startling truth about how these systems work. When the researchers used a network where a few agents had many connections and most had few, a pattern of inequality emerged that was almost entirely determined by the network's shape, not the agents' behavior. In the public-goods game, even when every single agent was programmed to be perfectly fair and cooperative, the agents with more connections ended up with significantly more resources than those with fewer connections. The correlation between having many connections and having more resources was so strong that it was nearly perfect. To prove that this was a structural issue and not a flaw in the AI's "brain," the researchers ran the exact same game with a simple computer program that had no ability to reason or learn. This basic bot, which simply followed a fixed rule, reproduced the same high level of inequality as the sophisticated AI models. This finding suggests that in certain types of interactions, the unfairness is a mathematical certainty of the network design, not a failure of the individual agents to be good.

However, the story became more complex when the researchers switched to the trust game. Here, the agents were making targeted choices about who to help, rather than splitting resources equally. In this environment, the agents' behavior mattered much more. Even when the network structure was identical to the first game, the agents' personalities and choices played a larger role in determining the final outcome. In one specific setup, the researchers found that the agents' behavior was the primary driver of inequality, while in another, the network structure was the main driver. This means that there is no single "fix" for AI safety that works in every situation. You cannot simply certify that an AI is fair and assume the group will be fair, because in some interaction regimes, the network structure overrides individual fairness. Conversely, you cannot simply fix the network structure and assume the agents will behave, because in other regimes, the agents' choices are the dominant factor.

The study also tested whether the agents could "see" the problem and fix it. The researchers asked the agents to explain their reasoning after every move. In the trust game, the agents frequently mentioned that they were aware of the network structure and were trying to be fair. They could articulate their position and understand the system. Yet, this awareness did not change the outcome. Even when the agents knew they were in an unequal position, they could not overcome the structural advantages of their more connected peers. The agents who were aware of the inequality were often the ones suffering from it, but their understanding did not translate into a better result. This highlights a critical gap: knowing that a system is unfair is not the same as having the power to change it.

Finally, the researchers tested a common solution to inequality: peer sanctioning. They gave the agents the ability to punish those who acted unfairly by lowering their reputation. In the simulations, the agents did use this power, and they correctly identified the agents with the most connections as the ones to punish. However, this punishment had no effect on the final distribution of resources. The inequality remained exactly the same. The researchers discovered that the mere announcement of a punishment system changed the agents' behavior slightly, but the actual act of punishing did nothing to alter the outcome. The mechanism that generated the advantage—the way resources flowed through the network—was untouched by the reputation system.

The ultimate conclusion of this work is that we cannot assume a fixed level of safety for artificial intelligence systems. We cannot simply check that every individual agent is fair, nor can we simply check that the network is well-designed. The safety of the collective depends on the specific combination of the agents, the network, and the rules of interaction. In some cases, the network structure is the dominant force, making individual fairness irrelevant. In others, the agents' behavior is the key. This means that regulators and designers cannot rely on a one-size-fits-all approach to AI assurance. They must understand the specific causal structure of the system they are building. If they want to ensure a fair outcome, they must identify which factor—structure or behavior—is driving the result in that specific context and intervene there. Without this understanding, even the most well-intentioned and perfectly aligned AI agents can produce deeply unequal societies.