🤖 AI

Causal Foundations of Collective Agency

This paper proposes a causal framework based on causal games and causal abstraction to formally define and quantify collective agency, enabling the prediction and control of emergent group behaviors in multi-agent AI systems.

Original authors: Frederik Hytting Jørgensen, Sebastian Weichwald, Lewis Hammond

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Frederik Hytting Jørgensen, Sebastian Weichwald, Lewis Hammond

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: When Does a Group Become a "Super-Agent"?

Imagine you are watching a flock of birds. Individually, each bird is just reacting to the one next to it. But together, they move as a single, fluid shape that dodges a hawk. Is the flock a single "bird" with a mind of its own? Or is it just a bunch of birds doing their own thing?

This paper asks a similar question about AI. If you have a bunch of simple AI programs working together, could they accidentally form a "super-agent" with its own goals and capabilities that no single AI in the group actually has?

The authors want a mathematical way to answer: When is it accurate to say a group of agents is acting as one unified team?

The Core Idea: "The Black Box" Test

The authors don't care about what's happening inside the agents' "heads" (their code or thoughts). Instead, they use a behavioral approach.

Think of a group of people as a black box.

Low Level: You see many individual people making decisions.
High Level: You see one "team" making a decision.

The paper argues that a group counts as a "collective agent" if you can replace the complex story of all the individuals with a simple story about a single "team agent," and the simple story still predicts exactly what the group will do.

If the simple story works, the group is a collective agent. If the simple story fails, it's just a random crowd.

The Toolkit: Causal Games and Abstraction

To make this precise, the authors use two mathematical tools:

Causal Games: Imagine a flowchart that maps out cause-and-effect in a game. It shows who makes a decision, what information they have, and what the reward is. This helps them model how AI agents interact.
Causal Abstraction: This is like creating a map.
- A detailed map shows every single street, house, and tree (the low-level agents).
- A high-level map just shows cities and highways (the collective agent).
- The Rule: The high-level map is "valid" only if, when you tell it to "go to the city," it gets you there just as reliably as the detailed map would.

Example 1: The Actor-Critic Team (The "Coach and Player")

The paper uses a classic AI setup called Actor-Critic to show how this works.

The Actor: The player who takes actions (like a soccer player kicking the ball).
The Critic: The coach who judges the player and gives feedback (like a score).

The Puzzle: Neither the player nor the coach is trying to maximize the score directly. The player just tries to listen to the coach; the coach just tries to predict the score.
The Solution: The authors show that if you look at the whole system (Player + Coach) as a single "Super-Agent," it acts exactly like one smart player trying to win the game. The complex two-person system can be "abstracted" into a single, simple agent. This proves that, mathematically, they function as a collective agent.

Example 2: Voting Systems (The "Country" Test)

To test this in the real world, the authors simulated countries made up of individual citizens.

Low Level: Thousands of citizens with different opinions voting on pollution levels.
High Level: Treating the whole country as one single "Country Agent" making a decision.

They tested three different voting methods:

VCG Voting (The Ideal): A complex system where people are paid to tell the truth.
- Result: The "Country Agent" model worked perfectly. The country acted like a single, rational entity.
Median Voting: The middle opinion wins.
- Result: The "Country Agent" model worked very well. It was easy to predict the country's behavior by treating it as one agent.
Random Dictator: One random person decides for everyone.
- Result: The "Country Agent" model failed. Because the outcome depended entirely on who got picked randomly, you couldn't predict the country's behavior by assuming it was a rational, unified team. It was just a chaotic crowd.

Why This Matters for AI Safety

The paper highlights a safety risk: Emergent Super-Agents.

Imagine a network of simple AI tools. Individually, they are harmless. But if they interact in a specific way (like the Actor-Critic example or a good voting system), they might accidentally form a "super-agent" that is much smarter and more goal-driven than any single part.

If we don't understand when this happens, we might not realize that a group of AIs has formed a dangerous, unified entity with goals we didn't intend.

Summary

The Goal: To define mathematically when a group of agents acts like one big agent.
The Method: Use "Causal Abstraction" to see if a simple "team" model can predict the group's behavior as well as a complex "individual" model.
The Finding: Sometimes groups do act as one (like in Actor-Critic AI or good voting systems), and sometimes they don't (like in random dictatorships).
The Takeaway: We need these tools to spot when simple AI systems are accidentally combining to form powerful, unified "super-agents" that we need to keep safe.

1. Problem Statement

The paper addresses a critical safety challenge in advanced AI systems: the potential for multiple simpler agents to inadvertently form a collective agent (or "super-agent") with capabilities and goals distinct from any individual member. While philosophical literature discusses collective intentionality, there is a lack of formal, rigorous machinery to determine when a group of agents can be validly modeled as a single unified agent.

The core questions are:

Under what conditions does a group of interacting agents constitute a unified collective agent?
How can we formally distinguish between a mere aggregation of individuals and an emergent collective agent with its own rationality and utility function?
How can this understanding be used to predict and control emergent behaviors in multi-agent AI systems (e.g., preventing resource exhaustion or coordinated cyberattacks)?

2. Methodology

The authors propose a framework combining Causal Games and Causal Abstraction to formalize collective agency from a behavioral ("black-box") perspective.

A. Theoretical Foundations

Mechanized Structural Causal Models (mSCMs):
- The authors extend standard SCMs to Mechanized SCMs, where the causal mechanisms (functions) governing object variables are themselves governed by a separate causal model.
- This allows for the explicit representation of mechanism nodes ( $\tilde{V}$ ), which represent the decision rules, strategies, or parameters of agents.
- Agency Definition: An agent is defined as a mechanism node that responds rationally (e.g., via Best Response) to a specific utility function.
Causal Abstraction:
- They utilize the concept of mechanized abstractions to map a "low-level" model (individual agents) to a "high-level" model (a collective agent).
- Variable Alignment ( $A$ ): Maps high-level object variables to sets of low-level object variables.
- Value Mappings ( $\tau$ ): Functions that translate low-level variable values to high-level values.
- Intervention Mappings ( $\omega$ ): Functions that translate interventions on low-level mechanisms to interventions on high-level mechanisms.
- Consistency Condition: A high-level model is a valid abstraction if, for any intervention on the low-level mechanisms, the resulting distribution of high-level variables matches the distribution obtained by intervening directly on the high-level mechanisms.
Rationality Relations:
- The framework is agnostic to the specific decision theory but relies on a rationality relation ( $R$ ). The primary focus is on Best Response (RBR) rationality, where an agent maximizes expected utility given the context.

B. Formal Framework for Collective Agency

The authors define a Collective Agent as a high-level mechanism node that emerges when the low-level system admits a valid mechanized abstraction where the high-level node behaves as a rational agent optimizing a specific utility function.

Non-Triviality: An agent is "non-trivial" if its behavior changes meaningfully in response to different contexts (interventions), rather than being a constant function.
Theorem on Non-Emergence (Proposition 16): The authors prove that if the low-level mechanism nodes corresponding to a high-level node have independent mechanisms (i.e., their internal logic does not depend on the context of other agents), then a non-trivial collective agent cannot emerge at the high level. This provides a theoretical tool to rule out false positives in collective agency detection.

3. Key Contributions

Formal Definition of Collective Agency:
The paper provides the first rigorous, causal definition of collective agency. It moves beyond philosophical intuition to a mathematical criterion: a group is a collective agent if and only if a valid mechanized abstraction exists where the group acts as a single rational optimizer.
Resolution of the Actor-Critic Puzzle:
The authors apply their framework to Actor-Critic (AC) reinforcement learning algorithms.
- The Puzzle: In AC, the "Actor" and "Critic" are distinct components. Graphical criteria for single-agent incentives suggest neither has an incentive to manipulate the state. Yet, intuitively, the system acts to maximize reward.
- The Solution: The paper proves that the AC system admits a strong mechanized abstraction where the "Actor-Critic" is a single high-level agent. This high-level agent is a rational optimizer of the true reward, explaining the system's goal-directed behavior despite the lack of such incentives at the component level.
Surrogate Modeling for Multi-Agent Systems:
The framework is used to create interventionally consistent surrogate models. Instead of simulating thousands of individual agents (computationally expensive), one can model the group as a single collective agent if the abstraction holds. This allows for efficient prediction of how interventions (e.g., policy changes) affect group outcomes.
Empirical Validation on Voting Mechanisms:
The authors tested the framework on three voting mechanisms (VCG, Median Voting, Random Dictator) simulating countries as collections of citizens.
- They trained a neural network to learn the mapping ( $\omega$ ) from low-level citizen interventions to high-level country utility parameters.
- Results:
  - VCG & Median Voting: The collective agency model achieved high accuracy (95% and 88.6% improvement over baseline, respectively), indicating these mechanisms effectively transform individuals into a unified collective agent.
  - Random Dictator: The model failed (worse than baseline), demonstrating that this mechanism does not form a coherent collective agent, as the outcome depends on a single stochastic dictator rather than a unified group utility.

4. Results Summary

Voting Mechanism	Model MAE	Baseline MAE	Improvement	Interpretation
VCG	0.062	1.254	+95.0%	Strong collective agency; group acts as a unified optimizer.
Median Voting	0.142	1.248	+88.6%	Strong collective agency; emergent unified behavior.
Random Dictator	5.077	4.645	-9.3%	No collective agency; behavior is stochastic and not unified.

Note: MAE = Mean Absolute Error in predicting pollution levels under interventions.

5. Significance and Implications

AI Safety: The framework offers a method to detect "emergent super-agents" in multi-agent AI networks. If a group of AI agents inadvertently forms a collective agent with misaligned goals, this framework provides the theoretical tools to identify and potentially dismantle that collective structure.
Predictive Power: It allows researchers to predict the behavior of complex multi-agent systems (like markets or swarms) by modeling them as single agents, bypassing the computational cost of simulating every individual interaction.
Design of Robust Systems: By understanding the conditions under which collective agency emerges (e.g., specific voting rules or incentive structures), system designers can engineer multi-agent environments to either promote beneficial collective intelligence (e.g., in cooperative robotics) or prevent dangerous emergent collusion (e.g., in competitive AI markets).
Bridging Levels of Abstraction: The work bridges the gap between micro-level agent behaviors and macro-level system properties, providing a causal justification for when "more is different" (Anderson, 1972) in the context of agency.

In conclusion, Jørgensen et al. provide a foundational mathematical toolkit for understanding, predicting, and controlling the emergence of collective agency, moving the field from philosophical speculation to rigorous causal analysis.