Interpreting Emergent Extreme Events in Multi-Agent Systems

1. Problem Statement

Large Language Model (LLM)-powered Multi-Agent Systems (MAS) are increasingly used to simulate complex human-like systems (economic, financial, social). However, these systems often exhibit emergent extreme events (Black Swans)—rare, high-impact phenomena like market crashes, hyperinflation, or social polarization.

The core challenge is the "black box of emergence." While these events can be simulated, their origins are obscured by complex, non-linear interactions between agents and the environment. Existing literature validates the existence of these events but lacks a framework to interpret them. Specifically, three fundamental questions remain unanswered:

1. When does the event originate? (Is it a sudden shock or a dormant risk?)
2. Who drives it? (Which specific agents are responsible?)
3. What behaviors contribute to it? (Which action patterns trigger the risk?)

2. Methodology

The authors propose the first framework to explain emergent extreme events in LLM-powered MAS by adapting Shapley values from cooperative game theory. The methodology consists of four main stages:

A. Problem Formulation & Attribution

• Modeling: The MAS is modeled as a trajectory $\tau$ of states and joint actions over time steps $T$. An extreme event occurs when a risk metric $R_T(\tau)$ exceeds a threshold $\rho$.
• Attribution Goal: Decompose the final risk $R_T(\tau)$ into contributions from individual actions $a_{i,t}$ taken by agent $i$ at time $t$.
• Shapley Value Adaptation:
  • The set of all actions $\Omega$ is treated as "players" in a game.
  • The "value" of a coalition of actions $S$ is defined as the risk of a counterfactual trajectory where actions in $S$ are preserved, and actions in $\Omega \setminus S$ are replaced by "safe" baseline actions.
  • The attribution score $\phi(a_{i,t})$ is calculated as the marginal contribution of an action across all possible permutations of action subsets.
  • Approximation: Since exact Shapley calculation is NP-hard ($O(2^{|\Omega|})$), the authors use Monte Carlo Sampling to approximate the values with linear complexity relative to the number of actions, verified to have high cosine similarity (>0.99) with ground truth at sufficient sample sizes.

B. Dimensional Aggregation

The action-level scores are aggregated along three dimensions to answer the core questions:

1. Time ($\phi^{tm}_t$): Sum of scores for all actions at time $t$.
2. Agent ($\phi^{ag}_i$): Sum of scores for all actions by agent $i$.
3. Behavior ($\phi^{be}_k$): Sum of scores for all instances of a specific behavior type $k$.

C. Quantitative Metrics

Based on the aggregated scores, the authors design five metrics to characterize event features:

1. Relative Risk Latency ($L^{tm}$): Measures the time gap between when the majority of risk accumulates and when the event actually occurs. Distinguishes between "dormant risks" (high latency) and "immediate shocks" (low latency).
2. Agent Risk Concentration ($G^{ag}$): Uses the Gini Coefficient to measure how concentrated the risk is among a few agents.
3. Risk-Instability Correlation ($C^{ag}$): Pearson correlation between an agent's risk contribution and the variance (instability) of their contributions over time.
4. Agent Risk Synchronization ($Z^{ag}$): Adapted from the Vicsek model to measure if agents increase/decrease risk simultaneously (directional alignment).
5. Behavior Risk Concentration ($G^{be}$): Gini Coefficient applied to behavior types to determine if risk stems from a few dominant patterns.

3. Key Contributions

1. First Framework for Extreme Event Interpretation: Proposes a unified approach to explain "Black Swan" events in LLM-MAS, moving beyond mere detection to causal attribution.
2. Faithful Attribution via Shapley Values: Adapts Shapley values to MAS to provide a prior-free, faithful attribution method that does not rely on statistical priors or external knowledge, unlike previous methods (e.g., prompting LLMs for scores).
3. Multi-Dimensional Analysis: Introduces a structured way to analyze events across time, agents, and behaviors, supported by five novel quantitative metrics.
4. Empirical Insights: Provides generalizable insights into the nature of extreme events in MAS (see Results below).

4. Experimental Results

The framework was evaluated across three diverse scenarios: EconAgent (macroeconomic), TwinMarket (financial), and SocialNetwork (opinion polarization), using various LLMs (GPT-4o, Llama, Claude, Qwen, DeepSeek).

A. Faithfulness Validation

• Metric: Risk drop after deleting top-ranked dangerous actions.
• Result: The proposed Shapley-based method achieved significantly higher risk drops (e.g., ~36-60% in TwinMarket) compared to baselines like Random, Failure Taxonomy (FT), Failure Attribution (FA), and Agent Tracer (AT). This confirms the method faithfully identifies the true drivers of risk.

B. Key Insights Derived from Metrics

1. Insight 1 (Temporal): Extreme events have distinct origins. In economic systems, they often stem from early dormant risks (high $L^{tm}$), whereas financial markets and social networks often result from immediate shocks (low $L^{tm}$).
2. Insight 2 (Agent Concentration): Extreme events are typically driven by a small subset of agents. High Gini coefficients ($G^{ag} > 0.4$) indicate risk is not system-wide but concentrated in critical agents.
3. Insight 3 (Instability): Agents with high risk contributions exhibit high instability (high variance in their actions). There is a strong positive correlation ($C^{ag} > 0.6$) between risk contribution and behavioral instability.
4. Insight 4 (Synchronization): Agents tend to act synchronously, either increasing or decreasing risk together ($Z^{ag}$ is generally high), suggesting herd behavior.
5. Insight 5 (Behavioral Concentration): A small number of behavior patterns contribute the majority of the risk ($G^{be} > 0.5$), indicating that specific stereotypical actions (e.g., short-term trading, specific interaction types) are the primary drivers.

5. Significance

• Safety & Governance: The framework provides actionable guidance for system safety. By identifying who and what drives risk, regulators or system designers can restrict high-risk agents or modify specific behavioral patterns to prevent disasters.
• Theoretical Advancement: It bridges the gap between complex system simulation and interpretability, offering a rigorous mathematical tool (Shapley values) to decode emergent phenomena in AI systems.
• Scalability: While Shapley calculation is theoretically expensive, the Monte Carlo approximation makes it feasible for systems with thousands of actions, though scaling to massive systems remains a future objective.

In conclusion, this paper establishes a foundational methodology for "opening the black box" of emergent extreme events in multi-agent systems, transforming them from unpredictable anomalies into analyzable phenomena with identifiable causes.