🤖 AI

Integrating Counterfactual Simulations with Language Models for Explaining Multi-Agent Behaviour

This paper introduces AXIS, a framework that leverages large language models to interrogate environment simulators with counterfactual prompts, thereby generating human-centered explanations for multi-agent systems that significantly improve perceived correctness and goal prediction accuracy compared to existing baselines.

Original authors: Bálint Gyevnár, Christopher G. Lucas, Stefano V. Albrecht, Shay B. Cohen

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bálint Gyevnár, Christopher G. Lucas, Stefano V. Albrecht, Shay B. Cohen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a complex dance performance with five dancers (the agents) moving around a stage. Suddenly, one dancer makes a strange move. You ask, "Why did they do that?"

In the world of autonomous systems (like self-driving cars), getting a good answer to that question is hard. The "dance" happens in a chaotic environment, and the dancers are reacting to each other in ways that are difficult to untangle.

This paper introduces a new method called AXIS (Agentic eXplanations via Interrogative Simulation) to help explain these moves. Think of AXIS not as a static report, but as a detective with a time machine and a magic scriptwriter.

Here is how it works, broken down into simple steps:

1. The Detective (The Large Language Model)

The "detective" is a smart AI (a Large Language Model or LLM). Its job is to talk to you and figure out the story behind the action. But it doesn't just guess; it has a special tool.

2. The Time Machine (The Simulator)

The detective has access to a perfect simulation of the world. This is like a video game that can replay the exact moment the action happened.

3. The Interrogation (Asking "What If?")

Instead of just looking at the video and guessing, the detective starts asking the time machine specific questions using "interrogation prompts." It's like a scientist running experiments:

"Remove": "What if we erased that other car from the scene entirely? Would our driver still have changed lanes?"
"What If": "What if that other car had stayed straight instead of turning? Would our driver have waited?"

4. The Synthesis (Putting the Puzzle Together)

The time machine runs these scenarios and gives the detective new data. The detective then looks at the differences:

Scenario A (Real life): Car 1 turns, Car 0 swerves.
Scenario B (Time machine): Car 1 stays straight, Car 0 does not swerve.

The detective realizes: "Aha! Car 0 swerved because Car 1 turned. If Car 1 hadn't turned, nothing would have happened."

The detective then writes a human-friendly explanation based on these "what-if" experiments, rather than just describing the raw data.

Why is this better than the old way?

The paper compares AXIS to two other methods:

The "Model Only" approach: This is like asking the detective to look at the video once and guess the reason without testing anything. The paper found this often leads to generic, wrong, or superficial answers (e.g., "They moved because there was space," missing the real reason).
The "No Explanation" approach: Just giving the user raw data to figure out themselves.

The Results:
When the researchers tested this on self-driving car scenarios (like merging on a highway or navigating a roundabout), AXIS performed significantly better:

More Accurate: The explanations were judged to be more correct (by about 7.7% to 17% more than the baseline).
Better Predictions: When people read the AXIS explanation, they were much better at guessing what the car would do next or where it was trying to go (up to 23% more accurate for some models).
Focus on Causality: The "detective" learned to ask the right "what-if" questions to find the true cause, rather than just listing facts.

The Catch (Limitations)

The paper is honest about where the system struggles:

Irrational Actors: If a "dancer" in the simulation does something completely crazy or breaks the rules of the road, the detective sometimes gets confused because it assumes everyone is acting logically.
Hidden Actors: If a car is hidden behind a building (occluded), the detective sometimes misses that hidden car is the real reason for the action, because it can't "see" it in the simulation.
The Judge: To test how good the explanations were, the researchers used another AI to grade them (since human studies are expensive). While this worked, the paper notes that AI judges might have their own biases, like preferring shorter answers or getting confused by too much detail.

The Bottom Line

AXIS is a system that helps AI explain its decisions by simulating alternative realities. Instead of just saying "I did X," it asks, "What would have happened if I didn't do X?" and uses the answer to tell a clear, cause-and-effect story to the human user. It works best when the agents are acting logically and the environment is visible.

Technical Summary: Integrating Counterfactual Simulations with Language Models for Explaining Multi-Agent Behaviour

Problem Statement
Autonomous Multi-Agent Systems (MAS), such as self-driving cars and trading algorithms, face significant trust challenges due to risks like miscoordination and goal misalignment. While explainable AI (XAI) is crucial for trust calibration, current methods for Explainable Sequential Decision-Making (SDM) face three primary roadblocks: (1) the complexity of environments (high-dimensional spaces, partial observability, non-stationary dynamics); (2) the "human factor," where explanations must align with diverse cognitive processes and preferences; and (3) a lack of standardized evaluation metrics. Existing approaches often focus on low-level mechanistic explanations (e.g., saliency maps) rather than high-level "action explanations" that describe agent behavior in terms of causal relationships among actions, beliefs, and outcomes.

Methodology: AXIS
The authors propose Agentic eXplanations via Interrogative Simulation (AXIS), a framework that generates human-centered action explanations by combining Large Language Models (LLMs) with environment simulators. AXIS operationalizes the Counterfactual Effect Size Model (CESM), which posits that a cause is selected for an explanation if it co-occurs with an event under different but similar background conditions.

The AXIS framework functions through a multi-round interrogation and synthesis process:

Contextualization: The system retrieves the current state of the MAS (modeled as a Partially Observable Stochastic Game, or POSG) and verbalizes observations, actions, and static environment elements into text for the LLM.
Interrogation: The LLM proposes intervention prompts (e.g., remove, whatif, add, what) to the simulator. These prompts are based on options (high-level action abstractions) to bridge the gap between low-level simulator states and LLM reasoning capabilities.
Simulation: The simulator executes the proposed intervention, forward-simulating the environment to generate counterfactual trajectories and rewards.
Synthesis: The results of the simulation are verbalized and fed back to the LLM, which synthesizes the counterfactual information into a partial explanation.
Iteration: This loop repeats for a maximum number of rounds ( $N_{max}$ ), allowing the LLM to refine its understanding of causal relationships before generating a final explanation.

Key Contributions

Formalization: The problem of action explanation for MAS is formalized within the context of partially observable stochastic games.
Framework: The introduction of AXIS, a novel method for generating action explanations using counterfactual simulations and LLMs.
Evaluation Methodology: A comprehensive evaluation framework for explainable SDM that combines:
- Qualitative correctness analysis.
- Robustness testing against hyperparameters and model variations.
- Subjective preference metrics (completeness, satisfaction, detail, trustworthiness).
- Objective actionability metrics (goal and next-action prediction accuracy).
- The use of an external LLM as a judge to scale evaluation.
Empirical Validation: Experiments conducted on autonomous driving motion planning across ten scenarios (rational, irrational, and occlusion) using five different LLMs.

Results
The evaluation, conducted on autonomous driving scenarios with up to five agents, yielded the following findings:

Perceived Correctness: AXIS improved the perceived correctness of explanations by at least 7.7% across all tested models compared to a "ModelOnly" baseline (where the LLM receives only the initial context without simulation).
Goal Prediction: AXIS improved goal prediction accuracy by 23% for four of the five models tested.
Action Prediction: Results were comparable or improved for action prediction across four models.
Interrogation Utility: Analysis of prompt usage revealed that models utilizing remove and whatif prompts (which facilitate counterfactual reasoning) achieved higher correctness scores. Models that failed to use these prompts (e.g., DeepSeek-V3) performed poorly.
Limitations in Reasoning: While effective in rational scenarios, AXIS struggled with irrational agent behavior and occluded factors. LLMs tended to assume rational adherence to rules and failed to propose interventions that forced irrational behavior or correctly inferred occluded agents, highlighting current limitations in LLM reasoning regarding partial observability.
Model Variance: No single LLM dominated all metrics. Stronger reasoning models (e.g., o4-mini, DeepSeek-R1) provided high perceived correctness but sometimes lower actionability scores, potentially due to excessive context "confusing" the evaluator LLM.

Significance and Claims
The paper claims that AXIS represents a significant step forward in explainable MAS by shifting from static, post-hoc feature attribution to dynamic, interactive counterfactual reasoning. The authors assert that their interrogation-synthesis process allows for the selection of the most relevant causal factors, resulting in explanations that are more accurate and actionable for downstream tasks like goal prediction.

However, the authors maintain a modest stance regarding their claims:

They acknowledge that the use of an LLM as an evaluator introduces potential biases, particularly in subjective preference scoring, and call for future human-subject studies.
They note that the method's success is currently bounded by the capabilities of the underlying LLMs, specifically regarding reasoning about irrationality and partial observability.
They highlight that the lack of standardized baselines in the field limits direct comparisons, though they established a "ModelOnly" baseline to contextualize their improvements.
The work is presented as a proof-of-concept for integrating simulation and language models, with scalability to larger agent counts and discrete domains remaining an open question for future research.