Technical Summary: XstrAI – A Multi-Agent System for Audience-Aware XAI Narratives

1. Problem Statement

Local feature-attribution methods, such as SHAP, provide essential instance-level evidence for model predictions but often fail to serve diverse audiences effectively. In high-stakes domains like medical AI, the same technical explanation must be communicated to clinicians, patients, and data scientists, each with distinct expertise, goals, and risks of misinterpretation.

Current approaches to generating natural language explanations from these attributions suffer from several limitations:

• Audience Agnosticism: Feature-attribution methods are designed to be audience-agnostic, leaving the adaptation of evidence to the consumer.
• Weak Grounding: Direct verbalization via Large Language Models (LLMs) or single-prompt role conditioning often results in outputs that are persuasive but unfaithful to the underlying model evidence.
• Causal Conflation: LLMs frequently conflate feature attribution with causal language, producing hallucinations or unsupported claims.
• Monolithic Generation: Existing Narrative-XAI systems often treat generation as a single, unstructured step, entangling evidence selection, audience adaptation, and validation, which limits inspectability and control.

The core challenge is to generate narratives that are not only fluent but also strictly grounded in fixed attribution evidence while being semantically and stylistically appropriate for specific stakeholder groups.

2. Methodology: The XstrAI Framework

The authors propose XstrAI, a multi-agent framework that decouples the representation of evidence from its communication. The system treats local explanations as immutable facts and structures their transformation into stakeholder-specific narratives through a controlled pipeline.

2.1 Core Architecture

The framework operates on three stakeholder profiles: Clinician, Patient, and Data Scientist. The architecture consists of the following components:

1. ExplanationCard (Input Representation):

   • A machine-readable, structured artifact serving as the "source of truth" for every prediction case.
   • It contains case information, model output, SHAP attributions, feature metadata, generation constraints, and the target stakeholder profile.
   • This card ensures that the underlying evidence remains fixed and identical across all audience pathways; differences in output arise solely from communicative choices, not evidence alteration.

2. Multi-Agent Pipeline:
   For each stakeholder pathway, three specialized LLM agents operate sequentially:

   • Framer Agent (Planning): Receives the ExplanationCard and generates a FramingTrace. This artifact defines the narrative strategy, including organization, feature presentation order, tone, required cautions, and operational instructions. It adapts the evidence to the stakeholder (e.g., prioritizing readability and non-alarming language for patients; retaining explicit attribution directions for data scientists).
   • Narrator Agent (Generation): Receives the ExplanationCard and the FramingTrace. It generates the candidate narrative, constrained by the factual evidence in the card and the communicative strategy in the trace. It must preserve attribution meaning and avoid causal or diagnostic claims unless permitted.
   • Reviewer Agent (Validation): Acts as an internal quality-control module. It evaluates the candidate narrative against five dimensions:
     1. Card Grounding: Anchored to evidence; no fabrication.
     2. SHAP Faithfulness: Direction and relative magnitude of contributions are preserved.
     3. Epistemic Framing: Describes model behavior, not diagnosis or causation.
     4. Communication Safety: No treatment, prognosis, or lifestyle advice (unless explicitly allowed).
     5. Audience Fit: Register matches the target stakeholder.
   • Bounded Revision Loop: If the Reviewer detects inconsistencies, it triggers a revision. The loop routes the task back to the Framer, the Narrator, or both. The system allows a maximum of two revision rounds (yielding up to three attempts per case).

2.2 Experimental Setup

• Datasets: Diabetes and Stroke Prediction datasets.
• Models: Random Forest classifiers with SHAP (TreeExplainer) for local attributions.
• Baselines: The system was evaluated against 11 configurations, including:
  • Explingo: A re-implementation of a state-of-the-art loop-based narrator.
  • Incremental Baselines (B1–B4): Variants of XstrAI that ablate specific components (e.g., single-shot generation, role prompting, revision loops, rule-based verification).
• LLMs: Qwen3.6:35B and Gemma4:31B were used for generation and review, with cross-family pairing to avoid bias.

3. Evaluation Protocol

The evaluation employs a dual-regime approach to measure both the fidelity of the text to the evidence and its appropriateness for the audience.

3.1 Intra-Narrative Evaluation

Measures the properties of the generated text itself:

• Form: Readability (Flesch–Kincaid), lexical richness (MATTR), verbosity, and repetition patterns.
• Content: Agreement with the ExplanationCard on feature values (Value Agreement), contribution direction (Sign Agreement), and feature ordering (Rank Agreement).
• Semantics: Semantic accuracy measured by projecting narrative sentences into reference corpora specific to each stakeholder (medical papers for clinicians/data scientists, lay articles for patients) using Sentence-BERT embeddings.

3.2 Extra-Narrative Evaluation

Measures how the text is received by external evaluators:

• LLM-as-a-Judge: Three independent LLM judges (Claude, GPT, Gemini) performed:
  • Re-identification Analysis: Assigning anonymized narratives to their intended audience.
  • Competitive Ranking: Ranking all 14 system outputs for each target audience, aggregated via Bradley-Terry/Elo scores.
• Human Evaluation: Surveys with 15 participants (5 per group: clinicians, data scientists, patients) rating narratives on Understandability, Clarity, Usefulness, Trust, and Cognitive Load.

4. Key Results

• Audience Adaptation: XstrAI narratives were consistently assigned to their intended audience by independent judges (100% accuracy in re-identification), significantly outperforming baselines.
• Performance vs. Baselines:
  • Clinician & Patient: XstrAI was preferred over all baselines in LLM rankings.
  • Data Scientist: XstrAI performed competitively but ranked slightly behind single-prompt baselines (B2) that maximize raw SHAP density.
• Fidelity and Structure:
  • XstrAI maintained high Sign Agreement (preserving SHAP direction) while allowing Rank Agreement to shift for patient narratives. Specifically, patient narratives systematically promoted "actionable" features (e.g., BMI) over non-actionable ones (e.g., age), prioritizing intervention-oriented communication over strict SHAP ranking.
  • XstrAI achieved lower repetition rates and higher lexical diversity compared to baselines, avoiding the template-based generation seen in systems with revision loops but no planning agent.
• Human Feedback: Clinicians rated XstrAI narratives highest in Understandability (4.7/5) and Trust (4.1/5). Patients found them highly understandable (4.2/5) with low cognitive load (2.2/5), though Trust scores were more moderate (3.0/5).

5. Significance and Contributions

The paper claims three primary contributions:

1. Novel Architecture: A multi-agent framework that decouples evidence representation, stakeholder-specific planning, linguistic realization, and validation into inspectable stages, treating local explanations as fixed evidence.
2. Dual Evaluation Protocol: A methodology that disentangles intra-narrative fidelity (truth to evidence) from extra-narrative appropriateness (fit for audience), assessed against reference corpora, multi-family LLM judges, and human subjects.
3. Empirical Validation: Evidence that audience-conditioned generation requires more than simple role prompting; a structured pipeline with planning and validation is necessary to produce narratives that are both faithful to the model and semantically aligned with diverse stakeholders.

The authors emphasize that XstrAI demonstrates that stakeholder adaptation can legitimately reshape explanation structure (e.g., re-ordering features for patients) without compromising the core evidence, provided the transformation is controlled and validated. The work suggests that safe and effective XAI deployment requires moving beyond monolithic generation toward structured, verifiable communication pipelines.