💬 NLP

On the Importance and Evaluation of Narrativity in Natural Language AI Explanations

This paper argues that Natural Language AI explanations should adopt a narrative structure to better support human understanding, proposes seven new automatic metrics to evaluate narrative quality across four key dimensions, and introduces generation rules to improve the explanatory power of XAI outputs beyond standard feature lists.

Original authors: Mateusz Cedro, David Martens

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mateusz Cedro, David Martens

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand why a mysterious machine decided to reject your loan application.

The Old Way (Descriptive Explanations):
The machine hands you a printed list. It says:

"Credit Amount: Bad."
"Age: Good."
"Housing: Bad."
"Sex: Neutral."

This is like a grocery receipt. It tells you what items you bought, but it doesn't tell you why you bought them or how they fit together. It's a static list of facts. You have to do all the heavy lifting in your brain to figure out the story: "Oh, I guess the high credit amount was the problem, but my age helped a little?" It's dry, repetitive, and hard to follow.

The New Way (Narrative Explanations):
The machine gives you a short story instead. It says:

"The machine looked at your high credit request first, which sounded risky. But then, it saw your young age, which acted like a safety net, pulling the decision back toward 'good.' Your renting status softened the financial worry a bit, and your gender added a tiny positive touch. Because these good factors outweighed the initial risk, the machine decided to say 'Yes'."

This is a story. It has a beginning, a middle, and an end. It explains why things happened, not just what happened.

The Problem the Paper Solves

The authors, Mateusz Cedro and David Martens, noticed that most AI explanations today are stuck in "Receipt Mode" (the list). They want to move AI to "Story Mode."

But here's the tricky part: How do you measure if an AI is telling a good story?

Currently, computers use standard math to check text quality. They look at things like:

Perplexity: How predictable is the text? (Is it boringly repetitive?)
Word Count: How many different words are used?

The authors argue these tools are broken for stories.

The "Gibberish" Loophole: You can trick these tools by writing nonsense that repeats the same words over and over. A computer might think, "Wow, this text is very predictable and uses common words, so it must be high quality!" But a human would say, "This makes no sense."
The "Receipt" Loophoe: A boring list of facts might score well on these tests because it's grammatically correct, even though it fails to tell a story.

The Solution: The "Story Detective" Tools

The authors created seven new tools (metrics) specifically designed to catch a real story. They look for four "superpowers" that make a narrative work:

The Thread (Continuous Structure):
- Analogy: Imagine a ball of yarn. In a story, if you pull one thread, the whole ball moves. In a list, the threads are cut and lying on the floor.
- The Test: The authors shuffle the sentences. If the text falls apart and becomes confusing, it was a real story (good!). If the text still makes sense, it was just a list (bad!).
The Cause-and-Effect Chain:
- Analogy: A domino effect. "Because the first domino fell, the second one fell."
- The Test: The computer scans for words like "because," "therefore," and "as a result." It checks if the story actually explains why something happened, rather than just stating facts.
The Flow (Linguistic Fluency):
- Analogy: A smooth river vs. a rocky path.
- The Test: Does the text flow naturally, or does it sound like a robot reading a manual? It checks if the text avoids repeating the same phrases (like "The feature is... The feature is...").
The Vocabulary (Lexical Diversity):
- Analogy: A painter using a full palette of colors vs. just using gray.
- The Test: Stories use many different verbs (action words) to describe how things change. Lists just use static nouns. The new tools check if the AI is using active, varied language.

What They Found

The authors tested these new tools on several AI systems that generate explanations.

The Old Tools Failed: Standard computer tests couldn't tell the difference between a helpful story and a boring list. Sometimes, the boring list even scored higher!
The New Tools Succeeded: The seven new "Story Detective" tools easily separated the good stories from the bad lists. They could tell which AI was actually explaining why and which was just reading a grocery receipt.
The Fix: They also gave the AI a set of "Story Rules" (e.g., "Start with a beginning, use connecting words, explain the cause"). When they gave these rules to the AI, the AI started writing much better stories.

The Big Takeaway

If we want people to trust AI, we can't just give them a spreadsheet of data. We need to give them a story.

This paper is like a guidebook for building AI Storytellers. It tells us:

Lists are not enough; we need narratives.
Old computer tests are too dumb to spot a good story.
We have new, smarter tests to measure storytelling.
If we teach AI to follow these story rules, the explanations will be much easier for humans to understand.

In short: Don't just tell the AI what happened; ask it to tell you the story of why it happened.

1. Problem Statement

The field of Explainable AI (XAI) aims to make machine learning models interpretable. While Natural Language Generation (NLG) has been integrated into XAI to make explanations more accessible, current methods largely produce static, descriptive lists of feature importances (e.g., "Feature A contributed X, Feature B contributed Y").

The Gap: These "XAI Descriptions" explain what influenced a prediction but fail to explain why the prediction occurred. They lack the rhetorical structure (cause-and-effect, continuity) required for human reasoning.
The Evaluation Failure: Standard Natural Language Processing (NLP) metrics (e.g., Perplexity, BLEU, ROUGE, distinct n-gram ratios) are insufficient for evaluating these explanations. The authors demonstrate that these metrics can be "gamed" by tautological, meaningless text that scores well on fluency and diversity but conveys no explanatory value. Furthermore, they fail to distinguish between a coherent narrative and a disjointed list of facts.

2. Methodology

A. Conceptual Framework: Descriptive vs. Narrative

Drawing from social sciences and linguistics, the authors define XAI Narratives as structured, story-like representations that emphasize relationships between features. They identify four defining properties of a narrative explanation:

Continuous Structure: Events are linked logically (beginning, middle, end) using connectives, rather than being a list of independent facts.
Cause-Effect Mechanisms: The explanation explicitly identifies causal links (e.g., "due to," "as a result") to clarify why a prediction was made.
Linguistic Fluency: The text avoids repetitive, list-like phrasing and flows naturally.
Lexical Diversity: The text uses a broad vocabulary, particularly active verbs, to describe dynamic interactions rather than static attributes.

B. Proposed Metrics (The 7 Narrativity Metrics)

To quantify these properties, the authors propose seven automatic metrics. Most rely on a Context Progression Rate ( $r$ ) derived from fitting a shifted exponential curve to the cumulative perplexity of the text as sentences are added. This measures how quickly the text becomes predictable as context accumulates.

Continuous Structure Rate (CSR, $\uparrow$ ): Measures the increase in perplexity when sentence order is shuffled. Narratives should show a high increase (disruption), while lists show a low increase.
Diversity-adjusted Context Progression Rate (DCPR, $\downarrow$ ): The context progression rate ( $r$ ) weighted by lexical diversity (distinct bigrams). Penalizes low diversity (repetition) even if the text is predictable.
Connectives-adjusted Context Progression Rate (CCPR, $\downarrow$ ): The rate $r$ weighted by the density of discourse connectives. Rewards texts where predictability is driven by logical connectors.
Cause-Effect-adjusted Context Progression Rate (CECPR, $\downarrow$ ): The rate $r$ weighted by the frequency of explicit causal markers.
Fluency-Diversity Rate (FDR, $\uparrow$ ): Combines perplexity and distinct bigram ratio to reward low perplexity only if accompanied by high lexical diversity (penalizing repetitive text).
Type-Token-adjusted Context Progression Rate (TTCPR, $\downarrow$ ): The rate $r$ weighted by the Type-Token Ratio (vocabulary richness).
Verb-adjusted Context Progression Rate (VCPR, $\downarrow$ ): The rate $r$ weighted by the ratio of verbs (narratives use more verbs than descriptive lists).

C. Experimental Setup

Datasets: Six tabular datasets (Compas, German Credit, Stroke, Diabetes, Fifa, Student).
Models: Random Forest classifiers trained on these datasets.
Baselines: Three state-of-the-art NLG methods were tested:
- TalkToModel: Template-based (list-like).
- Explingo: Few-shot LLM generation.
- XAIstories: Zero-shot LLM generation.
Extensions: The authors created modified versions of these methods (e.g., Explingo Narratives, XAIstories Narratives) by appending specific Narrative Generation Rules to the prompts to enforce the four narrative properties.
Evaluation: The methods were benchmarked using both standard NLP metrics and the proposed 7 narrativity metrics. Statistical significance was tested using Friedman and Nemenyi tests.

3. Key Contributions

Conceptual Distinction: Established a clear theoretical and practical distinction between "XAI Descriptions" (static lists) and "XAI Narratives" (causal, continuous stories).
Critique of Standard Metrics: Demonstrated that standard metrics (Perplexity, TTR, etc.) fail to capture narrative quality and can be satisfied by meaningless tautologies.
Novel Evaluation Suite: Introduced seven automatic metrics specifically designed to quantify the "narrativity" of AI explanations based on linguistic and social science principles.
Generation Rules: Formulated a set of problem-agnostic rules to guide LLMs in generating true XAI Narratives (focusing on continuity, causality, fluency, and diversity).
Benchmarking: Provided a comprehensive benchmark of current XAI generation methods, showing that while some LLMs produce better narratives than templates, explicit prompting rules significantly improve narrative structure in methods that lack it by default.

4. Results

Metric Performance: The proposed narrativity metrics successfully separated descriptive (template-based) methods from narrative (LLM-based) methods. Standard metrics (like Perplexity) failed to do this consistently; for instance, template-based methods often achieved the lowest (best) perplexity because of repetitive phrasing, despite being poor explanations.
Statistical Significance: Non-parametric tests (Friedman and Nemenyi) confirmed that the ranking of methods based on the proposed metrics was statistically significant. Template-based methods (TalkToModel) consistently ranked worst on narrativity metrics.
Impact of Rules:
- Explingo: Applying the narrative rules significantly improved its scores (e.g., in CSR), transforming its output from a list-like format to a more connected narrative.
- XAIstories: The original prompt already encouraged narratives, so adding the rules resulted in negligible changes, confirming the rules' effectiveness in guiding models that lack inherent narrative intent.
Tautology Test: The authors showed that a meaningless text ("The model made a prediction...") could achieve high scores on standard metrics (Dist2, TTR) comparable to real narratives, proving the necessity of structure-aware metrics like DCPR and CCPR.

5. Significance

Human-Centered XAI: The paper argues that for XAI to be truly useful, it must align with human cognitive processing, which relies on narrative structures to understand "why" events happen, not just "what" happened.
Evaluation Standard: It provides a robust, automated framework for evaluating the quality of explanation generation, moving beyond simple fluency or faithfulness to assess the explanatory structure.
Practical Guidelines: The proposed generation rules offer a practical recipe for developers to improve the interpretability of their AI systems without needing complex architectural changes, simply by refining prompt engineering.
Future Directions: The work highlights the need for evaluation beyond tabular data (e.g., images, graphs) and suggests future integration of human studies to validate if higher narrativity scores correlate with better user understanding and trust.

In summary, this paper bridges the gap between linguistic theory and AI explainability, proving that current evaluation tools are inadequate for narrative explanations and providing the metrics and rules necessary to build truly understandable AI narratives.