💬 NLP

Telling Speculative Stories to Help Humans Imagine the Harms of Healthcare AI

This paper presents a human-centered framework that uses speculative storytelling and multi-agent discussions to help users identify a broader and more diverse range of potential harms and benefits in healthcare AI, overcoming the limitations of automated risk detection methods that often overlook context-specific issues.

Original authors: Xingmeng Zhao, Dan Schumacher, Veronica Rammouz, Anthony Rios

Published 2026-04-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xingmeng Zhao, Dan Schumacher, Veronica Rammouz, Anthony Rios

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are about to build a brand-new, super-smart robot doctor. You want it to be helpful, but you're worried it might accidentally hurt someone because it doesn't understand human quirks, like how a shy person might smile when they are actually crying, or how a busy parent might look tired but not sick.

Usually, when engineers build these robots, they run strict math tests to check for errors. But the authors of this paper argue that math isn't enough. Math is like checking a car's engine with a spreadsheet; it tells you if the gears fit, but it doesn't tell you if the car will crash because the driver was distracted by a crying baby.

This paper proposes a different approach: Storytelling.

The Core Idea: "What If?" Stories

Instead of just looking at code, the researchers ask people to imagine specific, dramatic stories about how the robot doctor could go wrong. They call this "Speculative Storytelling."

Think of it like a fire drill.

The Old Way: You read a manual that says, "If the fire alarm rings, exit the building." You memorize the rule.
The New Way (This Paper): You run a simulation where the smoke starts filling the room, the lights flicker, and you have to actually act out what you would do. You realize, "Oh no, the exit door is blocked by a chair!" or "I forgot my cat!"

The researchers found that when people read these "what-if" stories, their brains wake up in a way that spreadsheets don't. They start seeing dangers they never thought of before.

How They Did It: The "World Simulator"

The team didn't just write stories by hand. They built a clever AI system that acts like a movie director and a scriptwriter working together.

The Concept: They started with a simple idea, like "An AI that checks your mood from your phone camera."
The Simulation (The Director): They told the AI to imagine a scene. It created two characters: a Doctor and a Patient. It also created a "World Agent" that tracks the environment (like the time of day, the patient's cultural background, or their mood).
The Drama (The Script): The AI ran a conversation between the doctor and the patient. It watched for moments where the AI might misunderstand the patient.
- Example: The patient is from a culture where smiling is polite even when sad. The AI sees a smile and says, "You are happy!" The doctor, trusting the AI, stops asking questions. The patient feels ignored and alone.
The Story: The system took this complex simulation and turned it into a short, 5-sentence story that anyone could read.

The Experiment: Reading vs. Not Reading

To test if this works, they ran a study with 45 people. They split them into three groups:

The Control Group: Just looked at a dry list of technical facts about the AI.
The Story-Only Group: Read the short "what-if" stories.
The Discussion Group: Read the stories and talked about them with AI agents acting as experts.

The Results were surprising:

The Control Group mostly worried about the same two things: "Will my data be stolen?" and "Will the AI make me feel bad?" (About 79% of their answers were just these two).
The Story Groups saw a much wider range of problems. They thought about:
- Cultural misunderstandings (like the smiling example).
- Political issues (like who gets access to the AI).
- Subtle harms (like a patient losing trust in their doctor).

It's like the Control Group was looking at a map and only seeing "Roads" and "Rivers." The Story Group looked at the map and saw "Roads," "Rivers," "Hidden Potholes," "Flood Zones," and "Places where the bridge is out."

Why This Matters

The paper argues that we can't wait for AI to break before we fix it. By the time a real AI causes harm, it's often too late.

This method is like building a time machine for ethics. It lets us travel into the future, see the mistakes before they happen, and fix the design while we are still in the "planning phase."

The Big Takeaway

You don't need to be a computer scientist to understand the risks of AI. You just need to be able to imagine a story.

Without stories: We see AI as a black box that either works or doesn't.
With stories: We see AI as a character in a play, interacting with real people, making mistakes, and causing ripple effects.

By using stories to "stress-test" our ideas, we can build AI that is not just smart, but also wise, fair, and safe for everyone.

1. Problem Statement

Artificial Intelligence (AI) is rapidly transforming healthcare (e.g., stress monitors, mental health chatbots), often through low-barrier development tools like "vibe coding." While this accelerates innovation, it introduces significant risks regarding bias, privacy, and unequal access, particularly when systems fail to account for diverse real-world contexts.

Current ethical assessment methods rely heavily on automation (e.g., automated risk detection, model cards, risk summaries). The authors argue that while automation scales assessment, it risks reducing human engagement in ethical reasoning. When AI automatically identifies risks, users may rely on the machine's judgment rather than developing their own critical understanding of how harms arise and who they affect. There is a lack of effective tools that help humans creatively speculate on potential harms and benefits before deployment, especially in complex healthcare settings where small errors can cause severe harm.

2. Methodology

The authors propose a human-centered framework that combines automated user story generation with structured multi-agent discussions to foster ethical foresight. The methodology consists of three main technical steps:

A. Automated User Story Generation

The system generates context-sensitive narratives through a three-step pipeline:

Mapping AI Concepts to Use-Case Scenarios:
- The system takes 38 consumer health AI concepts (sourced from Wired, PubMed, and industry docs).
- It generates structured 7-tuple specifications ( $S = \{a, u, s, x, b, h, f\}$ ) defining the AI capability, user, subject, context, expected benefit, potential harm, and failure trajectory.
Simulating Role-Playing and Environment Trajectories:
- Building on Solo Performance Prompting (SPP), the system uses a Large Language Model (LLM) to simulate interactions between an AI User (e.g., doctor), an AI Subject (e.g., patient), and a World Agent.
- The World Agent maintains environmental coherence, tracking state changes, tool usage, and non-dialogue events.
- The simulation logs detailed turn-by-turn interactions, including internal thoughts (in brackets), actions (in parentheses), and dialogue, creating a dynamic "world model" of the scenario.
Rephasing Logs into Stories:
- The structured simulation logs are rephrased by an LLM into concise, 5-sentence narratives. These stories highlight the causal chain of events, specifically focusing on how AI decisions lead to unintended harms or benefits.

B. User Study Design

A between-subjects user study ( $N=45$ ) was conducted to evaluate the framework's impact on ethical reasoning. Participants were divided into three conditions:

Control: Completed a speculative model card task directly.
Story-Only: Read text-based benefit/harm stories without discussion.
Story (Treatment): Engaged in a Story-Driven Red-Team Discussion Room, interacting with AI agents (simulating experts like doctors and ethicists) to discuss the stories before completing the model card.

The primary metric was the diversity and breadth of harms and benefits identified by participants, measured using Shannon Entropy across 17 predefined harm subtypes and 18 benefit subtypes.

3. Key Contributions

Human-Centered Speculative Framework: A novel method that uses AI not to detect risks, but to generate context-sensitive narratives that scaffold human ethical reasoning. It shifts the focus from automated risk reporting to human imagination of failure modes.
World-Modeling for Story Generation: The integration of a "World Agent" alongside role-playing agents allows for the simulation of evolving environments and state changes, producing more coherent and causally grounded stories than traditional plot-planning baselines.
Empirical Evidence of Broadened Ethical Foresight: The study demonstrates that engaging with these stories significantly improves human ability to identify a wider range of ethical risks and benefits compared to traditional documentation methods.

4. Results

A. Story Generation Quality

LLM-as-a-Judge Evaluation: The proposed "Storytelling" method outperformed traditional plot-planning baselines across all metrics (Creativity, Coherence, Engagement, Relevance, Likelihood).
- With the Gemma model, the method achieved an overall win rate of 91.21%.
- Ablation Studies: Removing the "Environment Trajectory" component caused the largest drop in performance (e.g., Coherence dropped by ~17.7 points for Gemma), proving that modeling event progression is vital for narrative logic. Removing "Role-Playing" reduced creativity and engagement.

B. User Study Findings

Increased Harm Diversity: Participants in the Story condition identified a significantly broader range of harms.
- Entropy Analysis: The Shannon entropy for harm identification increased from 2.329 (Control) to 3.701 (Story).
- Distribution: The Control group focused heavily on a few categories (Privacy, Well-being, Service Loss = 79.1%). In contrast, the Story group distributed responses more evenly across all 17 harm types.
- Novelty: Categories like "Cultural Harms," "Political/Civic Harms," and "Tech-facilitated Violence" appeared only in the Story condition.
Increased Benefit Diversity: Similarly, the Story condition showed higher entropy for benefits (3.868 vs. 2.407 in Control), surfacing less obvious benefits like "Accessibility," "Clinician Workload Relief," and "Transparency."
Qualitative Feedback: Participants reported that stories made abstract risks concrete, helping them articulate "what could go wrong" without needing prior technical expertise. They noted that stories helped surface subtle sociotechnical risks (e.g., cultural masking of emotions) often missed in formal documentation.

5. Significance

Proactive Safety: The framework moves AI safety evaluation from a reactive process (fixing deployed systems) to a proactive one (anticipating harms during design).
Human Agency: It reinforces the role of human judgment in ethical AI. Instead of relying on AI to list risks, the system uses AI to stimulate human imagination, ensuring that diverse perspectives and contextual nuances are considered.
Scalable Ethical Foresight: By using simulated agents and generated stories, the method offers a scalable way to conduct "red-teaming" and ethical exploration without the high cost of recruiting large panels of real-world experts for every new AI concept.
Bridging the Gap: The approach effectively bridges the gap between technical AI capabilities and the complex, messy realities of human healthcare contexts, particularly for underrepresented populations and cultural nuances.

In conclusion, the paper demonstrates that speculative storytelling, powered by world-modeling LLMs, is a highly effective tool for expanding the ethical imagination of developers and stakeholders, leading to more robust and inclusive healthcare AI systems.