💬 NLP

Spoiler Alert: Narrative Forecasting as a Metric for Tension in LLM Storytelling

This paper introduces the "100-Endings" metric, which quantifies narrative tension by measuring the frequency of prediction failures in a model's 100 simulated story endings at each sentence, and demonstrates that this approach not only correctly ranks human literary fiction above LLM-generated stories (unlike existing benchmarks) but also guides a structural generation pipeline that significantly improves narrative tension while maintaining EQ-Bench performance.

Original authors: Peiqi Sui, Yutong Zhu, Tianyi Cheng, Peter West, Richard Jean So, Hoyt Long, Ari Holtzman

Published 2026-04-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Peiqi Sui, Yutong Zhu, Tianyi Cheng, Peter West, Richard Jean So, Hoyt Long, Ari Holtzman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: AI Writes "Flat" Stories

Imagine you are reading a mystery novel. You turn the page, and your heart races because you don't know if the detective will catch the killer or if the hero will fall off the cliff. That feeling of "What happens next?" is called narrative tension. It's the engine that keeps you reading.

The authors of this paper discovered a weird glitch in Artificial Intelligence (AI). Even the smartest AI models are terrible at writing stories with this kind of tension. They write stories that are grammatically perfect and sound nice, but they feel like watching a movie where the ending is shown in the first five minutes. The AI reveals the secret too early, solves the mystery too quickly, and leaves the reader bored.

Even stranger, when we ask AI to judge stories (acting as a critic), the AI thinks its own flat stories are better than famous human stories from The New Yorker. It's like a robot chef thinking a bowl of plain oatmeal is a Michelin-star meal because it looks neat, while ignoring the fact that it has no flavor.

The Solution: The "100-Endings" Test

To prove that AI stories are boring, the researchers invented a new way to measure tension called the 100-Endings Metric.

Think of a story as a long road trip.

The Old Way (Rubrics): A critic reads the whole trip and gives it a grade based on how nice the car looked or how polite the driver was.
The New Way (100-Endings): Imagine you are driving, and at every single mile marker, you stop and ask a psychic: "Based on where we are right now, how will this trip end?"

The psychic tries to guess the destination 100 different times.

If the story is boring (AI): The psychic guesses the same thing 100 times. "Oh, we're going to the beach." It's obvious. The tension is zero.
If the story is exciting (Human): The psychic is confused. "Wait, are we going to the beach? Or maybe a mountain? Or did the car break down?" The psychic can't agree on an ending because the story is full of surprises and twists.

The researchers found that human stories (like those from The New Yorker) make the psychic guess wildly different endings, while AI stories make the psychic guess the same ending over and over again.

The Fix: The "Architect" Pipeline

The researchers didn't just want to point out the problem; they wanted to fix it. They realized AI fails because it tries to write the story in one giant leap, like a sprinter who runs the whole race without stopping to think.

So, they built a Story Pipeline that forces the AI to slow down and plan, like a master architect building a house.

Step 1: The Warm-up (The Tour): The AI reads a famous, high-tension human story (like a classic short story) and studies how the author built the suspense. It learns the "muscle memory" of a good story.
Step 2: The Blueprint (The Plan): Before writing a single word of the new story, the AI creates a detailed "to-do list." It decides exactly where to hide information, where to drop a bombshell, and how to keep the reader guessing. It's like drawing a map that says, "Here, we will pretend the hero is safe, but actually, he's in danger."
Step 3: The Build (The Writing): Finally, the AI writes the story, strictly following the blueprint.

The Result:
When they used this "Architect" method, the AI stories became much more like human stories. The "100-Endings" test showed that the AI could finally keep the reader guessing until the very last page. The stories weren't just grammatically correct anymore; they were actually compelling.

The Takeaway

This paper teaches us two big lessons:

AI is a mimic, not a master: Current AI is great at copying the style of writing (the words), but it doesn't understand the soul of storytelling (the suspense). It solves problems too fast because it's programmed to be helpful, but stories need problems to stay unsolved for a while.
Structure beats size: You can't just make the AI bigger or smarter to fix this. You have to give it a better structure, like a blueprint. If you force the AI to plan its "surprises" ahead of time, it can write stories that actually make us feel something.

In short: AI can write the words, but it needs a human-like plan to write the magic.

1. Problem Statement

Large Language Models (LLMs) currently struggle to generate compelling, human-like stories, particularly regarding narrative tension. While existing evaluation benchmarks (e.g., EQ-Bench) use LLM judges to rate stories based on rubrics (prose quality, character authenticity, etc.), these metrics fail to capture the structural mechanics of tension.

The Paradox: Current rubric-based judges often rank zero-shot LLM stories higher than professional human fiction (e.g., The New Yorker), despite human readers consistently finding LLM stories flat, clichéd, and lacking suspense.
The Gap: Existing metrics either omit tension entirely or measure it superficially (e.g., counting emotional words). They fail to operationalize tension as the structural management of information asymmetry and delayed resolution. Consequently, LLMs are not penalized for resolving plot points prematurely, which collapses narrative tension.

2. Methodology

A. The 100-Endings Metric

The authors introduce a new metric designed to measure narrative tension by quantifying unpredictability.

Core Concept: Tension is defined as the difficulty of predicting a story's ending given the text so far. A story with high tension resists easy prediction; a story with low tension (flat) is easily anticipated.
Procedure:
1. Sentence-Level Forecasting: For a story with $N$ sentences, the metric iterates through sentence boundaries from 10% to 99% of the text.
2. Generation: At each position $i$ , a generation model (Qwen3-32B, $T=1.2$ ) is prompted to generate 100 candidate endings based only on the text up to sentence $i$ .
3. Judgment: A separate judge model (Qwen3-8B, $T=0$ ) evaluates each of the 100 generated endings against the actual ground truth continuation. The judge determines if the generated ending matches the general plot trajectory (Y/N).
4. Calculation:
  - No-Rate: The fraction of the 100 predictions that fail to match the ground truth. A high no-rate indicates high unpredictability (high tension).
  - Mean No-Rate: The average no-rate across all evaluated positions.
  - Late-Stage No-Rate: The average no-rate specifically at positions $\ge 80\%$ through the story (measuring if tension holds until the end).
  - Inflection Rate: A geometric measure of how frequently the no-rate curve reverses direction, tracking plot twists and revelations.
  - Post-Spike Retention: Measures how much tension is retained after a peak, indicating whether the story sustains uncertainty or resolves it immediately.

B. The Narrative Generation Pipeline

To address the lack of tension in LLM outputs, the authors propose a structured generation pipeline grounded in narratology:

Step 0 (Warmup Analysis): The model analyzes a reference story (either a canonical short story or a New Yorker piece) to extract a 7-beat structural template, focusing on narrative techniques and information withholding.
Step 1 (Beat Adaptation): The model adapts this template to a specific story idea, creating a "thick" intermediate plan. This plan explicitly defines narrative functions, what information to reveal/withhold, and how to escalate stakes.
Step 2 (Story Generation): The model generates the final story, strictly following the structural constraints of the intermediate plan to maintain tension mechanics.

3. Key Contributions

A New Metric (100-Endings): A structural, forecasting-based metric that operationalizes tension as unpredictability, successfully distinguishing professional fiction from LLM outputs where rubric-based judges fail.
Empirical Evidence of LLM Failure: Demonstrated that top-tier LLMs produce stories with the lowest narrative tension of any corpus evaluated (even lower than amateur human writing), despite scoring highest on standard benchmarks like EQ-Bench.
A Structural Solution: Proved that injecting literary theory-driven structural constraints (the pipeline) significantly improves narrative tension, closing roughly half the gap between zero-shot LLMs and professional fiction.

4. Results

Evaluation of the Metric

Ranking Accuracy: The 100-Endings metric correctly ranks New Yorker stories (Mean No-Rate: 0.765) significantly higher than top-10 LLMs (Mean No-Rate: 0.630) and amateur human writing.
EQ-Bench Discrepancy: On EQ-Bench, LLMs scored ~82/100, while New Yorker stories scored ~78/100. The 100-Endings metric reversed this, showing LLMs have far lower tension.
Late-Stage Performance: The gap is most severe in the final acts of stories. New Yorker stories maintain a late-stage no-rate of 0.607, whereas LLMs drop to 0.215, indicating LLMs resolve plot threads too early.

Evaluation of the Pipeline

Tension Improvement: Applying the pipeline to three frontier models (Claude Sonnet 4.6, Opus 4.6, GPT-5.2) significantly increased all tension metrics.
- Example (Sonnet 4.6): Mean No-Rate increased from 0.606 (vanilla) to 0.747 (pipeline). Late-stage no-rate more than doubled (0.139 $\to$ 0.339).
Benchmark Performance: The pipeline improved the EQ-Bench score for Sonnet 4.6 to 85.03 (ranking #1 on the leaderboard), proving that structural constraints can enhance both tension and general quality scores simultaneously.
Case Study: In a romance prompt, the vanilla model resolved the romantic tension at 64% of the story (no-rate dropped to 0.29). The pipeline version sustained high unpredictability (no-rate > 0.85) until the end by utilizing subtext and delaying closure.

5. Significance and Conclusion

Beyond Scale: The paper argues that simply increasing model scale does not solve the problem of narrative tension; explicit modeling of structural constraints derived from literary theory is required.
Critique of Current Evaluation: It highlights a critical flaw in "LLM-as-a-Judge" paradigms: they favor surface-level fluency and resolution-oriented styles, missing the "visceral" experience of tension that defines high-quality fiction.
Humanistic AI: The work advocates for "literary-theory-informed" AI, bridging NLP with narratology to build systems that understand the mechanics of storytelling (information asymmetry, delayed resolution) rather than just statistical probability.
Future Directions: The authors suggest that while tension is a vital component, true "narrative machines" will require further work on tacit knowledge, cultural context, and simulating the editorial ecosystems that produce professional literature.

In summary, this paper provides a rigorous framework for measuring and improving the most elusive aspect of creative writing—tension—demonstrating that current AI is structurally incapable of sustaining suspense without deliberate, theory-driven intervention.