💬 NLP

Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives

This paper introduces NCP-Bench, a benchmark of 100 narrative environments designed to evaluate Long-Horizon Consistency in interactive storytelling, revealing that even state-of-the-art LLMs struggle significantly with maintaining logical commitment preservation and narrative integrity against unconstrained user interventions.

Original authors: Yingpeng Ma, Jianhao Yan, Bei Shi, Ka Hou Kam, Runnan Wang, Xuebo Liu, Yulong Chen, Yue Zhang, Derek F. Wong

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yingpeng Ma, Jianhao Yan, Bei Shi, Ka Hou Kam, Runnan Wang, Xuebo Liu, Yulong Chen, Yue Zhang, Derek F. Wong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a massive, endless role-playing game where the story never ends, and you can do absolutely anything you want. In this world, you might decide to punch a dragon, steal a king's crown, or convince a villain to become a baker. The magic that makes this possible is a type of super-smart computer brain called a Large Language Model (LLM). Think of these models as incredibly talented improvisational actors who have read almost every book ever written. They are great at sounding natural, creating atmosphere, and keeping the conversation flowing. But there's a catch: while they are excellent at sounding like they know the story, they often struggle to actually remember the rules of the world they are building. If the computer says "the door is locked," a few minutes later it might let you walk right through it as if it were open, or it might forget that you already found the key. This is a big problem for game designers and storytellers who want a world that feels real and consistent, no matter how wild the player gets.

This paper, titled "Can LLM Agents Stick to the Script?", dives deep into this exact problem. The researchers created a special testing ground called NCP-Bench (Narrative Commitment Preservation Benchmark) to see if these AI storytellers can keep their promises over a long, messy game session. They set up 100 different story worlds based on famous movie plots, like Iron Man or The Bourne Identity. In each world, the AI acts as the "Game Master," responsible for running the show, while a second AI acts as a "troublemaker" player, trying to skip ahead, break the rules, or force the story to go in impossible directions. The goal was to see if the Game Master could stay consistent—keeping track of facts, remembering what happened, and sticking to the plot's required milestones—even when the player was trying to break the game.

The results were a bit of a reality check for the AI community. The study found that even the most advanced, state-of-the-art AI models are surprisingly bad at sticking to the script when things get complicated. While these models can write beautiful, fluent, and exciting stories, they often lose the plot when the game goes on for too long. In the tests, the best-performing model (a version of GPT-5.2) managed to survive without making a logical error for only 42% of the time after just 20 turns of interaction. As the game went on, the survival rate dropped sharply. By the time the game hit the 100-turn limit, almost no model could finish the whole story without contradicting itself or forgetting a major plot point.

The researchers discovered that the most common mistake wasn't that the AI was being mean or ignoring the player; it was that the AI simply forgot the facts it had established earlier. About 40% to 68% of the failures were "fact conflicts," where the AI would say something that directly contradicted what it said ten minutes ago. For example, if the story established that a character was injured, the AI might later describe them running a marathon without a scratch. Even when the AI tried to be helpful and let the player skip boring parts of the story, it often did so by rewriting history in a way that broke the logic of the world.

Interestingly, the paper also tested a special kind of AI designed with a "memory" system to help it remember things better. While this memory system helped the AI last a little longer in the game, it didn't solve the problem. In fact, the memory system sometimes made the AI worse at listening to the player's specific actions, because it was too busy summarizing the story into big chunks and missing the small details. The study suggests that simply making AI smarter or giving it more memory isn't enough; we need a new way to teach these models how to treat the story's rules as unbreakable laws, not just suggestions.

In short, the paper concludes that while our current AI storytellers are fantastic at improvising and sounding cool, they are currently terrible at playing the long game. They can't reliably stick to the script when the player tries to break the rules. The authors hope that by providing this benchmark, other researchers can build better tools to fix this, so that one day, we can have truly endless, consistent, and magical interactive stories where the world makes sense no matter what we do.

Technical Summary: Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives

Problem Statement

The rapid advancement of Large Language Models (LLMs) has enabled open-ended, fluid interactive storytelling, yet existing systems struggle with a fundamental requirement: maintaining long-horizon logical consistency and narrative integrity against unconstrained user interventions. While LLMs can generate fluent and atmospheric stories, they often fail to preserve "narrative commitments"—the logical promises made by the game design or the causal history of the session.

In interactive narratives, a narrator agent must guide a player toward specific milestones while respecting world constraints. However, under free-form user interventions (e.g., a player insisting they already possess a key that was locked away, or attempting to skip mandatory plot sequences), current LLMs frequently resort to implicit retroactive rewriting of past events or allow impossible jumps. This creates a tension between unbounded user agency and rigid narrative goals. Unlike open-domain chatbots designed for chitchat, narrative orchestrators operate within a goal-directed structure where the "plot" functions as a set of rigid logic constraints (world-state facts, causal dependencies, and mandatory event sequences) that must be satisfied. Current evaluations, which are largely preference-based (focusing on perceived coherence or engagement), fail to explicitly detect or compare these logical failures.

Methodology

Narrative Commitment Preservation (NCP)

The authors formalize the challenge as Narrative Commitment Preservation (NCP). An NCP agent is defined by its ability to maintain:

An explicit fact ledger recording established facts.
An explicit set of non-optional narrative commitments (rules regarding invariants, ordering, and achievements).

The interaction is modeled as a stateful process where the agent generates a response, and an automatic procedure extracts state updates to validate consistency. The goal is to ensure the agent responds appropriately to users without violating the logical promises of the game design.

NCP-Bench: The Benchmark

To enable reproducible research, the authors introduce NCP-Bench, a benchmark comprising 100 narrative environments derived from movie synopses (specifically from the CMU Movie Summary Corpus).

Data Construction: The dataset is curated to ensure diversity across 18 genres (Action, Sci-Fi, Comedy, etc.) and includes a specific player character perspective to prevent "God's-eye" information leakage.
Narrative Specification: Each environment is structured as a tuple $\langle F_0, C, R \rangle$ $⟨ F_{0}, C, R ⟩$ :
- $F_0$ (Initial Fact Ledger): The world state at the start, containing only facts observable to the player.
- $C$ (Commitment Set): A fixed set of rules including Invariants (conditions that must remain true), Ordering (event sequences), and Achievements (goals to be met).
- $R$ (Reference Trajectory): An ordered sequence of plot steps (nodes) used to track progress.
Evaluation Framework: The framework decouples the narrator agent from the auditing mechanism.
- Adversarial Player Agent: Generates free-form, first-person inputs designed to stress-test the narrator (e.g., attempting to bypass constraints, accusing characters prematurely, or rushing the plot).
- Auditor: A fixed, prompt-based LLM auditor that performs a turn-level check:
  1. Conflict Check: Verifies the narrator's response against the active fact ledger, commitments, and player input.
  2. Fact Update: Extracts and commits state changes (added or negated facts) only if no conflict is found.
  3. Trajectory & Commitment Check: Updates progress along the reference trajectory and tracks commitment satisfaction.
- Termination: An episode ends in Conflict (logical violation), Survival (reaching 100 turns without conflict), or Success (satisfying all achievement commitments without conflict).

Key Contributions

Formalization of NCP: The paper defines Narrative Commitment Preservation as a distinct, checkable task, moving beyond subjective coherence metrics to explicit logical constraint satisfaction.
NCP-Bench Dataset: A curated benchmark of 100 movie-derived scenarios with structured specifications (facts, commitments, trajectories) that allow for automated, fine-grained auditing of narrative consistency.
Evaluation Protocol: A rigorous, reproducible framework that separates the agent under test from the auditing logic, utilizing adversarial player agents to systematically probe for consistency failures.
Comprehensive Analysis: The study provides the first large-scale empirical evidence of the "long-horizon consistency gap" in state-of-the-art LLMs, categorizing failure modes and analyzing the impact of memory architectures and input types.

Experimental Results

The authors evaluated six state-of-the-art LLMs (including GPT-5.2, GPT-4o-mini, DeepSeek-V3.2, Qwen3-235B-A22B, Kimi-K2.5, and Grok-4.1-Fast) on NCP-Bench.

Long-Horizon Consistency Gap: Even the best-performing model, GPT-5.2, achieved only a 42% survival rate after 20 turns. Survival rates for other models dropped to near zero well before the 100-turn limit.
Commitment Satisfaction: Only isolated runs satisfied all achievement commitments within the 100-turn limit. The highest percentage of satisfied commitments across models was roughly 13.42% (DeepSeek-V3.2).
Failure Modes: Fact conflicts were the dominant failure mode, accounting for 40% to 68% of interactions across models. Other modes included commitment violations (breaking ordering/invariants) and ignoring player input.
Qualitative Analysis: Failures were categorized into four types:
1. Hallucination & Factual Contradiction: Contradicting established world states.
2. Triggering Unknown Facts: Prematurely revealing hidden plot information.
3. Forcibly Changing Reality: Rewriting established facts to erase player actions.
4. Ignoring User Input: Failing to acknowledge or address player actions.
Memory Augmentation: Evaluating HiAgent (a hierarchical memory architecture) showed it extended interaction length and reduced commitment conflicts but failed to improve overall success rates and increased player-input conflicts, suggesting memory compression may obscure local intent.
Adversarial vs. Natural Inputs: While natural (cooperative) inputs led to longer interactions and fewer conflicts, they still failed to solve the task, confirming that the core challenge of long-horizon commitment preservation persists regardless of player intent.

Significance and Claims

The paper claims that linguistic fluency does not translate into reliable logical commitment preservation. The results demonstrate that current LLMs lack the mechanisms necessary for maintaining consistent world states under adversarial pressure, even when the text appears plausible.

The authors position NCP-Bench not as a measure of creative writing quality, but as a foundational testbed for the logical scaffold required for interactive narrative. They argue that logical consistency is a necessary (though not sufficient) condition for compelling stories. The benchmark serves as a stepping stone for developing agents that can honor persistent obligations under open-ended interaction, a capability relevant beyond narrative to domains like coding agents, instructional systems, and multi-agent coordination. The work emphasizes that without solving this baseline consistency problem, more advanced narrative features (dramatic tension, emotional resonance) cannot be reliably pursued.