Technical Summary: NARRA-Gym for Evaluating Interactive Narrative Agents

Problem Statement

Interactive narrative tasks require Large Language Models (LLMs) to sustain a coherent, evolving story world while adapting to a user over multiple turns. This setting demands a combination of capabilities—generation, memory, planning, and personalization—that must operate simultaneously under continuous interaction. However, existing benchmarks for this domain are limited. Standard evaluations often focus on static, single-turn prompts, isolated story generations, or post-hoc ratings. Consequently, current benchmarks fail to assess whether models can jointly manage story generation, long-context state, pacing, character simulation, empathic personalization, and story-grounded artifacts. A model may produce fluent individual responses yet fail to maintain consistency, manage long-term state, or adapt to a user's emotional trajectory, leading to broken interactions.

Methodology: NARRA-Gym

To address these gaps, the authors introduce NARRA-Gym, an executable evaluation environment that transforms a sparse emotional seed into a complete interactive story episode. The system logs the full model-in-the-loop trajectory, including story construction, memory updates, planning, pacing interventions, and optional artifact synthesis.

Environment Construction

The environment orchestrates models through a complete episode pipeline consisting of two main phases:

1. Story Construction (Narrative Architect): Upon receiving a user's emotional seed (free-text description of mood/situation) and profiling answers, the system decomposes story creation into five logged stages:
   • Story Foundation: Establishing title, premise, theme, and protagonist objectives.
   • Setting Construction: Translating emotional context into a concrete world and scene frame.
   • Character Construction: Building the cast, including backstories, personalities, and speech styles.
   • Act Structure: Drafting a multi-act outline, optionally refined via a critic-then-refiner loop.
   • Opening Scene: Generating opening prose, dialogue, branching choices, and hidden elements.
2. Continuous Interaction Loop: Once initialized, the episode enters a turn-level loop coordinated by four agent components:
   • Memory Agent: Assembles context from three layers: user profile (needs, tone, boundaries), story state (current goals, tensions, clues), and user journey (decisions, emotional trajectory). It updates these layers at different frequencies to avoid simple context stuffing.
   • Pacing Agent: Monitors whether the plot advances. It employs a three-layer defense against stagnation: runtime pacing controls (escalating from gentle encouragement to mandatory shifts), a stagnation detector (checking for repetitive patterns), and a post-generation structure guard that injects narrative shifts if the model output lacks progression.
   • Planning Agent: Reflects on the story state to provide structured guidance (unresolved tensions, user interests, advancement strategy) for subsequent turns.
   • Artifact Agent: Generates functional, story-grounded HTML/CSS/JavaScript artifacts (e.g., letters, maps, ciphers) when the narrative requires tangible props. A novelty scoring system prevents repetition by comparing artifact tags against recent history.

Evaluation Protocol

The authors employ two complementary protocols to evaluate nine frontier LLMs:

1. Controlled LLM-as-Judge Sweep: Each of the nine generator models is run on the same eight predefined personas (e.g., a grieving father, an overwhelmed new mother), yielding 72 complete episodes. These are scored by three independent judge models across an 11-dimensional rubric covering story quality (Relevance, Coherence, Empathy, Surprise, Engagement, Complexity, Character Shaping) and user experience (Satisfaction, Perceived Quality, Process Helpfulness, Reuse Intent).
2. Human Preference Evaluation: 12 participants engage in customized interactive experiences (rather than fixed personas) and rate anonymized model outputs. Ratings are converted to within-group rankings and aggregated using a Plackett–Luce model to estimate latent utilities for each model.

Key Contributions

1. An Executable Benchmark Environment: NARRA-Gym defines an interactive setting that jointly tests creative story generation, long-context state/pacing management, character simulation, empathic personalization, and artifact generation within a single interaction loop.
2. A Modular Narrative-Agent Pipeline: The authors implement a staged pipeline featuring multi-resolution memory, reflection-guided planning, anti-stagnation control, novelty-constrained artifact synthesis, and fail-soft structured generation. Every component is logged for inspection, allowing for the diagnosis of specific failure modes.
3. A Comparative Evaluation Protocol: The paper provides a human rating protocol with within-group rank aggregation and LLM-judge protocols. This setup exposes failures difficult to observe in static datasets, such as resistance-sensitive personalization and long-horizon coherence.

Results

The evaluation of nine frontier LLMs reveals substantial variation across models, personas, and dimensions:

• Performance Variability: Claude Sonnet 4.6 achieved the highest mean aggregate scores (StoryQ: 3.90, UX: 3.86) and led in all 11 fine-grained dimensions, followed by Claude Opus 4.6. Models like DeepSeek V4, GPT-5.4, and DeepSeek V3.2 were closely clustered in StoryQ but exhibited distinct profiles (e.g., DeepSeek V4 was stronger in character shaping, while GPT-5.4 excelled in empathy).
• Fluency vs. Robustness: Models producing fluent stories often failed on robustness, user experience, or resistance-sensitive personalization. High mean performance did not guarantee robustness; for instance, Opus 4.6 showed strong peaks but also severe collapse cases on specific personas, whereas Sonnet 4.6 maintained consistent high performance.
• Persona Difficulty: The difficulty of the benchmark varied significantly across personas. "Sara" (an overwhelmed new mother) was the hardest persona on average, while "Hye-jin" (a blocked composer) was the easiest. This suggests models handle creative blockage more reliably than emotionally ambivalent exhaustion.
• Human vs. Judge Discrepancy: While human ratings and LLM judges agreed on the top tier (Sonnet 4.6 and Opus 4.6), they diverged in the middle tier. Human raters favored models like Qwen3.5 and Doubao more than LLM judges did, particularly in User Experience dimensions. This indicates that automated judges are effective for broad screening but may miss nuances in pacing, texture, and emotional resonance that humans perceive.
• Failure Modes: Manual audits identified resistance-sensitive personalization as a primary bottleneck. Even grammatically clean episodes often failed to track what the user was resisting, offering premature reassurance or converting resistance into generic progress too quickly.

Significance and Claims

The paper claims that interactive narrative offers a useful benchmark for evaluating long-horizon, user-adaptive LLM behavior beyond isolated story quality. The findings suggest that strong narrative fluency does not guarantee robustness, human preference alignment, or the ability to sustain context, character, and empathy over time.

The authors position NARRA-Gym as a step toward evaluating how well models can manage the interplay of planning, memory, and empathy in sustained interaction. They note that the benchmark highlights specific failure modes—such as the inability to maintain a "resistance-sensitive" stance—that are critical for applications in creative domains, games, and interactive media, but which are overlooked by existing static benchmarks. The work emphasizes that future progress requires moving beyond isolated story quality metrics to measure sustained interaction capabilities.