Technical Summary: ARCANE (Arc-Aware Narrative Evaluation)

1. Problem Statement

Role-playing language agents (RPLAs) are increasingly expected to provide immersive, authentic interactions with fictional characters. However, current evaluation benchmarks largely treat characters as static entities, focusing on factual recall, trait consistency, or surface-level linguistic style at a single point in time. This approach fails to capture the dynamic nature of character development in long-form narratives, where a character's values, motivations, and behavioral patterns evolve significantly as events accumulate.

Existing benchmarks, such as TimeCHARA, primarily detect factual hallucinations (e.g., whether a character knows events that haven't happened yet) but do not assess whether the agent's behavioral and psychological responses align with the character's specific developmental stage. Consequently, RPLAs may produce plausible responses that are nonetheless "out of character" regarding the narrative timeline, failing to reflect the psychological trajectory (the "Character Arc") that defines the story.

2. Methodology

2.1 The ARCANE Benchmark

The authors introduce ARCANE (Arc-Aware Narrative Evaluation), an automatically constructed benchmark designed to evaluate whether an RPLA's responses shift faithfully in step with a character's evolving psychology across narrative phases. The dataset spans 17 novels and 80 principal characters, comprising 544 Character Arcs and 4,601 probes.

The construction pipeline involves two primary stages:

• Character Arc Construction:

  • Dual-Stream Extraction: The narrative is processed through two independent streams: an event stream (extracting psychologically impactful events) and a state stream (emitting cross-sectional psychological profiles).
  • Axis Induction: These streams induce candidate axes (psychological dimensions) tracking internal changes (e.g., beliefs, motives) and relational changes (e.g., trust, antagonism).
  • Reconciliation & Validation: An LLM reconciles candidates from both streams. For the evaluation set, three human annotators independently validate the axes based on literary validity, novel span, and importance. Only axes approved by a 2-of-3 majority enter the final benchmark.
  • Structure: Each arc is segmented into phases, defining a trajectory from an initial state to a final state along a psychological axis (e.g., Harry Potter's moral axis shifting from "Punitive Justice" to "Empathic Forgiveness").

• Probe Generation:

  • Scenario Design: For each arc, a single (scenario, question) pair is generated and tested across all phases of the arc.
  • Categories: Probes are categorized by their distance from the source text:
    • In-Scenario: Lifted verbatim from the source text.
    • In-World: Invented situations within the source setting but not explicitly written.
    • Out-of-World: Scenarios transposed to non-source eras or settings, requiring generalization of the character's arc.
  • Reference Responses: Each probe includes phase-specific reference responses (action + thought) that define the "gold" behavior for that specific narrative phase.

2.2 Evaluation Metrics

The evaluation employs two levels of metrics:

1. Per-Phase Metrics: Assess individual responses against the reference for that phase.
   • APF (Action Phase-Fidelity): Grades overt actions (strategy, valence, target).
   • RPF (Reasoning Phase-Fidelity): Parses reasoning into slots (trigger, appraisal, goal, strategy).
   • RAE (Reasoning–Action Entailment): Checks if the action is licensed by the reasoning.
2. Trajectory Metric:
   • PTF (Phase Trajectory Fidelity): Evaluates the sequence of responses across all phases to ensure the model tracks the direction and shape of the arc, penalizing models that collapse adjacent phases or move in the wrong direction.

2.3 Context Strategies

The study compares six context modes to determine which best grounds the model in the character's state:

• Vanilla: Identity and query chapter only.
• Summary: Recent chapter summaries.
• RAG: Retrieved source-text chunks.
• LifeChoice / TimeCHARA: Existing benchmark context formats.
• ARC (Proposed): The curated Character Arc truncated at the query chapter, exposing the model to the trajectory information used to construct the probes.

2.4 Model Training

The authors fine-tuned open-weight models (Qwen3-8B and Qwen3-32B) using a two-stage pipeline:

1. Supervised Fine-Tuning (SFT): Learning the response format and behavioral conventions of ARCANE.
2. Direct Preference Optimization (DPO): Learning to distinguish ground-truth character behavior from plausible but temporally displaced alternatives (e.g., actions fitting the character but belonging to a different narrative phase).
   This resulted in ARCANE-8B and ARCANE-32B.

3. Key Results

• Superiority of Arc-Grounded Context: Across six models (including DeepSeek-V4 and Qwen3 variants) and six context modes, conditioning on the Character Arc consistently achieved the highest Overall scores. The gap between Arc and the best non-Arc baseline ranged from 2.2 to 8.4 points.
• Generalization to Unseen Scenarios: The advantage of the Arc context was most pronounced in Out-of-World scenarios (where retrieval has no relevant source text to find). In these cases, retrieval-based methods (RAG, LifeChoice) failed to provide relevant grounding, while the Arc provided the necessary psychological state.
• Trajectory Tracking: The PTF metric showed the largest gains for Arc-grounded models. While non-Arc models could sometimes score well on individual phases in isolation, they often failed to maintain the correct directional shift across the narrative.
• Low-Popularity Novel Performance: The Arc advantage extended to low-popularity novels (e.g., The Underdogs, East Lynne) not included in the validation set, with performance lifts of +4.1 to +15.3 points over baselines.
• Training Effects: Fine-tuning with DPO further enhanced performance, particularly in In-World and Out-of-World categories, by teaching the model to distinguish between adjacent developmental stages. However, DPO sometimes reduced performance on In-Scenario tasks, suggesting a trade-off between strict canon adherence and arc tracking.

4. Significance and Claims

The paper claims that ARCANE addresses a critical gap in RPLA evaluation by shifting the focus from static persona consistency to dynamic behavioral evolution. The authors argue that:

1. Static Personas are Insufficient: Authentic role-playing requires agents to adapt their values and behaviors as the story progresses, not just recall facts.
2. Retrieval is Limited: Standard retrieval methods (RAG) are ineffective for evaluating or guiding character behavior in scenarios the source text never explores; explicit psychological trajectory (the Arc) is required.
3. Training on Arcs Works: Fine-tuning on contrastive arc data (SFT + DPO) enables models to better track developmental trajectories, reducing "character-behavioral hallucinations."

The authors conclude that while their dataset is currently restricted to English novels and single-character evolution, ARCANE provides a robust framework for evaluating the "living" nature of character simulation. They identify multi-turn dialogue, where the arc advances through interaction, as the natural next step for this research.

Limitations Acknowledged:

• The dataset is English-only and restricted to the novel domain.
• Evaluation focuses on single-character behavior over accumulating events, not user-character or character-character interactions.
• The source material includes 19th/early-20th-century novels with period-bound social attitudes, requiring careful handling to avoid reproducing harmful stereotypes.
• The artifacts are intended for research only.