If an LLM Were a Character, Would It Know Its Own Story? Evaluating Lifelong Learning in LLMs

This paper introduces LIFESTATE-BENCH, a novel benchmark utilizing narrative datasets like Hamlet to evaluate lifelong learning in large language models, revealing that while non-parametric methods outperform parametric ones in managing stateful interactions, all models still struggle with catastrophic forgetting over extended engagements.

The Gist

Imagine you are talking to a very talented actor who has memorized the entire script of a play. They can recite lines perfectly, but here's the catch: they have no memory of what happened in the previous scene. Every time you start a new conversation, they are a blank slate, pretending to be a character for the first time, even if you've been talking for hours.

This is how most Large Language Models (LLMs) work today. They are "stateless," meaning they don't naturally remember their own story as it unfolds.

This paper introduces a new way to test if we can teach these AI actors to actually remember their life story and evolve as characters, just like humans do. Here is the breakdown in simple terms:

1. The Problem: The "Amnesiac Actor"

Humans learn by accumulating experiences. If you meet someone today, remember them tomorrow, and meet them again next week, your relationship changes. You know their secrets, their moods, and your shared history.

Current AI models are like actors who forget the plot every time the curtain rises. If you ask them, "Do you remember that we fought yesterday?" they might say, "I don't know, who are you?" because they don't have a built-in "memory bank" that updates as the conversation goes on.

2. The Solution: "LIFESTATE-BENCH"

The authors created a new test called LIFESTATE-BENCH. Think of this as a long-running TV drama designed specifically to test the AI's memory.

Instead of short, random chats, they gave the AI a script (based on Shakespeare's Hamlet and some made-up stories) with a clear timeline. The AI had to play a character through multiple "episodes."

The test asks three specific types of questions to see if the AI is actually "living" the story:

• Self-Awareness: "Who are you right now?" (Does the AI remember its role?)
• Fact Memory: "What happened in the last scene?" (Did it remember the plot details?)
• Relationship Shift: "How do you feel about this other character now?" (Did the relationship change because of what happened yesterday?)

3. The Experiment: Two Ways to Remember

The researchers tested two different ways to help the AI remember:

• Method A: The "Photo Album" (Non-Parametric)
  Imagine giving the AI a giant scrapbook of everything that happened so far. Every time a new question comes, you hand the AI the whole book (or a summarized version of it) to read before answering.

  • Result: This worked much better. The more context the AI could read, the better it remembered the story.

• Method B: The "Brain Surgery" (Parametric)
  Imagine trying to permanently rewrite the AI's brain (its internal code) to "learn" the new facts, so it doesn't need the scrapbook. This is like trying to teach a dog a new trick by physically changing its brain structure.

  • Result: This was less effective. The AI tended to "forget" old things as it learned new things (a problem called "catastrophic forgetting"). It was like the AI was so busy learning the new scene that it erased the memory of the previous one.

4. The Big Discovery

The study found that current AI is still terrible at long-term storytelling.

• They forget easily: As the story got longer, the AI's performance dropped. It started mixing up who was the villain and who was the hero.
• Reading helps more than learning: The "Photo Album" method (giving the AI the history to read) was far superior to trying to "train" the AI to remember.
• Relationships are hard: The AI was okay at remembering facts ("The king died"), but terrible at understanding how relationships changed ("Now I hate my uncle because he killed my father").

The Takeaway

This paper is a reality check. While AI can chat like a human, it doesn't yet have a "soul" or a continuous life story. It's like a brilliant improvisational actor who forgets the plot the moment the scene ends.

To make AI truly useful for long-term companionship or complex storytelling, we need to stop trying to force them to "memorize" everything in their brain and instead give them better tools to review their history as the story progresses. The authors' new benchmark is a tool to help developers figure out how to fix this memory gap.

Technical Summary

1. Problem Statement

Large Language Models (LLMs) exhibit human-like dialogue capabilities but are fundamentally stateless due to their training on massive corpora as a "superposition of simulacra." Unlike humans, who accumulate experiences and memories to evolve their personality and state, LLMs typically lack persistent internal states across interactions.

While multi-turn, multi-agent interactions can induce emergent "character-like" behaviors, existing benchmarks fail to capture this dynamic evolution. Current evaluations are largely:

• Static: Focusing on open-ended or fixed responses rather than evolving narratives.
• Superficial: Prioritizing role-playing consistency or social intelligence without rigorous fact-checking against a timeline.
• Short-term: Lacking the ability to test long-term memory retention and the tracking of evolving relationships over extended episodes.

The core research question is: How can we quantify an LLM's "state evolution" (lifelong learning) as it transitions from a stateless superposition to a consistent, stateful character through cumulative interactions?

2. Methodology: LIFESTATE-BENCH

The authors introduce LIFESTATE-BENCH, a novel benchmark designed to evaluate lifelong learning through three synergistic components:

A. Cumulative Experience Modeling (Datasets)

The benchmark utilizes two episodic datasets to simulate a timeline of experiences:

1. Hamlet Dataset: Based on Shakespeare's Hamlet, featuring complex character relationships and plot progression. Character names were replaced to minimize pre-training data leakage.
2. Synthetic Dataset: Generated by Claude 3.5 Sonnet to ensure zero data leakage, featuring controlled plotlines, dynamic relationships, and emotional depth.

• Structure: Each episode ($E_t$) includes location, time, narration, and multi-character dialogues (avg. 28–66 turns per episode).
• Goal: To force models to accumulate structured, temporal experiences rather than isolated chit-chat.

B. Fact-Checking Mechanisms

To ensure objective evaluation, the benchmark employs a fact-based question-answering framework after each episode. Questions are categorized into three dimensions of the "State Space":

1. Self-Awareness: Does the model maintain its identity, role, and goals? (e.g., "Who are you?")
2. Factual Episode Memory Retrieval: Can the model recall specific past events without catastrophic forgetting? (e.g., "Did X agree to Y?")
3. Relationship Shift: Can the model track evolving relationships between agents? (e.g., "What is your relationship with X now vs. before?")

C. Memory Testing Approaches

The study evaluates two distinct strategies for managing memory:

• Non-Parametric Methods (Context-based):
  • Direct Concatenation: Appending all previous episode text to the current input.
  • Summary Concatenation: Using an LLM to summarize past episodes and concatenating the summary.
• Parametric Methods (Weight-based):
  • Knowledge Editing (GRACE): Directly updating model weights to integrate new episodic knowledge.
  • LoRA Fine-Tuning: Low-Rank Adaptation to fine-tune the model on historical context.

3. Experimental Setup

• Models Evaluated: Llama3.1-8B (Open-source), GPT-4-turbo (Closed-source), and DeepSeek R1 (Large reasoning model).
• Evaluation Metric: "LLM-as-a-Judge" using DeepSeek evaluator to score model outputs against ground-truth factual answers (1–100 scale).
• Metrics: Accuracy (ACC), Standard Deviation (Stability), and performance breakdown by state dimension.

4. Key Results

The experiments yielded several critical findings regarding the state of lifelong learning in LLMs:

• Non-Parametric > Parametric: Non-parametric methods (Direct/Summary Concatenation) significantly outperformed parametric methods (Knowledge Editing/LoRA).
  • Reasoning: Parametric methods struggle to integrate new episodic data without overwriting existing knowledge, leading to catastrophic forgetting.
  • Performance: Direct Concatenation achieved the highest accuracy (e.g., DeepSeek R1: 67.3% on Hamlet, 74.2% on Synthetic).
• Catastrophic Forgetting: All models exhibited performance degradation as the number of episodes increased.
  • Parametric models showed the steepest decline, particularly in Relationship Shift tasks.
  • Even non-parametric models struggled with long-term dependency tracking in complex narratives.
• Model Capabilities:
  • DeepSeek R1 demonstrated the most balanced performance, excelling in complex reasoning and relationship tracking.
  • GPT-4-turbo showed strong factual memory but higher variance in self-awareness.
  • Llama3.1-8B struggled significantly across all dimensions compared to larger models.
• Task Difficulty: Relationship Shift questions were the most challenging for all models, indicating a specific weakness in tracking dynamic social states over time compared to static fact retrieval.

5. Key Contributions

1. LIFESTATE-BENCH: A new benchmark featuring multi-agent episodic timelines (Hamlet + Synthetic) that moves beyond static role-playing to evaluate state evolution.
2. Fact-Checking Framework: A rigorous evaluation mechanism using self-awareness, memory retrieval, and relationship tracking questions with ground-truth answers, addressing the lack of objective metrics in current role-play benchmarks.
3. Empirical Insights: The discovery that non-parametric context management is currently superior to parametric weight updates for lifelong learning, and that catastrophic forgetting remains a critical bottleneck for maintaining consistent character states in long-term interactions.

6. Significance and Implications

This work highlights a fundamental gap in current LLM architectures: while they can simulate human-like dialogue, they lack the stateful memory architecture required for true lifelong learning.

• For Benchmarking: It shifts the focus from "how well does the model role-play?" to "how well does the model remember and evolve its state?"
• For Development: The results suggest that simply fine-tuning (parametric) is insufficient for stateful agents. Future research should focus on improving context window management, retrieval-augmented generation (RAG), and hybrid memory systems to prevent catastrophic forgetting in multi-turn, multi-agent scenarios.
• For AI Safety/Alignment: Understanding how LLMs track relationships and self-identity over time is crucial for deploying reliable autonomous agents in complex, long-term environments.