💬 NLP

CASPER in the Machine: Insights into Character Variety in LLM-Generated Stories

This paper investigates the similarities and differences between LLM-generated and human-written stories by applying narratological definitions to analyze eight intricate dimensions of character portrayal, aiming to determine if AI models produce characters with comparable variety and depth to human authors.

Original authors: Anneliese Brei, Abhisheik Sharma, Nicholas Sanaie, Lu Wang, Snigdha Chaturvedi

Published 2026-06-23✓ Author reviewed ⓘ

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anneliese Brei, Abhisheik Sharma, Nicholas Sanaie, Lu Wang, Snigdha Chaturvedi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a literary detective trying to solve a mystery: Can a robot writer create characters that feel as real and varied as the ones a human author writes?

This paper, titled "CASPER in the Machine," is the report from that investigation. The researchers built a special tool called CASPER (Character's Portrayal Classifier) to act as a magnifying glass, looking closely at the "soul" of characters in stories.

Here is the breakdown of their findings, using simple analogies:

1. The Tool: A Character "X-Ray"

Before this study, people mostly looked at what characters did (their job, their gender, or if they were "nice"). But the researchers wanted to know how the characters were built.

They used a framework based on old-school storytelling theory (like a recipe book for character building) that looks at 8 specific pairs of traits. Think of these as a set of scales where a character is weighed on one side or the other:

Stylized vs. Natural: Is the character like a cartoon superhero (exaggerated) or a neighbor you might meet at the grocery store (realistic)?
Coherent vs. Incoherent: Does the character act like a consistent person, or do they switch personalities randomly?
Whole vs. Fragmented: Do we know the character's full backstory and feelings, or are they just a silhouette with missing pieces?
Literal vs. Symbolic: Is the character just a person, or are they secretly representing a big idea (like "Hope" or "Greed")?
Complex vs. Simple: Do they have conflicting desires (like wanting to be brave but being scared), or are they one-note?
Transparent vs. Opaque: Do we know exactly what they are thinking, or are they a mystery box?
Dynamic vs. Static: Do they change and grow during the story, or are they the same person from start to finish?
Closed vs. Open: Does the story wrap up their life neatly, or does it leave their future hanging in the balance?

2. The Experiment: The "Taste Test"

The researchers gathered a massive buffet of stories:

200 Human-Written Stories: Collected from online writing communities (carefully checked to ensure they weren't written by AI).
4,400 AI-Generated Stories: Created by 7 different AI models (ranging from small to very large) using the exact same prompts as the human stories.

They fed all these stories into their CASPER tool to see how the characters stacked up.

3. The Findings: What the AI Got Right (and Wrong)

The "Safety First" Strategy

The Finding: Human writers love to leave things open-ended. They often create characters whose stories don't have a perfect ending, or who remain a bit mysterious.
The AI Difference: AI models are like overly polite butlers. They hate leaving a job half-finished. They almost always tie up every loose end. If a human character might wander off into the sunset with an uncertain future, the AI character will get a neat, wrapped-up conclusion. The AI "plays it safe" by ensuring the character's story is complete.

The "Growth Spurt" Bias

The Finding: AI characters are more likely to be dynamic. They almost always start sad and end happy, or start weak and end strong.
The AI Difference: Humans write characters who stay stubborn, stay confused, or stay exactly the same because that's how real life works. AI, however, seems to think every story needs a "moral lesson" where the character learns and grows. It's like the AI is trying to write a fable for every story.

The "Stereotype" Trap

The Finding: AI characters are more stylized (like archetypes) and less literal (just regular people).
The AI Difference: If you ask for a "hero," the AI gives you a classic, exaggerated hero. Humans are better at writing the messy, boring, or weirdly specific people who don't fit a mold. The AI tends to lean on familiar tropes rather than creating unique, quirky individuals.

The Size Doesn't Matter (Much)

The Finding: You might think a super-smart, massive AI (like a 70-billion-parameter model) would write more complex characters than a smaller one.
The AI Difference: Surprise! The size of the AI brain didn't really change the type of characters it wrote. A small AI and a giant AI both produced characters with very similar patterns. They both had the same "safety-first" and "growth-spurt" habits.

The Family Tree

The Finding: Different AI families (like Llama, Phi, Mistral) have different "personalities."
The AI Difference: The Phi family of models seemed to generate the most diverse mix of characters (like a box of assorted chocolates). The Llama family was the most repetitive (like a box of all the same flavor).

4. The Big Picture

The paper concludes that while AI is getting better at writing stories, it still thinks differently than humans.

Humans are comfortable with ambiguity, mystery, and characters who don't change. We are okay with loose ends.
AI is obsessed with clarity, neat conclusions, and characters who learn a lesson. It treats every story like a structured lesson plan rather than a slice of life.

In short: If you want a story that feels like a perfectly wrapped gift with a clear moral, the AI is your friend. If you want a story that feels like a messy, unpredictable, and open-ended human experience, you still need a human writer.

Technical Summary: CASPER in the Machine

Problem Statement

As Large Language Models (LLMs) are increasingly utilized for creative writing and fictional story generation, a critical question arises regarding the qualitative differences between machine-generated narratives and human-authored ones. While recent research has examined LLM-generated stories through the lens of plot development and narrative structure, there is a significant gap in understanding how these models portray characters. Existing automated analyses often rely on basic attributes like demographics or personality traits, failing to capture the deeper, artistic nuances of character portrayal that define a story's tone, themes, and cultural resonance. The authors posit that current methods lack the granularity to determine if LLMs generate characters with the same variety, depth, and artistic intent as human writers.

Methodology

To address this, the authors introduce CASPER (Character's Portrayal Classifier), a framework grounded in narratological theory designed to automatically analyze character portrayal in short stories.

1. Theoretical Framework

Instead of relying on the vague and subjective "flat" vs. "round" character dichotomy, CASPER utilizes a taxonomy proposed by Hochman (1985). This framework evaluates characters across eight category-pairs, where each primary category is contrasted with its opposite:

Stylization vs. Naturalism: Deliberate idealization/artifice vs. lifelike accuracy.
Coherence vs. Incoherence: Consistent actions/speech vs. contradictions/fragmentation.
Wholeness vs. Fragmentariness: Integrated being with complete personality vs. partial aspects with gaps.
Literalness vs. Symbolism: Individual within the story world vs. representation of abstract ideas/themes.
Complexity vs. Simplicity: Multiple conflicting traits vs. defined by one or few dominant traits.
Transparency vs. Opacity: Clear inner life (thoughts/motives) vs. hidden/ambiguous inner life.
Dynamism vs. Staticism: Noticeable development/transformation vs. fundamental unchangeability.
Closure vs. Openness: Narrative arc with clear resolution vs. unresolved fate/significance.

2. Dataset Construction

The authors constructed a high-quality, comparable corpus:

Human-Written Stories: 200 short stories collected from r/shortstories, filtered to ensure they were not LLM-generated (using AI detectors) and mapped to four genres: Domestic, Romance, Science-Fiction/Fantasy, and Suspense/Thriller.
LLM-Generated Stories: 4,400 stories generated by 8 different LLMs (across 4 families: GPT, Llama, Phi, Mistral) of varying sizes (3B to 70B parameters).
Comparability: To ensure fair comparison, the authors used Llama-70B to generate writing prompts from the human stories, which were then used to elicit stories from the target LLMs, ensuring thematic and genre alignment.

3. Classification Pipeline

Annotation: A test set of 100 stories (50 human, 50 LLM) was manually annotated by literary experts to establish ground truth. Inter-rater agreement (Cohen's $\kappa$ ) was moderate (0.56).
Automated Classification: The authors employed an "LLM-as-a-judge" approach using Llama-70B and GPT4o-mini. They systematically tested various prompting strategies (Zero-shot, In-Context Learning with different layouts, Likert scales) to determine the optimal setup for each category.
Best Practices: The study found that binary classification using separate inference calls for each category yielded the best results. Specific prompt templates (e.g., ICL-repeated for Stylization, Zero-shot for Dynamism) were selected based on performance against human annotations.

Key Results

The analysis of 4,400 LLM-generated stories against human-written counterparts revealed distinct patterns:

Closure and Safety: LLMs are significantly more likely to generate characters with Closed narrative arcs (definite conclusions) compared to humans, who frequently leave characters with Open arcs. This suggests LLMs "play it safe" by tying up loose ends, whereas human writers are more willing to use ambiguity.
Stylization and Dynamism: LLM-generated characters are statistically more likely to be Stylized (stereotypical/archetypal) and Dynamic (undergoing noticeable growth) than human characters. LLMs tend to create characters who fit archetypes and experience emotional shifts to convey moral lessons.
Transparency and Coherence: Both humans and LLMs overwhelmingly prefer Coherent, Whole, and Transparent characters. This indicates a lack of diversity in mystery or confusion, though the authors note this may be influenced by the focus on protagonists.
Model Size: Contrary to the expectation that larger models produce more nuanced characters, model size does not significantly affect character portrayal. Smaller and larger models produced nearly identical distributions across categories, with only minor differences in stylization.
Model Family Diversity: The Phi family generated the most diverse character distributions, while the Llama family generated the least diverse. Mistral models showed a preference for more stylized and less coherent characters.
Genre Influence: Genre significantly impacts character types. For instance, Domestic stories feature realistic but often symbolic characters, while Sci-Fi/Fantasy stories exhibit the highest rates of stylization and symbolism.
Inference Variability: When the same prompt is used multiple times, Literalness and Closure show the highest variability across generations, suggesting LLMs struggle to consistently adhere to context clues regarding how literal or resolved a character should be.

Contributions and Significance

The paper makes three primary contributions:

Framework Adaptation: It successfully adapts theory-grounded narratological methods (Hochman's taxonomy) into an automated framework (CASPER) for analyzing LLM-generated stories, providing a tool to track character portrayal evolution in future models.
Curated Corpus: It releases a carefully curated dataset of 4,400 LLM-generated and 200 human-written short stories, ensuring comparability across genres and themes for future research.
Empirical Analysis: It provides a detailed, fine-grained comparison of LLM and human characters, moving beyond basic traits to analyze how characters are portrayed within the artistic framework of a story.

Significance:
The authors claim that their work challenges the assumption that larger models inherently produce more human-like or diverse characters. Instead, they demonstrate that current LLMs exhibit specific biases toward "safe" storytelling (closure), archetypal characters (stylization), and moral growth (dynamism). By identifying these patterns, the paper offers a deeper understanding of the limitations and stylistic tendencies of current generative AI in creative domains, suggesting that while LLMs can mimic narrative structures, they currently lack the artistic liberty to embrace ambiguity and the full spectrum of human character complexity.