Technical Summary: AI as a Tool for Simulation-Based Experiments in Literary Studies

1. Problem Statement

Literary studies and history lack the capacity for controlled, laboratory-style experimentation. Scholars cannot rerun historical events, isolate specific causal variables (e.g., economic precarity, migration patterns), or test counterfactual conditions (e.g., "How would modernism have developed without James Joyce?") because the subjects are dead and the events have already occurred. While other fields like climate science and epidemiology utilize computational simulations to model phenomena that cannot be directly manipulated, literary studies has lacked a principled, grounded method for such inquiry.

The central problem addressed is the absence of a rigorous experimental toolkit to test claims about influence, context, and causation in literary history. The paper proposes that generative AI, specifically Large Language Models (LLMs), can serve as "silicon samples" or proxies for human populations to simulate cultural production, allowing for the systematic manipulation of variables (genre, period, authorial context) while holding others constant.

2. Methodology

The study employs a large-scale empirical design comparing AI-generated fiction against a corpus of human-authored novels. The methodology consists of three primary components:

2.1 Corpus Construction

• Human Reference Corpus: Derived from CONLIT, containing 2,754 volumes (2001–2021). The primary comparison set includes 258 prize-nominated or winning novels (Booker, National Book Award, etc.), labeled 'PW'. Additional subsets include Science Fiction (SF) and Mystery (MY).
• AI-Generated Corpus: A total of 3,800 chapters were generated using GPT-5 (via OpenAI batch API).
  • Complex Prompts (2,300 texts): Utilized a multi-component structure including a system prompt (expert novelist persona), a user prompt (write a prize-competitive first chapter), and synthetic author biographies. These biographies were generated to resemble real authors (e.g., an Ian McEwan-like persona) with specific birthplaces, careers, and stylistic traits, providing individuated cultural context. Some prompts included historical date directives (2005–2016).
  • Basic Prompts (1,500 texts): Minimal instructions ("Write the first chapter of a... novel") across five models (GPT-5, GPT-5-mini, GPT-5-nano, GPT-4.1, GPT-3.5-turbo) without biographies or specific constraints.

2.2 Analysis Framework

• Embedding Analysis: Texts were embedded using Qwen3-Embedding-8B (8B parameters, 32k token window) with a prompt-steered bias toward content, form, and style.
  • Human books were chunked (up to 32k tokens) and aggregated via length-weighted means.
  • AI chapters (avg. ~13k tokens) were treated as single chunks.
  • Similarity was measured via cosine similarity in the embedding space.
  • UMAP was used for 2D visualization, fitting the space to human texts and projecting AI texts into it.
• Lexical Analysis: The Fightin' Words method (Monroe et al., 2008) was applied to identify statistically overrepresented words in AI vs. human texts, using non-prize fiction as a Bayesian prior.

3. Key Results

3.1 Genre Structure and Distribution

• Preservation of Genre: AI-generated texts preserve the genre structure of human fiction. AI texts targeting prize-nominated (PW), SF, and Mystery genres are most similar to their human counterparts, not to other genres.
• Diversity vs. Homogeneity: Contrary to prior literature suggesting LLMs are highly homogeneous, this study found that complex-prompted AI texts are more diverse (less internally similar) than human texts within the same genre.
  • Human PW similarity: 0.635
  • Complex-Prompted AI PW similarity: 0.580
  • Basic-Prompted AI PW similarity: 0.682 (more homogeneous).
• Prompt Engineering Impact: Complex prompts with synthetic biographies significantly increased diversity and moved outputs closer to human distributions compared to basic prompts. Basic prompts produced tightly clustered, distinct outputs.

3.2 Temporal Conditioning

• Null Result: Prompting models to write as if in a specific year (2005–2016) produced no measurable difference in the embedding space compared to undated prompts.
• Implication: Simple prompt-level temporal conditioning is insufficient to alter the generative behavior of the model regarding historical period, consistent with findings that period-specific pretraining is likely required for historical fidelity.

3.3 Lexical Differences

• Human Texts: Characterized by third-person pronouns, past-tense markers, and colloquialisms, indicating embodied, past-tense narration.
• AI Texts: Showed higher usage of present-tense markers, negation (the most distinctive AI word), metafictional terms (e.g., "wrote," "sentence"), and first-person plural ("we").
• Magnitude: While detectable, the lexical differences were modest compared to distinct groups (e.g., political speeches), suggesting AI texts are "in-distribution" but distinguishable in register.

4. Key Contributions

1. Methodological Framework: Proposes a framework for using LLMs as instruments for simulation-based literary experiments, treating AI outputs as "simulated literary outputs" rather than creative products.
2. First Demonstration of In-Distribution Outputs: Provides the first evidence that AI models can generate synthetic samples of human-authored literature (including high-status prize genres) that fall within the distributional range of human texts, rather than being entirely out-of-distribution.
3. Role of Synthetic Biographies: Demonstrates that individuated authorial context (via synthetic biographies) is a critical factor in reducing the homogeneity of LLM outputs and improving simulation fidelity.
4. Limitations of Prompting: Establishes that simple prompt-level temporal conditioning fails to produce period-consistent outputs, highlighting the need for deeper interventions (model editing, unlearning, or pretraining) for counterfactual historical simulation.

5. Significance and Claims

The paper claims that AI-based simulation offers a "principled alternative" to direct experimentation for literary studies, allowing scholars to reason causally about historical development.

• Current State: The method is currently "partially but informatively" successful. It captures structural features (genre organization, effects of authorial individualization) but misses others (narrative register, temporal specificity).
• Feasibility: The authors argue that full-blown counterfactual simulations of contemporary literary-social processes are "on the edge of tenability," while historical simulations require further technical advances (e.g., model editing, period-specific pretraining).
• Future Outlook: The paper frames these findings not as evidence of fundamental impossibility, but as a starting point. It invites literary scholars to use these "imperfect but improvable" tools to address counterfactual questions and calls for NLP researchers to advance model editing, cultural conditioning, and diversity in generation.

The study concludes that while the simulation is not yet a perfect replica of the target system, it is "good enough" along specified dimensions to support useful inference, analogous to early-stage climate models that captured large-scale circulation before regional patterns.