Technical Summary: TimeCapsule

Problem Statement

Current Large Language Models (LLMs) are "temporally overexposed," trained on vast contemporary corpora that encode present-day concepts, scientific frameworks, and ethical assumptions. When tasked with simulating historical periods, these models suffer from a structural "presentism." Even when prompted to adopt a historical persona, they inevitably import anachronistic knowledge (e.g., defining an airplane as a vehicle that soars above clouds), resulting in what the authors term "costume drama"—stylistic mimicry that fails to capture the deeper semantic architectures and ontological constraints of the past. Existing fine-tuning approaches can approximate stylistic surfaces but cannot structurally prevent the model from accessing knowledge unavailable to the historical period in question.

Methodology

The authors introduce TimeCapsule, a generative archive designed to enforce "epistemological isolation" through Selective Temporal Training (STT). The core design principle is the creation of an "epistemological event horizon," a chronological boundary beyond which the model possesses no world knowledge.

1. Corpus Construction and Curation

• Temporal Scope: The training data is strictly bounded to the period 1800–1875, terminating just before major technological thresholds such as the telephone (1876) and practical electric lighting (1878).
• Data Source: The corpus consists of 89.63 GB of OCR-extracted text from the Internet Archive, comprising 136,302 documents.
• Composition: The dataset is dominated by Parliamentary/Legal records (37.14%) and Periodicals (16.29%), with significant representation of medical texts, poetry, literary fiction, and religious works. This diversity ensures the model internalizes the full heterogeneity of nineteenth-century English registers.
• Bias Preservation: Following the principle of "archival honesty," the authors explicitly do not correct for historical biases. The corpus retains the original gender asymmetry (a 4.99:1 male-to-female pronoun ratio) and imperial geographic distributions (e.g., high frequency of mentions of India, Egypt, and Ireland) to preserve the "colonial gaze" and structural inequalities of the era as empirical data.

2. Preprocessing and Tokenization

• Minimal Normalization: The cleaning pipeline preserves archaic spellings (e.g., "connexion," "colour") and Victorian punctuation to maintain forensic materiality. Only features impeding tokenization (e.g., OCR artifacts, long-s characters) are normalized.
• Custom Tokenizer: A 32K Byte-Pair Encoding (BPE) tokenizer was trained from scratch on the historical corpus to prevent the fragmentation of archaic terms (e.g., avoiding the splitting of "connexion" into modern subwords), thereby defining the model's representational primitives.

3. Model Architecture and Training

• Architecture: TimeCapsule is a 1.2B-parameter causal model based on the LLaMA architecture, utilizing Grouped-Query Attention (GQA) and a 2048-token context window to accommodate the dense syntactic rhythms of Victorian prose.
• Training Strategy: The model was trained on H100 hardware. Crucially, training was halted early (at 50% of a full epoch, approx. 13B tokens) once validation perplexity stabilized. This early stopping was a deliberate methodological choice to limit memorization and encourage generalization within the constrained domain.
• Environmental Cost: The training consumed an estimated 92 kWh, emitting approximately 37 kg CO2e, significantly lower than the cost of pre-training massive general-purpose foundation models.

Key Results

1. Quantitative Performance

• Perplexity Reduction: On a held-out test set of Victorian prose, TimeCapsule achieved a perplexity of 37.59, representing a 45.4% relative reduction compared to a GPT-2 baseline (68.83).
• Contextual Fit: While larger contemporary models (e.g., Mistral-7B) achieved lower raw perplexity (16.50), the authors argue this does not indicate historical authenticity, as those models encode post-1875 knowledge. TimeCapsule's lower perplexity specifically reflects a superior fit to the period's linguistic patterns.
• Tokenizer Efficiency: The custom historical tokenizer reduced subword fragmentation by 8.3% to 10.3% compared to modern tokenizers.

2. Diachronic Semantic Analysis

Vector projection analysis revealed measurable shifts in the "latent materiality" of the era:

• Industrialization of Time: The semantic association between "TIME" and "FACTORY" was 2.1 times stronger in TimeCapsule's latent space compared to a modern BERT baseline.
• Conceptual Drift: The concept of "VALUE" shifted significantly away from "VIRTUE" toward "COMMERCE," and "POWER" aligned closer to "STEAM" than abstract "AUTHORITY." These shifts quantitatively map the commodification of time and the rise of industrial capitalism within the model's weights.

3. Ontological Repair and Hallucination

When confronted with anachronistic tokens (e.g., "airplane," "computer," "internet"), TimeCapsule could not retrieve modern definitions. Instead, it performed ontological repair, generating historically plausible analogical explanations based on nineteenth-century conceptual primitives:

• Computer: Interpreted as a "hypertrophied lung" or "right lung enlarged," derived from the period's association of computation with actuarial tables, vital statistics, and medical pathology.
• Airplane: Described as a "useless air-pump" or a failed pneumatic experiment, reflecting the era's limited understanding of flight.
• Internet: Associated with "the sea" and maritime danger, linking the concept of a network to the era's primary medium of global connection.

4. Qualitative Hermeneutic Probe

A blind evaluation with two humanities experts (N=2) revealed a crisis of authenticity:

• Misclassification: Both experts misclassified approximately 40% of genuine Victorian excerpts as machine-generated.
• Heuristics: Experts relied on flawed heuristics, such as the "Boredom Heuristic" (identifying text as authentic because it was "boring") or the "Hyper-Real Trap" (rejecting text because it was "too clear").
• Implication: The model successfully reproduced the "generic mean" and mundane texture of the archive to a degree that destabilized expert judgments of provenance.

Significance and Claims

The paper argues that structural ignorance of the future is not a deficit but a generative design condition for historical sensemaking.

1. Generative Archive as Design Paradigm: TimeCapsule operationalizes a shift from information retrieval to "generative archiving." By constraining the model to the epistemic conditions of the past, it forces the system to engage in "computational sensemaking," reconstructing the impossible through the conceptual resources available to the historical moment.
2. Hallucination as Interpretive Probe: The paper reframes "hallucination" not as an error to be minimized, but as a window into historical logic. When the model "hallucinates" a definition for a modern concept, it reveals the underlying ontological structures and semantic associations of the nineteenth century.
3. Archival Honesty: The authors claim that preserving historical biases (e.g., colonialism, gender inequality) is an ethical necessity for decolonial critique. "Fixing" these biases through modern alignment (RLHF) would constitute historical revisionism and erase the structural realities scholars seek to study. TimeCapsule renders these ideologies empirically legible rather than sanitizing them.
4. Epistemic Event Horizon: The model demonstrates that a chronological cutoff creates a structural condition of generation where the model does not merely "pretend" not to know the future; it is structurally incapable of knowing it. This allows for a more authentic simulation of historical thought processes than current "costume drama" approaches.

In summary, TimeCapsule presents a method for creating "situated interpretive instruments" that mediate speculative dialogue with the past, using temporal constraints to reveal the semantic and ontological contours of nineteenth-century thought.