Technical Summary: Pretraining Language Models on Historical Text

Problem Statement

Modern foundation models, while impressive in general natural language tasks, suffer from significant biases when applied to historical settings. They exhibit lookahead bias, where models trained on modern corpora inadvertently "know" the outcomes of historical events, mistaking memorized hindsight for genuine foresight. Furthermore, their linguistic distributions are skewed toward modern stylistic and lexical patterns, making them poor tools for studying historical contexts.

Existing approaches to mitigate this face three primary challenges:

1. Data Scarcity and Quality: Unlike modern web-scale corpora (e.g., Common Crawl), historical data is limited, often derived from OCR-transcribed sources containing noise and modern metadata.
2. Temporal Leakage: Even with pre-training cutoffs, leakage occurs through modern annotations, OCR artifacts, and, crucially, during the post-training (instruction tuning) phase where synthetic data generated by modern LLMs reintroduces anachronistic information.
3. Evaluation Misalignment: Standard benchmarks often fail to distinguish between a model's inability to reason and its unfamiliarity with modern language or contexts, while also failing to rigorously test for temporal consistency.

Methodology

The authors introduce TYPEWRITERLM, a 7.24B parameter decoder-only Transformer (based on the Llama 3 architecture) trained exclusively on English text predating 1913. The methodology is structured around a leakage-controlled pipeline:

1. Corpus Construction: TYPEWRITERCORPUS

• Scale and Scope: A 54-billion-token corpus covering 1700–1913.
• Sources: Primarily digitized institutional books (97.7% of tokens) from Harvard and the British Library, supplemented by curated datasets including parliamentary discourse (Hansard), scientific papers (Royal Society), legal proceedings (Old Bailey), and literary texts.
• Cleaning: A rigorous pipeline removes OCR artifacts (e.g., hyphenation errors, column breaks), modern metadata (ownership stamps, URLs, HTML entities), and editorial annotations to minimize temporal contamination.

2. Pre-training

• Tokenizer: A custom Byte-Pair Encoding (BPE) tokenizer (32k vocab) trained on the historical corpus to avoid modern vocabulary contamination.
• Architecture: 7.24B parameters, grouped-query attention, RMSNorm, rotary position embeddings, and an 8,192-token context window.
• Training: Trained for 2.59 epochs on approximately 140B tokens using AdamW with bfloat16 mixed precision.

3. Historical Instruction Tuning: Lexically Grounded Framework

To prevent the reintroduction of modern information during fine-tuning, the authors propose lexically grounded instruction tuning.

• Constraint: Every token in a model's response must be derivable from the source historical text or a small allowlist of function words. This preserves temporal consistency in lexical, stylistic, and semantic patterns.
• Datasets:
  • HISTORY-LIMA: 1,000 high-quality, single-turn examples (plus 1,030 multi-turn variants) where responses are verbatim or lexically constrained extracts from pre-1913 texts.
  • HISTORY-SELFINSTRUCT: A scaled dataset of ~288k pairs generated by inverting the standard pipeline: the base model generates instructions conditioned on fixed, lexically grounded historical responses.
• Verification: A strict post-generation verifier discards any response containing non-compliant tokens, utilizing morphological normalization and OCR correction only if the recovered form remains derivable from the source.

4. Evaluation: HISTORY-EVENT

The authors introduce HISTORY-EVENT, a benchmark of 2,344 historical events (1700–2025) to evaluate temporal grounding and leakage.

• Surprisingness: Measures the model's perplexity (Bits-per-Byte) when presented with event descriptions and dates. A successful historical model should show high "surprise" (high perplexity) for post-cutoff events.
• Recall and Leakage: Tests whether models can recall pre-cutoff events (correctness) or inadvertently know post-cutoff events (leakage).

Key Results

General Capability

• Competitive Performance: TYPEWRITERLM (7.24B) outperforms smaller historical models (e.g., Mr. Chatterbox, TimeCapsuleLLM) on standard benchmarks like ARC-Challenge and HellaSwag.
• Instruction Following: Despite strict lexical constraints, the instruction-tuned variants achieve competitive win rates against GPT-1900 on filtered AlpacaEval prompts and perform comparably to other historical models on IFEval.
• Reasoning: The model demonstrates nontrivial reasoning capabilities on ARC-Challenge, comparable to GPT2-XL, despite its strict historical cutoff.

Temporal Consistency and Leakage

• Surprisingness Patterns: Unlike modern baselines (Llama-3.1-8B) which show flat perplexity across time, TYPEWRITERLM exhibits a sharp increase in perplexity for events post-1913, indicating the knowledge cutoff is reflected in the learned distribution.
• Leakage Detection: While the model largely respects its cutoff, the evaluation reveals minor leakage (0.0%–0.6% of post-cutoff events recalled) in both TYPEWRITERLM and other large historical models, suggesting that complete elimination of lookahead bias remains difficult.
• Benchmark Sensitivity: Experiments rewriting modern benchmarks (HellaSwag) into 1800s prose styles show that performance drops significantly for modern models when language style changes, whereas historical models are more robust to style shifts but sensitive to topic mismatches.

Significance and Contributions

The paper claims three primary contributions to the field of NLP and digital humanities:

1. Feasibility of Data-Constrained Pre-training: The authors demonstrate that it is possible to train competitive, historically grounded language models (7B scale) using limited, cleaned historical data, challenging the notion that massive, diverse web corpora are strictly necessary for capable models.
2. End-to-End Leakage Mitigation: The work introduces a comprehensive pipeline that controls leakage at every stage: strict cleaning for pre-training, lexically grounded instruction tuning to prevent anachronistic post-training, and a dedicated evaluation suite (HISTORY-EVENT) to verify temporal integrity.
3. Open Resources: The authors release TYPEWRITERLM (both base and instruction-tuned), TYPEWRITERCORPUS, and the associated datasets (HISTORY-LIMA, HISTORY-SELFINSTRUCT, HISTORY-EVENT) to support transparent research in historical language modeling.

The paper positions these models as essential tools for studying temporal distribution shifts, the interplay between reasoning and memorization, and diachronic language change, while acknowledging that future work is needed to address data efficiency and the societal risks of models reflecting historical worldviews.