Era-Aware Language Modeling: Training a Decoder-Only Transformer on a Balanced Gutenberg Corpus

Technical Summary: Era-Aware Language Modeling on a Balanced Gutenberg Corpus

Problem Statement
Current large language models (LLMs) are predominantly trained on web-scraped corpora (e.g., Common Crawl) that are heavily skewed toward contemporary English. This temporal bias limits the ability to study how the historical composition of training data influences a model's generative style. Furthermore, in naturally occurring web corpora, topical content and stylistic register are entangled, making it difficult to isolate the specific effect of "era" (the historical period of writing) on model behavior independent of subject matter.

Methodology
The authors present GutenbergLM, a small decoder-only transformer trained from scratch on a purpose-built, era-balanced corpus derived entirely from Project Gutenberg. The methodology is structured into four primary stages:

1. Corpus Construction and Balancing:

   • Source: The authors parsed the full Project Gutenberg RDF catalog (78,672 records), filtering for 62,497 English-language books.
   • Era Classification: Books were stratified into three eras based on author birth year (or issue date as a fallback): Early (≤1800), Middle (1801–1900), and Modern (>1900).
   • Balanced Sampling: To prevent the natural dominance of the "Middle" era (69.3% of the raw catalog), the authors performed deterministic, seed-based sampling to select exactly 5,334 books per era, resulting in a perfectly balanced 16,002-book corpus.
   • Deduplication and Cleaning: Near-duplicates were removed using MinHash with Locality-Sensitive Hashing (LSH) at a Jaccard threshold of 0.85, leaving 15,509 unique books. Text cleaning involved stripping Project Gutenberg boilerplate, repairing encoding artifacts, and normalizing whitespace.
   • Splitting: A book-level stratified split was applied (90% train, 5% validation, 5% test) to ensure no stylistic leakage between sets.

2. Tokenizer:

   • A 32,000-vocabulary SentencePiece byte-pair-encoding (BPE) tokenizer was trained on the training corpus.
   • It features byte-level fallback to guarantee zero out-of-vocabulary tokens and digit-splitting tuned for historical date formats.
   • Three explicit control tokens (<ERA_EARLY>, <ERA_MIDDLE>, <ERA_MODERN>) were reserved to enable era conditioning.

3. Model Architecture:

   • Scale: A 109.5M-parameter decoder-only transformer following LLaMA-style design choices.
   • Components: The architecture includes 12 layers, 768 model dimension, 12 attention heads, RMSNorm, SwiGLU feed-forward networks, and Rotary Position Embeddings (RoPE).
   • Optimizations: Weight tying between embeddings and the language modeling head, gradient checkpointing, and bias-free linear layers were implemented.

4. Training Procedure:

   • Hardware: A single NVIDIA A10G GPU (11 GB VRAM) on an AWS EC2 instance.
   • Configuration: The model was trained for one epoch over 1.46 billion tokens (11,000 optimizer steps).
   • Techniques: The pipeline utilized mixed-precision (BF16) optimization, AdamW with cosine learning-rate decay, and GPU-adaptive batch presets (effective batch size of 128).

Key Results

• Training Efficiency: The single-epoch run completed in 13.12 hours, sustaining a throughput of 31,405 tokens/second with peak GPU memory usage of 14.4 GB.
• Perplexity: The model achieved a final training loss of 2.93, a validation perplexity of 25.6, and a held-out test perplexity of 24.03. The gap between training and validation loss (0.316 nats) suggests the model is not yet in a strongly overfit regime.
• Generation Metrics: On the test set, the model achieved a BLEU-4 of 0.093, ROUGE-L F1 of 0.189, and BERTScore F1 of 0.837. The authors note that low n-gram overlap scores are typical for open-ended creative tasks, while the high BERTScore indicates semantic plausibility.
• Qualitative Generation: Era-conditioned generations demonstrated that the control tokens successfully steered the model toward distinct period-appropriate vocabulary, syntax, and subject matter without explicit topical prompting. For instance, <ERA_EARLY> generated archaic legal/biblical registers, while <ERA_MODERN> produced contemporary social scenes.

Significance and Limitations
The paper positions GutenbergLM as a reproducible reference point for studying the relationship between training-corpus temporal composition and stylistic behavior, free from the topical confounds of web-scale data.

The authors maintain a modest tone regarding their claims:

• Granularity: The era classification relies on a coarse heuristic (author birth year), which may not account for late-career stylistic shifts or translated works.
• Evaluation Gaps: The quantitative era-specific perplexity breakdown was only logged for the Modern subset; the Early and Middle subsets lack specific quantitative metrics in this report.
• Training Scope: The model was trained for only one epoch. The learning rate schedule was configured for a longer horizon, implying potential for further perplexity reduction with extended training.
• Scale: The study is limited to a single-GPU environment and a sub-100M parameter scale, though the authors note that the unused memory headroom suggests potential for larger batch sizes or context lengths.

The work does not propose new applications or future architectural changes but rather establishes a controlled pipeline for isolating the variable of "era" in language modeling.