A Study of Temporal Fusion Strategies for Named Entity Recognition in Historical Texts

Technical Summary: A Study of Temporal Fusion Strategies for Named Entity Recognition in Historical Texts

Problem Statement
Named Entity Recognition (NER) in historical texts faces a unique challenge: temporal variation. Entities in historical documents drift in surface form, salience, and meaning across time. While large language models (LMs) have advanced general NLP tasks, their ability to reason about temporality, particularly in diachronic contexts, remains limited. Existing studies indicate that even advanced models exhibit directionality biases, poor temporal calibration, and difficulty retaining temporally anchored facts. Furthermore, while temporal representations (e.g., time vectors, temporal graphs) have been explored for other tasks, they remain largely disconnected from token-level tasks like NER. Historical NER is further complicated by diachronic drift, OCR degradation, and multilingual variation, yet few studies have systematically compared architectural strategies for integrating temporal metadata into NER models.

Methodology
The authors treat historical NER as a token classification task where the input consists of a token sequence $X$ and a document publication year $year$. They employ a Transformer-based architecture and systematically inject temporal information using explicit year embeddings ($y = \text{Emb}(year)$). The study compares two modes of temporal encoding:

1. Absolute Mode: The embedding index corresponds directly to the publication year (e.g., 1889).
2. Time-Distance Mode: The embedding represents the number of years between the document's publication and a fixed reference year (2025), assigning lower indices to more recent documents.

The core contribution lies in the comparison of temporal fusion strategies, categorized into early fusion (injecting time before or during encoding) and late fusion (applying time to the encoder output):

• Baseline: No temporal fusion.
• Early Fusion:
  • Cross-Attention Fusion: Injects temporal information via cross-attention between token embeddings and the year embedding before encoding.
• Late Fusion:
  • Adapter Fusion: A lightweight MLP processes the year embedding and adds the result to each token representation.
  • Concatenation Fusion: The year embedding is concatenated to each token vector and projected back to the original dimensionality.
  • Relative Temporal Fusion: A nonlinear encoder transforms the year embedding into a relative representation used in a feature-wise linear modulation (FiLM) mechanism.
  • Cross-Attention Fusion (Late): Fuses temporal information with the encoder output using cross-attention after encoding.

Experimental Setup
The models were evaluated on the HIPE-2020 dataset, specifically the French and German subsets, which contain publication year metadata. The datasets cover a wide temporal range (French: 1798–2018; German: 1798–1950) and include significant OCR noise (approx. 33% for French, 43% for German). The authors used a multilingual historical variant of BERT as the base model, fine-tuning with micro-averaged F1 scores as the primary metric.

Key Results

1. Performance Gains: Late fusion strategies consistently yielded more robust and temporally generalizable performance compared to early fusion and the baseline. This was particularly evident in early, noisy periods (1800–1850) and for longer, structurally complex entity mentions.
2. Temporal Modes: The impact of encoding mode (absolute vs. time-distance) was language-dependent. In German, time-distance encoding provided notable gains (up to +3 F1) for strategies like concatenation and adapters. In French, results were less consistent, with some strategies performing better with absolute encoding.
3. Entity Sensitivity: Late fusion strategies showed a stable or positive gain for longer entities (20+ characters), suggesting that injecting temporal signals at later stages helps preserve surface-level distinctions critical for complex entities.
4. Internalization of Time: Linear probing experiments, where the true document year was replaced with a random year during inference, confirmed that late fusion models genuinely internalize temporal signals in their latent space. Late cross-attention, adapter, and concatenation strategies achieved the highest probing accuracy, whereas early fusion and baseline models showed minimal temporal encoding.
5. Statistical Significance: While temporal fusion generally improved performance, the gains were often subtle. Paired t-tests revealed that most strategies did not yield statistically significant improvements at the $p < 0.05$ threshold, with the exception of the late-cross-attention strategy under the absolute temporal mode ($p = 0.041$).

Significance and Claims
The paper claims that structurally injecting time into Transformer architectures via modular fusion strategies yields modest yet consistent gains for historical NER across languages and decades. The authors emphasize that late fusion is the most robust approach, particularly for handling noisy, early-period data and recognizing temporally variable entity types like locations.

The study concludes that while the approach is not novel in terms of generative capabilities, it offers a practical, lightweight enhancement for diachronic corpora. This is significant because real-world historical corpora often face constraints (e.g., privacy, restrictive policies) that prevent the use of massive generative LLMs. In such contexts, structured methods leveraging metadata like publication dates remain a vital source of information for interpretable models. The authors recommend adopting late fusion, testing both temporal encoding modes based on the specific language and setup, and utilizing temporal fusion as a targeted enhancement for noisy historical data.

Limitations
The authors acknowledge several limitations: the study is restricted to year-level granularity (lacking finer temporal resolution); results are confined to French and German subsets of HIPE-2020; probing analysis focuses only on linear decodability; and evaluations were conducted under controlled conditions with a single backbone architecture, meaning real-world applications with missing metadata may yield different outcomes.