💬 NLP

Dynamic Topic Modeling for Cross-Corpus Temporal Analysis

This paper proposes a Dynamic Embedded Topic Model framework that establishes a shared topic space across multiple corpora with corpus-specific residual adaptations, enabling stable and interpretable cross-corpus temporal comparisons while preserving unique lexical variations.

Original authors: Ruoxuan Li, Bruce Kogut

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ruoxuan Li, Bruce Kogut

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

For decades, social scientists and historians have relied on computers to make sense of vast libraries of text, from old newspapers to business magazines. The goal is to find the hidden themes that run through these documents, much like a librarian sorting books by subject rather than by author. A common tool for this is called a topic model, which scans thousands of pages to group words that frequently appear together, revealing what people were talking about at a given time. When researchers want to see how these conversations change over years or even centuries, they use a dynamic version of this tool that tracks how the meaning of these word groups shifts as time passes. However, a major hurdle has always existed when trying to compare different collections of text. If you analyze a collection of business magazines and a separate collection of labor union reports, the computer usually treats them as two completely different worlds. It might find a topic about "money" in the business magazines and a topic about "wages" in the union reports, but it has no built-in way to know that these are actually two sides of the same coin. Traditionally, researchers had to run the analysis separately and then try to manually match the results afterward, a process that often failed to line up the themes correctly, leaving the comparison shaky and unreliable.

A team of researchers at Columbia University has developed a new way to solve this problem, allowing computers to compare different types of text collections while keeping the themes perfectly aligned across time. Instead of analyzing each collection separately and trying to force a match later, their method builds a single, shared map of ideas first. Imagine this shared map as a common coordinate system, like a standard set of latitude and longitude lines, that covers all the different text collections at once. The computer learns the main themes and how they evolve over a ninety-seven-year period using this combined map. Once this shared foundation is established, the researchers allow each specific collection—whether it is business news, labor reports, or general historical writing—to make small, controlled adjustments to the map. This lets the business magazines use their own specific vocabulary for a theme while the labor reports use theirs, all while staying anchored to the exact same underlying idea. The result is a system where the computer knows that the "money" topic in one collection and the "wages" topic in another are not just similar, but are actually the same topic viewed through different lenses.

The researchers tested this approach on three massive collections of text spanning from 1922 to 2019. One collection contained historical American English from magazines and newspapers, another held articles from the Harvard Business Review, and the third consisted of reports from the International Labour Review. These sources cover very different worlds: one is a broad mix of general history, one focuses on corporate management, and the third deals with workers' rights and social policy. The team compared their new method against older techniques where the collections were analyzed separately and then matched up later. The difference was stark. When using the old method of separate analysis, the computer could only correctly identify matching themes between the collections about 18 percent of the time. In contrast, the new shared-map method achieved a success rate of nearly 98 percent. This means that almost every time the computer looked at a theme in the business magazines, it could instantly and accurately find the corresponding theme in the labor reports, even though the words used were completely different.

The study also showed that this new method did not sacrifice the quality of the analysis for the sake of alignment. The computer was still able to learn the specific nuances of each collection, capturing the unique vocabulary of business executives and the distinct language of labor organizers. By keeping the main map fixed and only allowing small adjustments, the system preserved the ability to track how a single theme, such as economic transformation, evolved over the decades. The researchers were able to trace a single thread of conversation about the economy as it shifted from the industrial crises of the 1930s, through the post-war expansion of corporate management, and into the digital age of the late twentieth century. In each era, the shared map showed how the business magazines focused on profit and stock markets, while the labor reports focused on wages and worker protections, yet both were clearly discussing the same historical moment.

This work suggests that the key to comparing different groups of texts over long periods is to treat the alignment of themes as a fundamental part of the learning process, rather than an afterthought. The researchers found that when they tried to fine-tune the entire system for each collection separately, the connection between the themes broke down, and the computer lost its ability to see the big picture. Their findings indicate that by building a stable, shared foundation first, it is possible to see how different communities talk about the same big ideas in their own unique ways. This approach offers a more reliable way for historians and social scientists to compare diverse sources of information, ensuring that when they trace a theme from one century to the next, they are truly following the same story, even if the words on the page have changed.

Technical Summary: Dynamic Topic Modeling for Cross-Corpus Temporal Analysis

Problem Definition
Dynamic Embedded Topic Models (D-ETM) offer an interpretable framework for modeling the semantic evolution of topics over time within a single corpus. However, applying these models to cross-corpus analysis presents a significant challenge: standard approaches train topic models independently on each corpus and attempt to align the resulting topics post-hoc. This process fails to guarantee stable topic correspondence across different corpora and time periods. Because independently trained models assign arbitrary indices to semantic topics, comparing trajectories requires explicit alignment procedures (e.g., Hungarian matching), which are non-trivial in dynamic settings where topic identities must remain identifiable across both sources and time. Existing methods either treat alignment as a post-processing step or rely on joint training without mechanisms to preserve a shared coordinate system while allowing for corpus-specific lexical specialization.

Methodology
The authors propose a Shared-Backbone Dynamic Topic Model (SB-D-ETM) framework designed to learn a common semantic coordinate system across multiple corpora while permitting source-specific adaptation. The framework operates in two stages:

Shared Dynamic Topic Backbone (Stage 1):
- A single D-ETM is trained on a merged collection of all corpora.
- To prevent larger corpora from dominating the learning process, a source-aware weighting scheme is applied during training, ensuring each corpus contributes equally to the shared objective.
- This stage learns a set of shared dynamic topic trajectories ( $\alpha^{(0)}_{k,t}$ ) and a shared word embedding matrix ( $\rho$ ). These parameters define a common "backbone" where topic index $k$ represents a specific thematic trajectory across all sources and time slices.
Source-Specific Residual Adaptation (Stage 2):
- The shared backbone parameters ( $\alpha^{(0)}_{k,t}$ and $\rho$ ) are frozen.
- For each source $s$ , the model learns a source-specific residual offset ( $\Delta\alpha_{s,k,t}$ ) for each topic and time slice.
- The adapted topic embedding is defined as $\tilde{\alpha}_{s,k,t} = \alpha^{(0)}_{k,t} + \Delta\alpha_{s,k,t}$ .
- This design allows each corpus to specialize lexically (drift from the shared topic) while maintaining the same topic index $k$ across all corpora.
- Two regularization terms are applied to the residuals:
  - Anchor Penalty: Keeps the magnitude of the residual drift small, ensuring the adapted topics remain close to the shared backbone.
  - Temporal Smoothness Penalty: Ensures the residual drift evolves smoothly over time, preventing abrupt changes in corpus-specific specialization.

Key Contributions

Framework Proposal: The introduction of a shared-backbone D-ETM framework that treats cross-corpus comparability as a modeling assumption rather than a post-hoc alignment problem. It enables corpus-specific adaptation while preserving aligned topic identities.
Empirical Evaluation: A comprehensive evaluation comparing independent training, full fine-tuning, and the proposed residual adaptation across three temporally structured corpora (Corpus of Historical American English, Harvard Business Review, and International Labour Review) spanning 97 years.
Qualitative Analysis: Case studies demonstrating how the framework supports comparative over-time interpretation, showing that the shared backbone captures historically meaningful macro-trends while residuals capture domain-specific lexical variations.

Results
The framework was evaluated using predictive fit (perplexity), topic quality (coherence, diversity), and novel trajectory-level alignment metrics (Same-index JSD, Trajectory Margin, Retrieval@1).

Alignment Performance: The proposed method (SB-RA) achieved significantly stronger cross-corpus alignment than both independent training with post-hoc matching and full fine-tuning (SB-FT).
- Retrieval@1: SB-RA achieved 97.5 ± 0.7%, meaning the inherited topic index correctly identified the corresponding topic in another corpus nearly 100% of the time. In contrast, full fine-tuning dropped to 17.9 ± 1.1%, and independent training required post-hoc matching to achieve any correspondence.
- Trajectory JSD: SB-RA maintained a low Same-index trajectory JSD (0.169), indicating that topics with the same index remained semantically close across corpora.
Predictive Fit and Quality: While full fine-tuning (SB-FT) achieved the lowest perplexity (best predictive fit), SB-RA improved upon the unadapted shared backbone and remained competitive with independently trained baselines. SB-RA preserved usable topic quality, though it showed sensitivity in coherence metrics for the ILR corpus, which the authors attribute to sparse word co-occurrence patterns rather than a loss of structure.
Ablation Studies: The results confirmed that the alignment preservation is driven primarily by the anchor penalty and the fixed-backbone parameterization. Removing regularizers or allowing full fine-tuning resulted in a collapse of same-index correspondence.
Sensitivity: The alignment advantage of SB-RA over SB-FT was consistent across different numbers of topics ( $K=10, 20, 30$ ).

Significance and Claims
The paper claims that incorporating topic alignment directly into the model architecture supports more stable over-time cross-corpus comparisons. By learning a shared dynamic coordinate system and modeling deviations as residuals, the framework allows researchers to trace thematic evolution across heterogeneous corpora without relying on unstable post-hoc matching procedures. The authors emphasize that this approach retains corpus-specific lexical variation while ensuring that topic indices serve as reliable semantic coordinates across time and source. The findings suggest that for diachronic cross-corpus analysis, preserving a stable semantic coordinate system is critical for interpretable comparative research. Future work is suggested to explore human assessments of interpretability and the extension of these principles to other architectures supporting explicit temporal trajectories.

Technical Summary: Dynamic Topic Modeling for Cross-Corpus Temporal Analysis

More like this