Mapping Scientific Literature with Large Language Models and Topic Modeling

Technical Summary: Mapping Scientific Literature with Large Language Models and Topic Modeling

Problem Statement
Scientific literature is increasingly fragmented by disciplinary boundaries, specialized terminology, and sparse keyword systems, hindering the ability to capture the evolving structure of modern science. Traditional bibliometric mapping methods (e.g., co-word analysis, citation networks) often rely on subjective keyword selection or suffer from text sparsity. Statistical topic modeling techniques, such as Latent Dirichlet Allocation (LDA) and Non-Negative Matrix Factorization (NMF), struggle with the complexity of scientific texts, often producing topics that are collections of words ("tea leaves") lacking semantic interpretability. Furthermore, newer deep embedding methods like BERTopic still face challenges regarding interpretability and the determination of an optimal number of topics without ground truth. Existing Large Language Model (LLM) approaches for topic modeling often rely on outdated evaluation metrics, arbitrary topic seeding, or lack robust mechanisms for filtering noisy documents and outliers.

Methodology
The study proposes a hybrid, two-stage classification pipeline designed to autonomously map the thematic landscape of scientific research without relying on prior seed topics. The framework was applied to a 20-year corpus (2005–2024) of 1,519 engineering-related articles from the Proceedings of the National Academy of Sciences (PNAS).

1. Primary Abstract Classification (Unsupervised LLM-Refined Clustering):

   • Embedding and Clustering: Abstracts are encoded into dense vectors using text-embedding-3-small. An iterative K-means clustering process partitions the abstracts into $k$ clusters.
   • LLM Annotation and Refinement: For each cluster, an LLM (GPT-4o-mini) generates a thematic title and description.
   • Consensus Reclassification: To mitigate hallucinations and stochasticity, the LLM reclassifies each abstract into the generated themes five times. A "3-out-of-5" consensus criterion is applied; an abstract is assigned a label only if at least three runs agree.
   • Agreement Scoring and Recursive Refinement: An agreement score ($A_j^t$) measures the alignment between the original K-means cluster and the LLM reclassification. Clusters with scores below a threshold ($\tau = 0.60$) are deemed unstable. Their constituent abstracts are carried forward to the next iteration for re-clustering.
   • Termination: The recursion continues until the remaining unclassified abstracts fall below a termination threshold ($\delta = 0.10$ of the initial dataset size). These remaining abstracts are grouped into an "Other" category.
   • Output: A stable set of $n^*$ thematic categories with human-interpretable labels.

2. Secondary Full-Text Classification (Multi-Label Analysis):

   • Full-text PDFs are parsed into segments (paragraphs/sentences).
   • Unlike the primary stage, which assigns a single topic per abstract, the LLM classifies each text segment against the stable set of themes.
   • Segments can be assigned multiple topics, revealing latent cross-topic connections and methodological applications not evident in the abstract.

3. Evaluation and Validation:

   • Automated Metrics: Coherence (CV, UMass), Topic Diversity (TD), and Jaccard similarity were calculated against baselines (LDA, NMF, BERTopic, TopicGPT, PDDP).
   • Manual Validation: A blind review of 20% of abstracts (303 samples) assessed Top-1 and Top-3 accuracy against human judgment.
   • NLP Analysis: Bag-of-Words (BoW) and class-based TF-IDF (c-TF-IDF) were used to verify that LLM-generated topics corresponded to meaningful linguistic patterns and statistical distributions (e.g., Zipf's law).
   • Bipartite Network: A graph linking abstract-level topics to segment-level classifications was constructed to visualize cross-topic relationships and compare them against PNAS's editorial dual-classification system.

Key Contributions

• A Hybrid Framework: Introduces a structured, iterative protocol combining embedding-based clustering with LLM-guided refinement to discover topics without seed inputs.
• Robustness Mechanisms: Implements a "3-out-of-5" consensus rule and an explicit "Other" category to filter noise and outliers, addressing the instability often found in LLM-based topic modeling.
• Dual-Level Analysis: Moves beyond abstract-only analysis by utilizing full-text segmentation to identify secondary themes and cross-topic relationships, offering a more granular view of interdisciplinary connections.
• Validation without Ground Truth: Demonstrates a methodology for evaluating topic models in the absence of ground-truth labels by combining automated metrics, manual validation, and traditional NLP statistical checks.

Results

• Topic Discovery: The pipeline identified 16 stable research topics (e.g., Tissue Engineering, Diagnostic Technologies, Material Science) and an "Other" category (9.8% of abstracts).
• Performance: The proposed method achieved the highest topic diversity (TD = 0.909) and lowest Jaccard similarity (6.15 × 10⁻³) among compared models, indicating distinct, non-overlapping topics. While MALLET LDA showed higher coherence scores, the LLM approach offered superior interpretability and diversity.
• Accuracy: Manual validation yielded a Top-1 accuracy of 52.5% and a Top-3 accuracy of 75.9%. The higher Top-3 score reflects the cross-topic nature of the corpus, where many articles span multiple subdomains.
• Cross-Topic Insights: The bipartite network analysis revealed that while some topics (e.g., Material Science) are highly specialized (low Lift), others (e.g., Tissue Engineering, Synthetic Biology) act as hubs with high interdisciplinary Lift, often bridging engineering with biological sciences. The full-text analysis showed that 75% of text segments were assigned multiple topics, highlighting the limitations of single-label abstract classification.
• Linguistic Consistency: c-TF-IDF and BoW analyses confirmed that the LLM-generated topics aligned with distinct vocabulary distributions and followed known statistical laws (Zipf's law), suggesting the model uncovered meaningful semantic structures rather than arbitrary groupings.

Significance and Claims
The paper claims that the proposed framework offers a powerful tool for mapping science and identifying emerging cross-topic connections without prior knowledge of a journal's schema. It demonstrates that LLMs can independently recover much of the editorial dual-classification structure of PNAS while revealing latent relationships not observable through keywords or abstracts alone.

The authors emphasize that this approach translates dense, specialized jargon into a shared, interpretable thematic space, potentially aiding in science communication and meta-research. However, the paper remains modest regarding the limitations of LLMs, acknowledging the risk of hallucination and the need for structured safeguards (like consensus voting and shuffling inputs). The study concludes that while LLMs are not a replacement for human judgment, their integration with traditional NLP validation techniques provides a flexible, robust approach for exploratory analysis in domains lacking formal ground-truth labels. The framework is presented as a dynamic system capable of tracking the rise and fall of research paradigms over time, particularly through the monitoring of the "Other" category as a trigger for re-evaluation.