Anticipating Innovation Using Large Language Models

Technical Summary: Anticipating Innovation Using Large Language Models

Problem Statement
Forecasting innovation, defined as the emergence of new technological combinations, remains a fundamental challenge for science and policy. While genuine innovations are unprecedented by definition, the "adjacent possible" framework suggests that the space of realizable innovations is bounded by current knowledge configurations. The core difficulty lies in identifying which combinations at this frontier are most likely to be actualized before they occur. Existing approaches often rely on network science techniques applied to International Patent Classification (IPC) co-occurrence networks or static embeddings of patent texts. However, these methods often fail to capture the dynamic, linguistic evolution of how technologies are described across the corpus prior to their formal combination, or they treat IPC codes as static entities rather than context-dependent concepts.

Methodology
The authors introduce TechToken, a transformer-based language model designed to treat technological classification codes as integral parts of the language model's vocabulary.

• TechToken Architecture: Unlike standard approaches that generate a single vector for a patent and average them to represent an IPC code, TechToken expands the vocabulary of a pre-trained BERT model by introducing new tokens corresponding to specific IPC codes. During fine-tuning on patent abstracts, the model processes sequences in the format: [CLS] patent title [SEP] patent abstract [SEP] [TT1] [TT2] ... [TTN], where TTi represents an IPC code.
• Contextual Embeddings: This architecture allows the attention mechanism to operate directly between natural language words and technological tokens. Consequently, each IPC code acquires a distinct embedding for every patent in which it appears, capturing the polysemy of technologies across different domains.
• Context Similarity (CS): To quantify linguistic convergence, the authors define CS as the cosine similarity between the embeddings of two technologies. For TechToken, which generates multiple embeddings per code (one per patent), CS is calculated as the average of the top 1% highest cosine similarities among all possible pairs of token embeddings from the two codes across the corpus. This aggregate statistic serves as a global, emergent measure of how the linguistic contexts of two technologies are converging.
• Innovation Definition: The study defines a "realized innovation" not merely as a first co-occurrence, but as a statistically significant one. Using a Chung-Lu null model, the authors calculate a z-score for co-occurrences to distinguish genuine novel combinations from random noise, ensuring the prediction targets high-impact innovations.
• Baseline Comparisons: The study compares TechToken against:
  • Average Embedding: Standard methods where patent embeddings (generated by BERT4Patents or LLaMA 3.1) are averaged to represent an IPC code.
  • Large Language Models: Including a fine-tuned LLaMA 3.1 8B model.
  • State-of-the-Art Models: Specialized models like PatentSBERTa and Paecter for specific tasks.

Key Results

• Predictive Power of Linguistic Convergence: The analysis demonstrates that linguistic convergence precedes technological convergence. The Context Similarity (CS) between two IPC codes begins to rise significantly years—often up to two decades—before the codes first co-occur in a patent. This signal is an emergent property of the entire corpus, not attributable to any single document or inventor.
• Superior Forecasting Performance: TechToken substantially outperforms all baseline models in predicting first co-occurrences of IPC codes.
  • In terms of AUC-ROC, TechToken (BERT-based) achieves scores around 0.936–0.940, significantly surpassing the fine-tuned LLaMA 3.1 8B model (approx. 0.856) and standard BERT4Patents models (approx. 0.725–0.765).
  • Notably, the TechToken approach enables a relatively small BERT-scale model to outperform a model roughly 25 times larger in parameter count.
  • Performance improves when focusing on the most statistically significant innovations (higher z-scores), indicating the model captures technological meaning rather than mere patent volume trends.
• General Representation Quality: Beyond forecasting, TechToken achieves state-of-the-art performance on standard patent-related tasks, including IPC classification, citation prediction, and title–abstract matching, confirming the high quality of its internal representations.

Significance and Claims
The paper claims two primary contributions:

1. Empirical: It provides evidence that the "adjacent possible" leaves a legible, quantifiable trace in the collective language of patents well before new combinations are actualized. The study posits that shifts in how communities of practice write about technologies are an early, observable signature of impending innovation. This signal is distributed and emergent, visible only at the level of the full patent population.
2. Methodological: It introduces a general framework for embedding symbolic classification codes directly into a language model's vocabulary. By treating codes as tokens, the model learns context-dependent representations of structured entities. The authors suggest this approach is broadly applicable to any domain where documents are tagged with controlled vocabularies (e.g., scientific publications with MeSH codes or trade classifications).

The authors conclude that the collective language of invention acts as a leading indicator of technological change. They argue that relatively lightweight language models, when equipped with the appropriate symbolic scaffolding (like TechToken), can read these indicators with precision, offering a principled and early window into future technological possibilities.

Limitations
The authors acknowledge that their analysis operates at the IPC group level, which may coarse-grain the technological space and miss recombination within a single group. Additionally, the current framework focuses on pairwise combinations, whereas innovation often involves higher-order assemblies of three or more technologies. These are identified as natural extensions for future work.