Technical Summary: Large Language Models for Patent Classification

Problem Statement

Patent classification into Cooperative Patent Classification (CPC) codes is a foundational task for analyzing technological change, yet it presents significant challenges due to its hierarchical, multi-label, and highly imbalanced nature. While supervised encoder-based models (e.g., BERT, SciBERT) have become the de facto standard for large-scale classification, they exhibit a well-documented bias toward frequent subclasses, struggling to accurately classify rare, emerging, or cross-domain technologies. Conversely, Large Language Models (LLMs) offer zero-shot and few-shot capabilities that may better handle these "long-tail" categories. However, the extent to which LLMs can complement encoders in this specific domain, and whether their benefits justify their substantially higher computational and environmental costs, remains an open empirical question.

Methodology

The authors conduct a systematic comparison between supervised encoder-based classifiers and open-weight instruction-tuned LLMs using the USPTO-70k benchmark dataset (70,250 patents). The study employs a temporally split protocol (training: 2006–2017; validation: 2018; test: 2019) to emulate realistic deployment conditions.

Models Evaluated

1. Encoder Baselines: Three Transformer-based encoders were fine-tuned on the training data:
   • BERT: General-purpose.
   • SciBERT: Domain-adapted to scientific literature.
   • PatentSBERTa: Domain-adapted specifically to patent corpora.
2. Large Language Models: Four instruction-tuned LLMs in the 7–8B parameter range (chosen for feasibility on single-GPU workstations) were evaluated without full fine-tuning initially:
   • Phi-3 Mini, Mistral-7B, LLaMA-3.1-8B, and Qwen-2.5-7B.

Inference Strategies for LLMs

LLMs were evaluated under four prompting configurations to assess the impact of context:

• Zero-shot: No examples provided.
• Few-shot: Includes 5 labeled examples (2 static, 3 dynamically retrieved based on semantic similarity).
• Retrieval-Augmented Generation (RAG): Retrieves the top-20 CPC subclass definitions (codes and descriptions) relevant to the patent text to constrain the output space.
• Hybrid: Combinations of Few-shot and RAG.

Adaptation and External Validation

• Parameter-Efficient Fine-Tuning: The best-performing LLM (Qwen-2.5-7B) was further adapted using LoRA (Low-Rank Adaptation) to test if lightweight supervised learning could close the performance gap.
• External Validation: To assess generalizability, the models were evaluated on an independent dataset of 10,000 European Patent Office (EPO) patents (2019–2022) without retraining.
• Efficiency Metrics: Computational cost, inference time, and energy consumption (CO2 emissions) were quantified using the CodeCarbon framework.

Evaluation Metrics

Performance was measured using Micro-F1 (dominated by frequent labels), Macro-F1 (label-balanced), and Hierarchical F1 (accounting for CPC taxonomy structure). Statistical significance was assessed using paired non-parametric tests (Wilcoxon signed-rank) on per-subclass performance.

Key Results

1. Performance Trade-offs: Frequency vs. Rarity

• Encoder Dominance on Frequent Subclasses: Supervised encoders, particularly SciBERT, achieved superior Micro-F1 scores (0.439 vs. 0.182 for the best LLM configuration). This performance is driven by their ability to correctly classify the high-frequency subclasses that dominate the dataset.
• LLM Advantage on Rare Subclasses: LLMs, especially when augmented with Few-shot + RAG, demonstrated significantly better performance on rare and infrequent subclasses (bottom 20% of the frequency distribution). They achieved higher Macro-F1 and Hierarchical F1 scores, indicating better coverage of the long tail.
• LoRA Limitations: Parameter-efficient fine-tuning (LoRA) of Qwen-2.5-7B did not yield consistent improvements over prompt-based inference alone, suggesting that for this task, inference-time contextualization (prompting and retrieval) is more effective than lightweight supervised adaptation.

2. Computational and Environmental Costs

• Efficiency Gap: Encoder-based models are orders of magnitude more efficient. They complete inference in 1–2 minutes with energy consumption of ~0.01 kWh.
• LLM Overhead: LLM inference requires 30 minutes to over 4 hours per run, consuming 0.2–2.1 kWh. Encoders are approximately 15–250× faster and 20–200× more energy-efficient than LLMs.
• Zero-shot Inefficiency: Zero-shot LLM configurations were found to be particularly inefficient, offering low classification performance at high energy costs.

3. External Validity

The performance trade-offs observed on the USPTO dataset were robust when applied to the EPO dataset. Encoders maintained superior Micro-F1, while LLMs continued to show better coverage of rare subclasses, confirming that the findings are not specific to a single patent office or dataset.

Significance and Contributions

The paper argues that neither approach is universally superior; rather, they play complementary roles in patent classification:

1. Hybrid Potential: The study motivates hybrid classification approaches where efficient encoders handle routine, high-volume classification of established technologies, while LLMs are selectively deployed to assist with rare, emerging, or cross-domain technologies where encoders exhibit low recall.
2. Methodological Reflexivity in Scientometrics: The authors highlight that the choice of classification model systematically shapes the visibility of technological categories in downstream analyses. Relying solely on encoders may introduce a bias that renders emerging or weakly institutionalized technologies "invisible" in patentometric indicators.
3. Responsible AI Deployment: The findings underscore the need to balance predictive performance with sustainability. The high energy cost of LLMs makes them unsuitable for indiscriminate, large-scale application, suggesting their use should be targeted and purpose-driven.

In conclusion, the paper provides a controlled empirical basis for understanding the trade-offs between the scalability of encoder-based models and the semantic flexibility of LLMs, advocating for a pluralistic approach to patent classification that accounts for data imbalance, computational constraints, and the specific goals of the analysis.