💬 NLP

Is It Novel and Why? Fine-Grained Patent Novelty Prediction Based on Passage Retrieval

This paper introduces FiNE-Patents, a fine-grained dataset and a novel LLM-based workflow that shifts patent novelty prediction from coarse binary classification to a granular retrieval and reasoning task, demonstrating superior performance in identifying specific prior art passages and robustness against spurious correlations.

Original authors: Valentin Knappich, Anna Hätty, Simon Razniewski, Annemarie Friedrich

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Valentin Knappich, Anna Hätty, Simon Razniewski, Annemarie Friedrich

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a patent examiner. Your job is to decide if a new invention is truly "new" or if someone else has already described it in a library of old documents (called "prior art").

Traditionally, computer programs tried to help with this by acting like a yes/no security guard. You'd feed them a patent claim and a library of old documents, and the computer would just shout, "Novel!" or "Not Novel!" based on a simple guess.

The problem, as this paper explains, is that these security guards were cheating. They weren't actually reading the documents to find the truth. Instead, they were looking for shortcuts. For example, they noticed that if a patent claim was very long, it was usually "new" (because inventors add details to fix old rejections). If the claim was short, it was usually "old." The computer learned to just count the words and guess, without ever understanding the invention.

This paper introduces a new, smarter way to do this, called FiNE-Patents.

The New Approach: The "Detective" vs. The "Guessing Game"

Instead of asking the computer to just guess "Yes" or "No," the authors ask it to act like a detective who has to solve a puzzle step-by-step.

Here is how their new system works, using a simple analogy:

1. The Old Way (The "Guessing Game"):
Imagine a student taking a test. The teacher asks, "Is this story new?" The student doesn't read the story. Instead, they notice the story is 10 pages long. They remember that "long stories are usually new," so they write "Yes." They get the right answer, but they didn't actually learn anything. This is what the old computer models were doing.

2. The New Way (The "Detective"):
The new system (FiNE-Patents) forces the computer to break the patent down into tiny pieces, like individual Lego bricks.

Step 1: Break it down. The computer takes the big patent claim and splits it into small features (e.g., "a camera," "a specific angle," "a moving screen").
Step 2: Find the evidence. For each tiny Lego brick, the computer has to go into the library of old documents and find the exact page where that specific brick was mentioned before. It's like saying, "Show me the paragraph in the old book that talks about this specific camera."
Step 3: Make the verdict. Only after finding the evidence for every single piece does the computer decide: "Okay, this specific camera was old, but this specific angle was new. Therefore, the whole invention is new."

The New Dataset: A "Gold Mine" of Clues

To teach the computer this detective work, the authors built a massive new dataset called FiNE-Patents.

Where did it come from? They looked at real rejection letters from the European Patent Office. When an examiner rejects a patent, they write a detailed report (called an ESOP) explaining exactly which parts of the invention are old and pointing to the specific pages in the old documents that prove it.
What's special about it? Previous datasets were like a blurry map that just said, "The old stuff is somewhere in this book." This new dataset is like a highlighted textbook. It says, "Feature A is on page 5, paragraph 2. Feature B is on page 10, line 4."
The Size: They collected 3,658 patent claims with these precise, page-by-page clues.

The Results: Smart Models vs. Cheating Models

The authors tested this new system using powerful AI models (Large Language Models, or LLMs) and compared them to the old "cheating" models.

The Cheaters: The old models (like a standard BERT classifier) were great at the "Yes/No" test if they were allowed to use their shortcuts (like counting words). They got 75% accuracy. But, when the authors created a "trick test" where the word count didn't matter, the cheaters failed miserably. They realized the models hadn't actually learned to read; they just memorized patterns.
The Detectives: The new LLM-based workflows, which broke the patent down into features and hunted for evidence, were much better at finding the specific pages in the old documents.
- They didn't rely on shortcuts. Even on the "trick test," they stayed strong.
- They were much better at finding the exact sentences in the old documents that matched the new invention.

The Big Takeaway

The paper argues that we need to stop treating patent novelty as a simple "Yes/No" question. It's too complex for that.

Instead, we should treat it like a forensic investigation. We need systems that can:

Break an invention into small parts.
Find the specific evidence for each part.
Explain why it is new or old by pointing to the exact source.

The authors released their new dataset and their "detective" code to the public. They hope this will help build AI tools that patent examiners can actually trust, because these tools won't just guess—they will show their work and prove their conclusions with evidence.

Technical Summary: Is It Novel and Why? Fine-Grained Patent Novelty Prediction Based on Passage Retrieval

Problem Definition

Patent novelty assessment is a critical component of the patent examination process, requiring examiners to determine if an invention is disclosed in prior art. Traditionally, this has been modeled as a binary classification task at the claim level (novel vs. not novel). However, the authors argue that this formulation is flawed for two primary reasons:

Lack of Granularity: A binary label provides no insight into which specific features of a claim are disclosed or novel, limiting its practical utility for patent professionals.
Spurious Correlations: Prior work has shown that claim-level classifiers often achieve high accuracy by exploiting superficial shortcuts (e.g., claim length, reference numerals, or specific linguistic patterns) rather than performing genuine semantic reasoning about the relationship between the claim and prior art.

The paper posits that a robust novelty assessment system must operate at the feature level, requiring models to identify specific passages in prior art that disclose individual claim features and to determine which features render a claim novel.

Methodology

The FiNE-Patents Dataset

To support fine-grained evaluation, the authors introduce FiNE-Patents, a dataset of 3,658 first patent claims derived from European Patent Office (EPO) examination records.

Source: The dataset is constructed from European Search Opinion (ESOP) documents, where examiners reject initial applications (A1) for lack of novelty and provide detailed, feature-level references to the closest prior art document ( $D$ ).
Annotation: Unlike previous datasets that use aggregated paragraph ranges from European Search Reports (ESR), FiNE-Patents extracts precise passage references for each specific claim feature.
Mitigation of Biases: To address spurious correlations, the authors implemented:
- Length Stratification: Sub-sampling to ensure novel and non-novel claims have identical length distributions.
- Reference Numeral Removal: Stripping reference numerals (e.g., "device (10)") which are often added during prosecution and serve as strong shortcuts.
- Adversarial Test Set: A subset of 278 claims filtered to be misclassified by a standard BERT classifier trained on claim text alone, serving as a diagnostic for models relying on shortcuts.

Evaluation Protocol

The authors propose a three-subtask evaluation framework:

Passage Retrieval: Retrieving relevant passages from the prior art document for each claim feature.
Novel Feature Identification: Identifying which features of a claim are novel (typically corresponding to text added during prosecution to overcome objections).
Claim Novelty Prediction: Predicting the binary novelty status of the entire claim.

LLM Workflows

The authors implement and evaluate Large Language Model (LLM) workflows using open-weight models (Qwen series) via DSPy. They compare two approaches:

Single-Step Examination: The model receives the full claim and prior art document and outputs a structured response (passages, feature novelty, claim novelty) in one step.
Hierarchical Examination: A multi-step process where:
- The claim is segmented into features.
- Each feature is analyzed individually against the prior art to retrieve passages and determine novelty.
- Results are aggregated to form a final claim-level prediction.

Key Results

Passage Retrieval and Feature Identification

Performance: LLM-based workflows significantly outperform embedding-based baselines (e.g., Qwen3-8B-Embeddings) and lexical overlap baselines (ROUGE-L).
Hierarchical Advantage: The hierarchical workflow yields consistent improvements in passage retrieval and novel feature identification. For example, on the 30B model, hierarchical examination improved the F1 score for novel feature identification from 38.6 to 53.7.
Granularity: Performance differences between models are more pronounced at the feature level than the claim level, validating the need for fine-grained evaluation.

Claim Novelty Prediction

Baseline Vulnerability: A fine-tuned BERT classifier achieved 75.2% accuracy on the test set without accessing the prior art document, confirming the presence of exploitable spurious correlations.
LLM Robustness: In contrast, LLM workflows (particularly those with "thinking" modes) demonstrated robustness. When evaluated on the adversarial test set (where the BERT classifier failed), the LLMs maintained performance, showing low agreement with the BERT classifier (negative Cohen's Kappa on the adversarial set). This suggests LLMs rely on semantic reasoning rather than dataset shortcuts.
Prediction Bias: While reasoning-enabled models showed balanced predictions, the non-thinking Qwen3-30B-Instruct exhibited extreme bias, predicting only 3.2% of claims as novel.
Limitations: Hierarchical examination did not improve claim-level prediction accuracy and, in some cases, reduced it (e.g., dropping to 57.9% accuracy for the 30B model), suggesting that intermediate summaries may introduce noise.

Significance and Contributions

The paper claims the following contributions:

Paradigm Shift: It challenges the binary classification paradigm for patent novelty, arguing for a shift toward feature-level retrieval and reasoning to ensure transparency and practical utility.
FiNE-Patents Dataset: It provides the first dataset with fine-grained, feature-level prior art references extracted from ESOP documents, enabling rigorous evaluation of model reasoning.
Robustness Evidence: It demonstrates that while traditional classifiers are susceptible to spurious correlations in patent data, LLMs can be robust against these shortcuts if prompted to perform structured reasoning.
Actionable AI: The proposed workflows offer a path toward AI systems that provide transparent, verifiable assistance to patent examiners by explicitly linking predictions to specific source passages, rather than providing opaque binary labels.

The authors conclude that while absolute performance levels still require improvement, the ability of LLMs to avoid spurious correlations and perform granular analysis represents a significant step toward reliable, transparent patent analysis tools.