Technical Summary: Self-Filtered Distillation with LLM-generated Trust Indicators for Reliable Patent Classification

1. Problem Statement

Organizing large-scale patent corpora is a critical information management task essential for prior art retrieval, technology discovery, and intellectual property decision-making. However, automated patent classification faces unique challenges, including specialized terminology, complex claim structures, and strategic ambiguity, making it more difficult than general-domain text classification.

Recent approaches attempt to leverage Large Language Models (LLMs) to generate natural language rationales (explanations) to improve interpretability and train smaller "student" models via knowledge distillation. However, a significant flaw exists in this paradigm: LLM-generated rationales often contain logical inconsistencies, label mismatches, and semantic misalignments with domain-specific taxonomies (e.g., the Cooperative Patent Classification or CPC system). Treating these rationales as ground-truth supervision indiscriminately absorbs these errors during training, undermining classification reliability and propagating errors into downstream information processes. Existing methods often lack a mechanism to quantitatively assess rationale quality during the learning process, leading to the propagation of hallucinations and domain drift.

2. Methodology: Self-Filtered Distillation (SFD)

The authors propose Self-Filtered Distillation (SFD), a framework that reinterprets LLM-generated rationales not as ground-truth labels, but as trust indicators. Instead of post-hoc correction, SFD embeds quality assurance directly into the training loop by dynamically modulating the contribution of each training instance based on a unified trust score.

The framework operates in three stages:

A. LLM-Based Prediction and Reasoning Generation

A teacher LLM processes an input patent text $x$ to generate:

1. A set of predicted labels $\hat{Y}$.
2. A natural language rationale $r$ explaining the prediction.
   The student model is trained to minimize loss against the teacher's labels, but the rationale $r$ is used solely to estimate sample reliability, not as a direct supervisory signal.

B. Reasoning Quality Evaluation (Unsupervised Trust Scoring)

To quantify trustworthiness without human annotation, SFD integrates three complementary unsupervised metrics into a Combined Trust Score (CTS):

1. Self-Consistency (SC): Measures the logical stability of the reasoning. The teacher model generates $k$ rationales for the same input. SC is calculated as the average pairwise cosine similarity of their sentence embeddings. High SC indicates stable reasoning; low SC suggests stochastic or unstable generation.
2. Class Entailment Alignment (CEA): Evaluates semantic coherence between the rationale and the assigned class definition. The system queries an LLM for the definition of the predicted CPC class ($d_{\hat{y}}$) and computes the cosine similarity between the rationale embedding and the class definition embedding. This ensures the reasoning aligns with the specific domain taxonomy.
3. LLM Agreement Scoring (LAS): Assesses external plausibility using an independent verifier LLM ($LLM_{judge}$). This external model evaluates the $(r, \hat{y})$ pair and outputs a confidence score, mitigating the generator's potential overconfidence.

The CTS is computed as the simple average of these three scores:
$$CTS(x) = \frac{1}{3}(SC(x) + CEA(x) + LAS(x))$$

C. Trust-Aware Training Control

The CTS is used to modulate the learning contribution of each sample in two ways:

1. Soft Weighting: The loss function for each sample is scaled proportionally to its CTS. Samples with low trust scores contribute less to the gradient updates.
2. Hard Thresholding: Samples with a CTS below a predefined threshold $\tau$ are completely excluded from training.

The final training objective minimizes the trust-weighted loss:
$$L = \sum_{x \in D} \mathbb{1}\{CTS(x) \ge \tau\} \cdot CTS(x) \cdot Loss(f_\theta(x), y)$$

3. Key Contributions

The paper outlines three primary contributions:

• Trust-Aware Selective Distillation: The proposal of SFD, which reinterprets LLM rationales as trust indicators and integrates three unsupervised metrics (SC, CEA, LAS) to dynamically modulate training contributions, advancing knowledge distillation under noisy supervision.
• Reliable Information Organization Framework: The trust score mechanism provides quantitative confidence measures for each decision, enabling auditability and transparent quality control over automated patent knowledge organization at scale.
• High-Stakes Domain Validation: The validation of SFD on patent classification, a domain where misclassification has tangible legal and economic consequences, demonstrating that trust-aware distillation is effective precisely where reliability is most critical.

4. Experimental Results

The framework was evaluated on the USPTO-2M benchmark, comprising over 2 million US utility patents across 637 CPC subclasses.

• Performance vs. Baselines: SFD achieved a 0.960 Micro-F1 score, surpassing the strongest baseline (Rationale-Augmented Distillation) by 1.6 percentage points. Notably, single-metric filtering baselines (e.g., Self-Consistency only or LLM-as-a-Judge only) underperformed even naive distillation, highlighting the necessity of a multi-faceted approach.
• Macro-F1 Improvement: On larger student architectures (e.g., Falcon3-3B), SFD achieved a 38.7% relative improvement in Macro-F1 compared to baselines, demonstrating that the trust mechanism effectively enables learning from high-quality, filtered data even for minority classes.
• Ablation Studies:
  • CEA was identified as the most critical metric; removing it caused a severe drop in performance (Subset Accuracy fell from 0.930 to 0.695), confirming that taxonomic alignment is essential for patent classification.
  • LAS provided complementary filtering against factual hallucinations.
  • SC primarily stabilized performance for minority classes.
• Model Agnosticism: The framework consistently improved performance across four diverse student architectures (ranging from 0.6B to 3.8B parameters), proving that the trust signal is independent of specific model inductive biases.
• Human Correlation: The trust scores showed a strong correlation with expert human judgments ($r = 0.685$), validating that the unsupervised metrics effectively approximate genuine reasoning quality.

5. Significance and Claims

The paper claims that SFD represents a paradigm shift in automated knowledge organization. Rather than relying on a sequential pipeline of "classify first, verify later," SFD embeds quality assurance directly into the learning process.

• From Post-Hoc to Intrinsic: The framework transforms quality control from an external auditing function into an intrinsic property of the learned model.
• Decomposable Confidence: By decomposing the trust score into SC, CEA, and LAS, the system provides "self-documenting" classification outcomes. If a decision has high internal stability but low taxonomic alignment, the system explicitly signals this specific failure mode, aiding human review.
• Scalability and Reliability: The authors argue that SFD resolves the tension between automation scale and organizational reliability. It allows for the processing of massive patent corpora while ensuring that the training signal itself is curated for trustworthiness, preventing error propagation in high-stakes intellectual property workflows.

The work concludes that while the trust mechanism drives performance improvements, the specific gains in Macro-F1 are capacity-dependent, suggesting that larger models are better able to leverage the high-quality, filtered samples to learn complex decision boundaries for rare classes.