Pointer-Augmented Autoregressive Generation of Patent Claims with Joint Topology and Content Decoding

Technical Summary: Pointer-Augmented Autoregressive Generation of Patent Claims with Joint Topology and Content Decoding

Problem Statement
Autoregressive language models generate text as flat token sequences, lacking explicit mechanisms to reason about hierarchical relationships among output segments. This limitation is acute in patent claim generation, where a claim set forms a "Claim Dependency Forest." In this structure, independent claims root trees, and dependent claims must cite antecedent claims through hierarchical references. Crucially, the hierarchy carries statutory constraints: the scope of protection must narrow monotonically with depth. Failure to adhere to these constraints (e.g., a dependent claim broadening its parent's scope or failing to cite a valid antecedent) constitutes a ground for rejection by patent offices.

Existing approaches suffer from three primary limitations:

1. Architectural Separation: Standard autoregressive decoding cannot model non-linear parent/child dependencies, often producing broken dependency chains. Methods that predict topology in isolation from content or enforce surface grammar without semantic consistency fail to ensure that a dependent claim's wording reflects its parent's scope.
2. Lack of Scope Regularization: No existing method provides an inductive bias for the statutory requirement that scope narrows with depth, leading models to generate dependent claims that inadvertently broaden or contradict their parents.
3. Absence of Negative Signals: Aligning models with these constraints requires contrastive examples of deficient outputs. However, public patent corpora consist almost exclusively of granted applications, leaving no "negative signal" (defective forests) for the model to learn from.

Methodology: SPG (Structure-aware Patent Generation)
The authors propose SPG, a framework that predicts topology inside the autoregressive pass, ensuring that the representation emitting a claim's tokens is the same one deciding where the claim attaches. The framework operates in two stages:

• Stage I: Structure-Aware Training

  • Pointer-Augmented Decoding: The autoregressive decoder is augmented with a pointer head. At the opening delimiter of each dependent claim, this head selects a parent from the set of preceding claims. Crucially, the pointer reads the same hidden states used for token prediction. This allows gradients from the parent-selection task to reshape the decoder's representations, making the decoder inherently topology-aware.
  • Auxiliary Objectives:
    • Structure Consistency Loss: A cross-entropy loss supervises the pointer's parent selection against ground-truth links.
    • Hierarchical Scope Regularization: A depth-adaptive margin constraint encourages the representation of a dependent claim to remain close to its parent's representation. The allowed radius for this distance shrinks geometrically with depth, enforcing the statutory requirement that scope narrows as the tree deepens.

• Stage II: Legal Preference Optimization (LPO)

  • To address the lack of negative signals in granted patent corpora, the authors synthesize deficient candidates by sampling from the Stage I policy at different temperatures.
  • A deterministic scoring function (PatentScorerule) evaluates these candidates based on claim count, structural patterns, antecedent consistency, length, and content overlap.
  • Violation-Weighted Objective: Unlike standard Direct Preference Optimization (DPO), which treats all preference pairs equally, SPG scales the contrastive loss by the severity of the deficiency in the rejected sample. This prevents "mode collapse," where the model might treat a missing semicolon with the same penalty as a missing entire subtree.

Key Contributions

1. Joint Decoding: A pointer head predicts parent dependencies within the autoregressive pass, allowing topology and content to be learned in a single forward pass without a separate parsing stage.
2. Scope Regularization: A depth-adaptive margin with geometrically decaying radii tightens the parent-child coupling as depth increases, acting on the specific pairs identified by the pointer objective.
3. Severity-Weighted Alignment: By scaling preference pairs based on the deficiency of the rejected sample, the method averts mode collapse, a failure mode induced by uniform weighting in structured generation tasks.
4. Empirical Dissociation: The study demonstrates that structural compliance does not scale with parameter size or domain pretraining. Models ranging from 6B to 141B parameters fail to learn hierarchical competence without explicit inductive biases, whereas fluency scales naturally.

Results
Evaluated on the HUPD-DCG dataset using a Llama-3-8B-Instruct backbone:

• Structural Accuracy: SPG recovers 79.0% of gold parent links (Dependency-Edge Accuracy), a metric never directly supervised by the training reward.
• Referential Consistency: The model improves antecedent consistency (AC) from 0.292 (supervised baseline) to 0.478, indicating better resolution of references across deep tree structures.
• Fluency: The model achieves state-of-the-art BLEU (37.66) and ROUGE-1 (64.18) scores among fine-tuned models, matching the fluency of the supervised baseline while significantly outperforming it in structural metrics.
• Expert Evaluation: Two patent practitioners rated the full model significantly higher (4.05 overall) than the supervised baseline (3.52) and zero-shot baselines, corroborating the automatic metrics.
• Scaling Laws: Increasing model size from 6B to 141B parameters improved fluency but did not improve structural pattern scores, confirming that hierarchical competence requires explicit architectural bias rather than capacity.

Significance and Claims
The paper argues that for generation tasks where outputs carry enforceable structure (legal or programmatic), structural decisions must be integrated into the generation process rather than handled post-hoc or via surface constraints. The authors posit three main findings:

1. Scale is not a substitute for structural supervision: Explicit structural mechanisms are required before increasing model size when tasks involve relations between non-adjacent segments.
2. Preference optimization requires severity-aware weighting: In structured generation, deficiencies are not exchangeable; weighting gradients by the severity of the violation is necessary to prevent collapse.
3. Structure and content are better separated in time: Over-constraining the structural scaffold first (Stage I) and then optimizing content within that scaffold (Stage II) yields better results than balancing competing objectives simultaneously.

The authors maintain modesty regarding their claims, noting that the system is intended to assist patent professionals and requires expert review. They acknowledge limitations regarding jurisdiction-specific dependency rules (e.g., multiple dependencies in EPO vs. single-parent in USPTO) and the fact that their evaluation relies on single runs without significance testing. The work focuses strictly on claim text and dependency structure, excluding broader patent drafting strategies like novelty argumentation.