💬 NLP

Heterogeneous Dependency Graph-Guided Attentionfor Patent Representation Learning

This paper introduces PHAGE, a novel Patent Heterogeneous Attention Graph Encoder that enhances patent representation learning by modeling claim-level dependency hierarchies through a typed graph and connectivity mask to overcome the limitations of flat token sequences in pre-trained language models.

Original authors: Yongmin Yoo, Qiongkai Xu, Zhangkai Wu, Longbing Cao

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yongmin Yoo, Qiongkai Xu, Zhangkai Wu, Longbing Cao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a complex legal document, like a patent. Traditionally, computers read these documents like a standard novel: they read from the first word to the last, line by line. They assume that words next to each other are related, and words far apart are not.

The Problem: The "Flat" Misunderstanding
In a patent, this "flat" reading is a disaster. A patent isn't just a story; it's a dependency tree.

Claim 1 might say: "A machine with a wheel."
Claim 2 might say: "The machine of Claim 1, where the wheel is made of rubber."
Claim 5 might say: "The machine of Claim 3, where the wheel is red."

Notice that Claim 5 depends on Claim 3, not the one right next to it (Claim 4). If a computer reads them in a straight line, it gets confused. It thinks Claim 5 is related to Claim 4, missing the crucial legal link back to Claim 3. It's like trying to understand a family tree by reading a list of names in alphabetical order instead of seeing who is the parent of whom.

The Solution: PHAGE (The "Smart Connector")
The authors created a new system called PHAGE (Patent Heterogeneous Attention-Guided Graph Encoder). Think of PHAGE as a smart librarian who doesn't just read the book; they first draw a map of how the claims connect before reading a single word.

Here is how PHAGE works, using simple analogies:

1. Drawing the Map (The Heterogeneous Graph)

PHAGE first builds a "Claim Dependency Graph." It looks at the patent and draws arrows between claims to show who depends on whom.

The "Legal" Arrows: These are the official, rock-solid links (e.g., "The method of Claim 1..."). The computer treats these like a highway—very reliable and fast.
The "Technical" Arrows: These are looser links based on how words are used (e.g., "a wheel" in one claim becoming "the wheel" in another). The computer treats these like local roads—useful, but maybe a bit noisier.

By separating these into different types of roads, PHAGE knows which connections are legally binding and which are just technical hints. It doesn't treat a noisy technical hint the same as a strict legal rule.

2. The Smart Traffic Light (Connectivity Mask & Bias)

Standard AI models (like Transformers) are like a crowded room where everyone shouts to everyone else at once. PHAGE puts up traffic lights.

The Mask (The Gatekeeper): This is a hard rule. If two claims aren't connected on the map, the traffic light turns red. The computer is forbidden from listening to them. It forces the AI to focus only on the relevant claims.
The Bias (The Volume Knob): Even if two claims are connected, they might need to be heard at different volumes. PHAGE has little "volume knobs" for each type of connection. It learns to turn up the volume on the "Legal" highways and turn down the volume on the "Technical" local roads. This helps the computer understand the strength of the relationship.

3. Learning from Two Angles (Dual-Granularity Training)

To get really good at this, PHAGE practices two different drills:

Drill A (The Big Picture): It learns to group patents that belong to the same category (like "cars" vs. "bicycles").
Drill B (The Inside Job): It learns to understand the internal structure of a single patent. It practices matching the "parent" claims with their "child" claims.

By doing both, the AI learns that the internal structure of a patent (how its own claims relate) is actually a stronger clue for understanding it than just looking at how it relates to other patents.

The Big Surprise

The most interesting finding is that the internal map matters more than the external network.
Usually, researchers try to help AI by showing it how millions of patents cite each other (a huge external web). PHAGE found that you don't need that massive web. If you just teach the AI to understand the internal family tree of a single patent very well, it becomes a much better expert.

The Best Part: No Map Needed Later

Here is the magic trick: PHAGE needs to draw the map to learn how to read. But once it has learned, it doesn't need the map anymore.
When you give PHAGE a new patent to analyze, it doesn't need to draw the graph first. It has already "memorized" the pattern of how claims connect. It just runs through the text normally, but its brain is now wired to automatically know which claims are related, even without the visual map.

In Summary:
PHAGE is a patent reader that stops reading in a straight line. Instead, it learns to see the hidden "family tree" inside every patent. It learns to distinguish between strict legal rules and loose technical hints, and once it learns this skill, it can read any new patent instantly without needing to draw a diagram first.

Technical Summary: Heterogeneous Dependency Graph-Guided Attention for Patent Representation Learning

Problem Statement

Pre-trained language models (PLMs) have advanced patent classification and retrieval by encoding claims as flat token sequences. However, this approach overlooks the inherent directed dependency hierarchy among patent claims. In patent documents, dependent claims inherit limitations from antecedent claims and add narrower technical details, creating a structure where adjacent text is not necessarily related, while distant claims may be directly connected via dependency references.

Current PLM-based encoders (e.g., BERT-for-Patents, PatentBERT) operate at the token level, capturing semantic correlations but failing to encode explicit legal dependencies. This conflation of structurally proximate yet unrelated claims blurs legal scope and technical dependencies. Furthermore, incorporating hierarchy into self-attention presents two specific challenges:

Heterogeneous Reliability: Claim dependencies involve relation types with varying reliability. Treating noisy technical relations indiscriminately with clean legal citation signals can corrupt the representation.
Granularity Mismatch: When defined over claims, dependency graphs do not align with the token-level operation of Transformers. Broadcasting claim-level adjacency can dilute structural information across unrelated token pairs.

Existing graph-based approaches often treat structure as external supervision or auxiliary input at the document level, rather than injecting claim-level dependencies directly into the encoder's representation flow.

Methodology: PHAGE

The authors propose the Patent Heterogeneous Attention-Guided Graph Encoder (PHAGE), a model that integrates claim-level dependency structure directly into the self-attention mechanism of a pre-trained encoder. The framework operates in three stages:

1. Construction of Heterogeneous Claim Dependency Graph (CDG)

PHAGE constructs a deterministic, heterogeneous Directed Acyclic Graph (DAG) $G = (V, E, \psi)$ where nodes are claims and edges represent dependencies. The edge set is categorized into four distinct relation types to handle varying extraction reliability:

$r_{cite}$ (Explicit Citation): Derived from formal legal references (e.g., "The method of claim 1..."). These are near-deterministic.
$r_{term}$ (Technical Term Flow): Traces the introduction of technical noun phrases (indefinite articles) and their subsequent reference (definite articles/demonstratives).
$r_{func}$ (Functional Coupling): Captures functional relationships via syntactic parsing of verb-object triplets (e.g., "couple," "connect").
$r_{both}$ : Assigned when both term-flow and functional-coupling signals co-occur, serving as a high-confidence signal.

The graph is represented by distinct adjacency matrices $\{A^{(r)}\}$ for each relation type, allowing the model to learn differential weights for each.

2. Relation-Aware Attention Mechanism

PHAGE injects the CDG into the self-attention mechanism of a Transformer encoder (specifically BERT-for-Patents) using two components:

Connectivity Mask ( $C$ ): A hard structural gate that restricts attention to structurally related claims. If a token pair $(u, v)$ does not correspond to a connected claim pair in the CDG, the attention score is masked to $-\infty$ . The [CLS] token is exempt to ensure unconstrained document-level aggregation.
Learnable Relation-Aware Bias ( $B^{(l)}$ ): A lightweight set of learnable scalar parameters added to the attention logits. These biases are conditioned on the specific edge type ( $r \in \{r_{cite}, r_{term}, r_{func}, r_{both}\}$ ) and the layer depth $l$ . This allows the model to modulate the strength of information flow based on dependency semantics without significantly increasing model capacity (only 5 scalar parameters per layer).

3. Dual-Granularity Contrastive Fine-Tuning

To refine the embedding space, PHAGE employs a joint objective function combining two levels of supervision:

Document-Level Objective ( $L_{doc}$ ): Aligns patent representations based on International Patent Classification (IPC) taxonomy using a contrastive loss (pulling patents in the same subclass together, pushing different subclasses apart).
Claim-Level Objective ( $L_{claim}$ ): Grounded in the CDG, this loss enforces topological proximity between claims. Crucially, it assigns learnable weights $w_r$ to each relation type, allowing the model to calibrate the alignment pressure based on the reliability of the edge type.

The final loss is $L = L_{doc} + \lambda L_{claim}$ .

Key Contributions

Intra-document Claim Topology as Inductive Bias: The authors identify that the directed dependency hierarchy among claims is an informative signal discarded by existing encoders. PHAGE injects this topology directly into the encoder's representation flow rather than treating it as auxiliary input.
Topology-Injected Patent Encoder: A novel architecture that combines a deterministic heterogeneous graph construction pipeline with a connectivity mask and learnable relation-aware biases to integrate claim-level dependencies into self-attention.
Empirical Finding on Structural Inductive Bias: Experiments demonstrate that intra-patent claim topology provides a stronger inductive bias than inter-patent citation structures. Furthermore, the learned structural priors are internalized into model weights, allowing the model to perform effectively at inference without requiring graph construction.

Experimental Results

PHAGE was evaluated on three tasks: Patent Classification, Retrieval, and Clustering, using datasets CLEF-IP, USPTO-70k, and DAPFAM. The evaluation protocol used a frozen encoder, where only the relation-aware biases and contrastive weights were optimized, ensuring fair comparison against baselines.

Classification: PHAGE outperformed all baselines, including domain-adapted models (BERT-for-Patents) and citation-aware models (PaECTER, PatEmbed). On CLEF-IP, it achieved a Macro-F1 of 0.635, a significant improvement of +9.7 over BERT-for-Patents. Notably, Graph Neural Network (GNN) baselines applied post-hoc to embeddings degraded performance, suggesting that injecting structure directly into the encoder is superior to post-processing.
Retrieval: On the DAPFAM benchmark, PHAGE achieved an NDCG@100 of 0.498, outperforming the best citation-aware baseline (QaECTER, 0.379) by 11.9 points and BERT-for-Patents by 27.0 points.
Clustering: PHAGE achieved the highest NMI (0.653) and Homogeneity (0.704) on CLEF-IP, indicating that claim-level topology sharpens intra-cluster purity.

Ablation Studies confirmed that:

The structural injection (CDG + attention) is the primary driver of performance gains.
The hard connectivity mask is more critical than the soft relation bias, though both contribute.
Learnable relation weights provide consistent gains, with the model learning to up-weight implicit relations in lower layers and amplify intra-claim biases in upper layers.

Significance and Claims

The paper claims that the structural bottleneck in patent representation learning lies within patents (intra-document claim topology) rather than between them (inter-patent citation networks).

Key findings highlighted by the authors include:

Superiority of Intra-Document Structure: Despite using only within-patent dependencies, PHAGE surpasses citation-aware baselines that leverage millions of examiner-verified cross-patent links. This suggests that the explicit inheritance and narrowing of scope within a single patent provide a more potent signal for representation learning.
Internalization of Structural Bias: The model successfully internalizes the structural inductive bias during training. At inference, PHAGE operates as a standard forward pass without graph construction, yet retains nearly all performance gains. This eliminates the computational overhead of graph construction during deployment.
Depth-wise Specialization: The learned biases induce specialization across Transformer layers, with lower layers focusing on cross-claim aggregation and upper layers refining within-claim representations.

The authors conclude that PHAGE offers a scalable solution for patent analysis by internalizing complex legal and technical dependencies into the model parameters, removing the need for external graph structures during inference while significantly outperforming existing domain-adapted and citation-aware approaches.