💬 NLP

DuConTE: Dual-Granularity Text Encoder with Topology-Constrained Attention for Text-attributed Graphs

The paper proposes DuConTE, a dual-granularity text encoder that integrates graph topology into the attention mechanisms of cascaded language models to capture both word-level and node-level semantic dependencies for text-attributed graphs, achieving state-of-the-art performance.

Original authors: Lexuan Liang, Tao Zou, Xuxiang Ta, Zekun Qiu

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lexuan Liang, Tao Zou, Xuxiang Ta, Zekun Qiu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a massive, chaotic library where every book is connected to other books by invisible strings. Some books are about cooking, some about space, and some about history. But here's the twist: the books don't just have text inside them; they also have "tags" written on the outside that describe what's inside.

In the world of computers, this is called a Text-Attributed Graph. It's a mix of a social network (the connections) and a library catalog (the text).

The problem is that existing computer programs are bad at reading this library. They usually do one of two things:

They read the text of a single book very carefully but ignore the other books it's connected to.
They look at the connections between books but don't really understand the words inside them.

DuConTE is a new, super-smart librarian (an AI model) designed to solve this. Here is how it works, broken down into simple concepts:

1. The "Double-Decker" Reading Strategy (Dual-Granularity)

Most librarians read a book word-by-word. That's good, but they miss the big picture. DuConTE reads in two layers:

Layer 1: The Micro-Reader (Word-Token Level): Imagine a magnifying glass. DuConTE first looks at the individual words in a book. It understands that "apple" and "fruit" go together. It reads the text of the target book and the books connected to it, mixing them all together to get a detailed, word-by-word understanding.
Layer 2: The Macro-Reader (Node Level): Now, imagine stepping back to look at the whole shelf. DuConTE takes the "gist" of that first reading and treats the entire book (and its neighbors) as a single unit. It asks, "What is the main theme of this cluster of books?"

By doing both, it captures the tiny details and the big picture, just like reading a sentence and then summarizing the whole paragraph.

2. The "VIP Rope" System (Topology-Constrained Attention)

In a normal library, if you ask a question, the librarian might shout it to everyone in the building, even people in the "Cooking" section when you are looking for "Space" books. This creates noise and confusion.

DuConTE uses a special Rope System (Topology-Constrained Attention):

It puts a rope around the books that are physically connected.
When the AI reads a book, it is only allowed to "talk" to the books connected by that rope.
Why? Because in these libraries, connected books usually talk about similar things (a concept called homophily). If Book A is connected to Book B, they are likely in the same genre.
This stops the AI from getting distracted by irrelevant books and forces it to focus on the most relevant neighbors.

3. The "Smart Highlighter" (Node Representation Composer)

Once the AI has read all the words, it needs to write a summary card for the book. A bad librarian might just highlight every word equally.

DuConTE uses a Smart Highlighter that asks two different questions for every word:

"Does this word make sense for this specific book?" (The Center-Node Context)
"Does this word make sense because of the neighbors?" (The Neighborhood Context)

Example: If you are reading a book about "Java" (the programming language), the word "coffee" might be confusing.
- If you only look at the book, "coffee" is irrelevant.
- But if the neighbors are all about "Java" (the island) and "coffee," the AI realizes, "Ah, in this specific neighborhood, 'coffee' is actually important context!"
- DuConTE weighs these two perspectives to decide exactly how important a word is, creating a much more accurate summary.

The Result

When the researchers tested this new librarian (DuConTE) on famous datasets (like academic citation networks and product catalogs), it became the top performer. It was better at guessing what category a book belonged to than any other method, even those using massive, expensive super-computers.

In short: DuConTE is a librarian that reads both the fine print and the big picture, only listens to the books that are actually friends with the one you're reading, and knows exactly which words matter most based on who is sitting at the table with them.

1. Problem Statement

Text-Attributed Graphs (TAGs) are graphs where nodes are associated with textual descriptions (e.g., academic papers with abstracts, social media posts). The goal is to learn high-quality node representations that effectively capture both the semantic information of the text and the topological structure of the graph for downstream tasks like node classification.

Limitations of Existing Approaches:

Granularity Mismatch: Most current methods using Language Models (LMs) perform semantic interaction only at the word-token granularity. They fail to capture meaningful interactions at the node granularity, where the text of different nodes should be treated as holistic units interacting across the graph.
Decoupled Encoding: Many approaches either ignore graph structure during the LM encoding phase or inject structural signals insufficiently. The LM and Graph Neural Network (GNN) often operate in a decoupled manner, hindering deep fusion of structural and semantic information.
Representation Composition: Existing methods lack effective mechanisms to compose node representations from word-token embeddings, often treating all tokens equally or failing to distinguish between the context of the center node and its neighbors.

2. Methodology: DuConTE

The authors propose DuConTE, a dual-granularity text encoder with a topology-constrained attention mechanism. It acts as a plug-and-play module in a standard TAG learning pipeline (Text Encoder $\to$ GNN).

A. Dual-Granularity Cascaded Architecture

DuConTE employs two pretrained Language Models (LMs) in a sequential, cascaded manner:

Word-Token Encoder ( $M_L$ ): Encodes the combined text of a target node and its sampled neighbors. It captures semantics at the word-token granularity.
Node Encoder ( $M_N$ ): Takes the aggregated node representations from the first stage and encodes them at the node granularity, modeling interactions between nodes as holistic units.
Fusion: The final node representation is a weighted combination of the first-stage (word-token derived) and second-stage (node-derived) representations.

B. Topology-Constrained Attention Mechanism

To ensure the LMs are aware of the graph structure without architectural changes, DuConTE introduces a specific attention masking strategy based on the homophily assumption (connected nodes tend to share similar labels/semantics).

Word-Token Mask: In the first LM, attention is restricted. A token can attend to another token only if they belong to the same node or to connected nodes in the graph. [SEP] tokens can attend to any token to preserve boundary awareness.
Node Mask: In the second LM, node representations can only attend to other node representations if the corresponding nodes are identical or connected in the graph.
Effect: This forces the model to learn semantic correlations strictly informed by the graph topology, preventing noise from unrelated nodes.

C. Node Representation Composer

A specialized module aggregates word-token embeddings into node representations. It distinguishes between two contexts for the target node:

Center-Node Context: The importance of a token based on its relationship within the target node's own text.
Neighborhood Context: The importance of a token based on its relationship with the texts of neighboring nodes.

Mechanism: The composer uses a dual-perspective attention mechanism to calculate a weighted importance score ( $\mu$ ) for each token, fusing the center and neighborhood influences via a learnable coefficient $\beta$ . This allows the model to capture salient semantic information relevant to the specific node's role in the graph.

D. Two-Stage Training Procedure

Stage 1: Train the Word-Token Encoder ( $M_L$ ) and the Center-Node Composer ( $f_1$ ) to produce high-quality first-stage node representations.
Stage 2: Freeze Stage 1 components and train the Node Encoder ( $M_N$ ) and the Neighbor Composer ( $f_2$ ) to refine representations using the structural context.

3. Key Contributions

Dual-Granularity Encoding: A novel architecture that models semantics at both the word-token and node levels, addressing the limitation of existing methods that only operate at the token level.
Topology-Constrained Attention: A masking strategy that injects graph structural information directly into the self-attention mechanism of pretrained LMs, enabling structure-aware encoding without modifying the core Transformer architecture.
Context-Aware Representation Composer: A mechanism that explicitly differentiates and fuses the influence of the center node and its neighborhood when aggregating token embeddings, leading to more precise node representations.
State-of-the-Art Performance: The method achieves superior results on multiple benchmarks, demonstrating the effectiveness of deep semantic-structural fusion.

4. Experimental Results

The authors evaluated DuConTE on five benchmark datasets: Cora, CiteSeer, WikiCS, ArXiv-2023, and OGBN-Products.

Performance: DuConTE achieved State-of-the-Art (SOTA) results on the majority of datasets.
- On Cora, it reached 95.24% accuracy (vs. 93.60% for the previous best, GraphBridge).
- On ArXiv-2023, it reached 90.31% (vs. 86.43% for GraphBridge).
- On OGBN-Products, it reached 78.80%.
Ablation Studies:
- Removing the dual-granularity design (NoDual) caused a significant performance drop, confirming the necessity of node-level encoding.
- Removing topology constraints (NoMask) degraded performance, proving that structural guidance is critical even at the finest granularity.
- Using simple mean pooling instead of the context-aware composer resulted in lower accuracy, validating the importance of distinguishing center vs. neighborhood contexts.
Efficiency: Despite using two smaller LMs (RoBERTa-base), DuConTE outperformed baselines using larger models (RoBERTa-large), highlighting its parameter efficiency. It also showed strong performance in link prediction tasks.

5. Significance

This work bridges the gap between Language Models and Graph Neural Networks in a more principled way. By enforcing topology constraints within the attention mechanism, DuConTE ensures that the powerful pre-trained knowledge of LMs is effectively guided by the graph structure. The dual-granularity approach acknowledges that text-attributed graphs possess inherent multi-scale semantic structures (words and nodes), which previous methods failed to capture simultaneously. The results suggest that for text-rich graph tasks, the integration of structural constraints directly into the language modeling phase is a highly effective strategy, offering a new paradigm for scalable and high-performance TAG representation learning.