🤖 machine learning

When Structure Doesn't Help: LLMs Do Not Read Text-Attributed Graphs as Effectively as We Expected

This paper challenges the conventional assumption that structural information is essential for graph reasoning by demonstrating that large language models often achieve strong performance on text-attributed graphs using only node textual descriptions, while most explicit structural encoding strategies provide marginal or even negative gains.

Original authors: Haotian Xu, Yuning You, Tengfei Ma

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haotian Xu, Yuning You, Tengfei Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant, well-read librarian (the Large Language Model, or LLM) how to understand a complex map of connections, like a social network, a citation list of research papers, or even a molecule.

For years, the standard advice has been: "To understand the map, you must show the librarian the roads connecting the places." In the world of computer science, these "roads" are called graph structures. Researchers have spent years building complex tools (like GNNs) to highlight these roads, believing that without them, the librarian would be lost.

This paper, titled "When Structure Doesn't Help," is a reality check. The authors ran a series of experiments and discovered something surprising: The librarian doesn't actually need the roads drawn on the map to do a great job.

Here is the breakdown of their findings using simple analogies:

1. The "Unordered List" Discovery

The Old Way: Imagine describing a party to the librarian. You say, "Alice is standing next to Bob, who is talking to Charlie." You are giving the librarian the structure of the room.
The New Finding: The authors found that if you just give the librarian a list of the guests and what they are wearing (the text descriptions), the librarian can figure out who knows whom just by reading the descriptions.

The Analogy: If you tell the librarian, "Alice is wearing a red hat and loves jazz," and "Bob is wearing a red hat and loves jazz," the librarian can guess they are friends without you ever saying, "Alice is standing next to Bob."
The Result: When the authors removed the "road maps" (structural data) and just gave the librarian the text descriptions of the nodes, the librarian performed just as well, and sometimes even better, than when the roads were included.

2. The "Over-Engineered GPS" Problem

The Old Way: Researchers tried to feed the librarian complex, pre-drawn maps (using tools called GNNs or Laplacian embeddings) to help them navigate.
The New Finding: These complex maps often confused the librarian or made them slower. It was like giving a human a GPS that kept recalculating the route every second; it was distracting.

The Analogy: Imagine trying to read a book while someone keeps shouting directions like "Turn left at the letter 'A'!" The librarian found that ignoring the shouting and just reading the story (the text) was more effective.
The Result: Adding structural "priors" (pre-made rules about how things connect) often didn't help. In fact, on some tricky graphs (where friends don't necessarily look alike, called heterophilic graphs), the extra structure actually made the librarian's performance worse.

3. Does a Bigger Brain Change Anything?

The Question: Maybe the librarian just wasn't smart enough to read the maps? What if we use a super-smart librarian (a larger model with more parameters)?
The Finding: No. Even when they used a much larger, more powerful model (scaling up from 7 billion to 13 billion parameters), the result was the same. The bigger librarian still preferred the text descriptions and largely ignored the structural maps.

The Analogy: Giving a genius a complicated, confusing blueprint doesn't make them understand the building better if they can just read the description of the bricks.

4. What About Real-World Maps (Molecules)?

The Question: What about things where the "shape" is the most important part, like a molecule in chemistry? Surely, the shape matters there?
The Finding: Even for molecules, the librarian did surprisingly well just by reading the list of atoms and their descriptions, without needing a 3D model of how they are connected.

The Analogy: If you describe a Lego castle by listing every brick's color and type, a smart person can often guess the shape of the castle without you showing them the blueprint. The "semantic" description (the text) was enough to solve the puzzle.

5. The "Reasoning" Test

The Question: What about models specifically trained to be "reasoning experts"? Do they finally use the maps?
The Finding: Even models designed to solve logic puzzles and follow structured rules didn't suddenly start caring about the graph structure. They still relied on the text.

The Analogy: Even a detective trained to follow footprints (structure) decided that reading the suspect's diary (text) was a better way to solve the case.

The Bottom Line

The paper concludes that for Text-Attributed Graphs (graphs where the nodes have rich text descriptions), we have been over-complicating things.

The Old Belief: "We must build complex structures to help the AI understand the graph."
The New Reality: "The AI is so good at reading text that it can infer the connections itself. Adding complex structural maps is often unnecessary, and sometimes, it's just noise that gets in the way."

The authors suggest that instead of building fancy structural encoders, we should focus on how we order the text we feed the model. If you present the information in a logical sequence, the model can do the rest. It's less about drawing the map and more about telling the story clearly.

1. Problem Statement

The prevailing paradigm in graph learning assumes that integrating structural information (topology, connectivity, message passing) is essential for reasoning, especially when combined with Large Language Models (LLMs). Recent efforts to create "Graph Foundation Models" often rely on hybrid architectures (LLM + GNN) or complex prompt engineering to inject structural priors (e.g., Laplacian embeddings, neighborhood aggregation) into LLMs.

The Core Question: Is explicit structural encoding actually necessary for LLMs to perform well on Text-Attributed Graphs (TAGs)? The authors challenge the foundational assumption that LLMs inherently benefit from or effectively utilize graph structure, suggesting that current benchmarks and methodologies may overestimate the utility of structural signals in the era of powerful language models.

2. Methodology

The authors conducted a systematic empirical study across diverse graph types, encoding strategies, and model architectures to isolate the impact of structural information.

Datasets:
- Text-Attributed Graphs (TAGs): Six datasets including citation networks (Cora, Citeseer, Pubmed), web pages (School), Wikipedia (Roman Empire), and e-commerce (Amazon Ratings). These cover both homophilic and heterophilic patterns.
- Molecular Graphs: Three datasets from MoleculeNet (BACE, BBBP, HIV) representing intrinsic structural data without rich text attributes.
- Large-Scale Graphs: Products and ArXiv datasets to test scalability.
- Geometric Deep Learning: Davis Drug-Target Interaction (DTI) dataset.
Experimental Paradigms:
1. Template-Based Encoding: Compared the standard ND (Neighborhood Detail) template (which uses handcrafted subgraphs and Laplacian positional encodings) against structure-agnostic variants:
  - HN (Hop Neighbor): Randomly sampled $k$ -hop neighbors.
  - CO (Center Only): Only the central node description.
2. GNN-Based Encoding: Evaluated hybrid frameworks (e.g., GraphToken) where GNNs (GCN, GAT, GIN) act as adapters to encode structure. These were compared against simple MLP adapters that lack message-passing mechanisms.
3. Ablation Studies:
  - Model Scaling: Tested LLaMA2-7B vs. LLaMA2-13B.
  - Semantic Density: Compared full node descriptions (abstracts) vs. sparse descriptions (titles).
  - Pretrained Encoders: Compared Graph-specific encoders (GraphMVP) vs. Language encoders (TinyBERT) for molecular tasks.
  - Reasoning Models: Tested Large Reasoning Models (OpenReasoning-Nemotron-7B).
Tasks: Node Classification, Link Prediction, Molecular Property Prediction, and Drug-Target Interaction.

3. Key Contributions

Empirical Challenge to Structural Necessity: The paper provides robust evidence that for TAGs, explicit structural encoding often yields marginal or negative performance gains compared to relying solely on node textual semantics.
LLM Behavior Characterization: The authors demonstrate that LLMs tend to treat input graphs as unordered sets of text rather than topological structures. They rely heavily on the semantic content of node descriptions to infer relationships, effectively "reading" the graph as a collection of documents.
Identification of Harmful Effects: In heterophilic graphs and deep GNN configurations, adding structural signals can lead to oversmoothing and performance degradation, as the structural noise interferes with the rich semantic signals.
Re-evaluation of Benchmarks: The study suggests that many current graph benchmarks may not truly test "relational reasoning" but rather semantic retrieval, calling for a shift toward semantics-driven or geometry-aware tasks.

4. Key Results

Template-Based Findings (TAGs):
- The CO (Center Only) and HN (Hop Neighbor) templates often outperformed the structure-rich ND template.
- On heterophilic graphs (e.g., School, Roman Empire), structural encodings significantly degraded performance, while the "Center Only" approach remained competitive.
- Conclusion: LLMs can infer connectivity from semantic context alone; explicit topology is often redundant.
GNN-Based Findings:
- MLP vs. GNN: Simple MLP adapters (no message passing) performed comparably to or better than GNN-based adapters (GCN, GAT, GIN) in node classification tasks.
- Depth Sensitivity: Increasing the depth of GNN-based adapters led to significant performance drops (oversmoothing), whereas MLPs remained stable.
- Conclusion: The inductive bias of GNNs does not translate effectively to LLMs when node semantics are rich.
Molecular & Geometric Tasks:
- Even in domains where structure is intrinsic (molecules), TinyBERT (language encoder) embeddings performed on par with or better than GraphMVP (graph encoder) embeddings when fed into an LLM.
- In the Davis DTI task, sequence-based (semantic) representations outperformed structure-based encodings.
Robustness Checks:
- Scaling: Increasing model size (7B $\to$ 13B) did not improve the model's ability to utilize structural signals.
- Semantic Sparsity: Even with sparse node descriptions (titles only), structure-free methods outperformed structure-aware ones.
- Large Reasoning Models: Specialized reasoning models (Nemotron) also failed to leverage structural encodings effectively, relying instead on semantic cues.

5. Significance and Future Directions

Paradigm Shift: The paper argues for a fundamental shift in graph learning for the LLM era. Instead of forcing LLMs to learn structural inductive biases (via GNNs or complex templates), researchers should prioritize high-quality semantic representations and optimal node sequencing.
Efficiency: Removing complex structural encoding modules (GNNs, Laplacian calculations) simplifies pipelines, reduces computational overhead, and avoids the risk of oversmoothing.
Future Work: The authors suggest exploring graph transformations (like Graph Diffusion Convolution) that condense information into sparse, center-focused subsets, which might better align with LLM capabilities. They also call for new benchmarks that genuinely test relational reasoning beyond semantic retrieval.

Final Takeaway: In the context of Text-Attributed Graphs, "Structure Doesn't Help." LLMs are powerful enough to reconstruct relational dependencies from semantic content alone, making explicit structural priors often unnecessary and sometimes counterproductive.