💬 NLP

LLMs Underperform Graph-Based Parsers on Supervised Relation Extraction for Complex Graphs

This paper demonstrates that while large language models are promising for relation extraction, smaller graph-based parsers significantly outperform them on complex linguistic graphs with high numbers of relations, making the lighter parsers a superior choice for such tasks.

Original authors: Paolo Gajo, Domenic Rosati, Hassan Sajjad, Alberto Barrón-Cedeño

Published 2026-04-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Paolo Gajo, Domenic Rosati, Hassan Sajjad, Alberto Barrón-Cedeño

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Big Brain" vs. The "Specialized Tool"

Imagine you have two workers trying to solve a puzzle:

The Big Brain (LLM): This is a massive, highly educated expert who knows everything about the world. They can write poetry, code software, and tell jokes. They are huge, expensive to hire, and take a long time to think.
The Specialized Tool (Graph-Based Parser): This is a small, lightweight, laser-focused mechanic. They only know how to fix one specific type of engine, but they are incredibly fast and efficient at it.

The Paper's Discovery:
The researchers wanted to see who was better at a specific task: Relation Extraction. In plain English, this means reading a sentence and figuring out how the different parts connect to each other (like finding out who works for which company, or which drug causes which side effect).

They found a surprising twist: When the puzzle is simple, the Big Brain is great. But when the puzzle gets complicated and messy, the Specialized Tool wins every time.

The Analogy: The "Web of Connections"

Think of a sentence as a map of a city.

Simple City (Small Graph): A small town with just a few streets and intersections.
Complex City (Large Graph): A massive metropolis like New York or Tokyo, with hundreds of streets, bridges, and traffic loops all tangled together.

1. The Big Brain's Struggle (The "Over-Thinker")

When the Big Brain (the LLM) tries to map the Complex City, it tries to read the whole map out loud, word by word, like a tour guide.

The Problem: To get from Point A to Point Z in a tangled city, the Big Brain has to read the entire description of the city again and again. It gets confused by the sheer volume of text. It starts to lose track of which street connects to which building because the "distance" between the relevant words in its memory gets too long.
The Result: It gets tired, makes mistakes, and takes hours to draw a map that is full of errors.

2. The Specialized Tool's Success (The "Direct Connector")

The Specialized Tool (the Graph-Based Parser) doesn't read the city like a story. Instead, it looks at the map as a structure.

The Advantage: It sees the connections instantly. It doesn't care about the "story" of the city; it just sees the lines connecting the dots. Because it is built specifically to handle these tangled webs, it can process a massive, complex city in seconds without getting confused.
The Result: It draws a perfect map almost instantly, even when the city is huge.

The Experiment: Testing on Different "Cities"

The researchers tested both workers on six different datasets (different types of text):

Small Towns (Simple Data): Datasets like CoNLL04 (short news snippets) or ADE (drug reports). These have very few connections.
- Result: The Big Brain did a great job here. It was fast enough and smart enough to handle the simple layout.
Mega-Cities (Complex Data): Datasets like ERFGC (cooking recipes with flow charts) or SciDTB (scientific abstracts). These have dozens or even hundreds of connections per sentence.
- Result: The Big Brain started to crumble. As the number of connections grew, its performance dropped sharply. The Specialized Tool, however, kept getting better and better.

The "Tipping Point":
The researchers found a specific threshold. When a text has more than about 18 connections (edges) between words, the Specialized Tool becomes significantly better than the Big Brain. The more complex the text, the wider the gap becomes.

Why Does This Happen? (The "Noise" Factor)

Imagine the Big Brain is trying to find a specific friend in a crowded stadium.

The Specialized Tool walks straight to the friend's seat.
The Big Brain has to shout the friend's name, wait for an echo, read a script, and then shout again. Every time it adds a new instruction (like "format this as a JSON list" or "explain the context"), it adds noise.

In complex texts, this "noise" and the extra steps the Big Brain takes to format its answer create a lot of interference. It dilutes its attention. The Specialized Tool doesn't have this baggage; it just does the math directly.

The Takeaway for the Real World

Don't use a sledgehammer to crack a nut: If you have a simple task, a giant AI model is fine. But if you are dealing with complex documents (like legal contracts, scientific papers, or detailed recipes) where you need to extract hundreds of relationships, using a massive LLM is inefficient and less accurate.
Size isn't everything: Just because a model has billions of parameters (is "big") doesn't mean it's better at everything. Sometimes, a smaller, specialized tool is superior.
Efficiency matters: The Specialized Tool is thousands of times cheaper and faster to run. If you are building a system to process millions of documents, using the Big Brain for complex tasks is like hiring a team of 100 PhDs to do a job a single intern with a specific tool could do in half the time.

In short: For simple tasks, the AI superstar shines. But for complex, tangled webs of information, the humble, specialized mechanic is the true champion.

1. Problem Statement

Relation Extraction (RE) is a fundamental NLP task aimed at extracting structured RDF triples (head, relation, dependent) from unstructured text to build knowledge graphs. While Large Language Models (LLMs) have gained popularity for RE due to their success in in-context learning (ICL) and supervised fine-tuning, their performance relative to specialized, smaller architectures remains unclear, particularly regarding graph complexity.

The core problem addressed is whether LLMs can effectively handle RE tasks where the underlying linguistic graph (the network of entities and relations within a text) is large and complex. Previous literature has not directly compared supervised LLMs against graph-based parsers in a controlled setting to determine if model size correlates with performance on complex dependency structures.

2. Methodology

Datasets

The authors evaluated models on six datasets with varying graph complexities (measured by the number of relations $k$ per document):

Low Complexity: CoNLL04, ADE, SciERC (Most samples have $k \le 5$ ).
High Complexity: enEWT (English Web Treebank), SciDTB (Scientific Discourse TreeBank), and ERFGC (Culinary Recipe Flow Graphs). These contain graphs with significantly more nodes and edges, with ERFGC averaging ~49 relations per document.

Models Compared

Graph-Based Parser: A lightweight, supervised architecture based on Dozat and Manning's (2017) biaffine attention parser, extended with GNN layers (Bhatt et al., 2024).
- Size: ~124M parameters (110M frozen BERT encoder + ~14M trainable parser).
- Mechanism: Embeds tokens, processes them via recurrent/graph neural networks, and uses biaffine attention to score edges and relations simultaneously.
Large Language Models (LLMs): Four models ranging from 7B to 70B parameters:
- Mistral-7B-Instruct-v0.3
- Qwen3-14B-Base (non-instruction tuned) and Qwen3-32B
- Llama-3.3-70B-Instruct
- Training: All LLMs were fine-tuned using LoRA (Low-Rank Adaptation) targeting only attention layers (Key, Query, Value).
- Prompting: Evaluated with various prompt strategies: No Description (NoDesc), Description (Desc), UUID (random code to trigger task recognition without semantic instruction), and Adversarial prompts.

Evaluation Protocol

Metric: Micro-F1 score using exact evaluation (triples are correct only if head, tail, and relation match exactly).
Baselines: LLMs were tested zero-shot (no fine-tuning) and after fine-tuning (1 epoch and 3k steps).
Fairness: To ensure computational fairness, LLMs were trained for 1 epoch (matching the parser's effective data passes) and tested on subsets for larger models due to resource constraints.

3. Key Results

Performance vs. Graph Complexity

Simple Graphs: On datasets with few relations (CoNLL04, ADE, SciERC), LLMs (particularly Qwen3-14B-Base) matched or slightly outperformed the graph-based parser.
Complex Graphs: As the number of relations increased (enEWT, SciDTB, ERFGC), the performance gap widened significantly in favor of the graph-based parser.
- On ERFGC (most complex), the graph-based parser achieved an F1 of 0.713, while the best LLM (Qwen3-32B) only reached 0.581 (a gap of ~13.2 F1 points).
- The graph-based parser consistently outperformed LLMs on texts with an average of >18 edges.

Correlation Analysis

Negative Correlation for LLMs: There is a strong negative Pearson correlation between the number of relations ( $k$ ) and LLM performance. As graph complexity increases, LLM F1 scores drop sharply.
Stability for Parsers: The graph-based parser showed a much lower correlation between graph size and performance, maintaining robustness even on large graphs.

Parameter Efficiency

Despite the graph-based parser having two orders of magnitude fewer parameters (124M vs. 32B/70B), it outperformed the massive LLMs on complex tasks. This highlights that parameter count does not equate to structural reasoning capability in this specific domain.

Prompting and Fine-Tuning

Instruction Irrelevance: In supervised settings, prompt variations (descriptions, UUIDs, adversarial prompts) had minimal impact on final performance once fine-tuned. Even adversarial prompts (asking the model not to perform the task) were ignored by fine-tuned models, indicating the task is learned via weights rather than prompt adherence.
Zero-Shot Failure: Without fine-tuning, LLMs performed near-randomly on complex datasets, regardless of in-context examples.

4. Key Contributions

Empirical Evidence of LLM Limitations: The study provides the first direct comparison showing that supervised LLMs degrade significantly as linguistic graph complexity increases, whereas specialized graph parsers scale effectively.
Efficiency Benchmark: Demonstrates that a 124M parameter graph parser is superior to 7B–70B parameter LLMs for RE on complex texts, offering a more computationally efficient solution.
Analysis of Failure Modes: Identifies that LLMs struggle with complex graphs likely due to attention dilution. The requirement to format outputs as text (JSON/strings) injects noise into causal attention mechanisms and increases the token distance between relevant entities, a problem that scales with graph size.
Dataset Utilization: Leverages a diverse set of six datasets, including the underutilized ERFGC, to rigorously test the limits of graph complexity.

5. Significance and Conclusion

The paper challenges the prevailing assumption that "bigger is better" for structured extraction tasks. It concludes that for supervised Relation Extraction involving complex linguistic graphs, specialized graph-based parsers are the superior choice over LLMs.

Practical Implication: For applications requiring high-fidelity knowledge graph construction from dense, complex texts (e.g., scientific literature, procedural recipes), deploying massive LLMs may be inefficient and less accurate than lightweight, structured parsers.
Future Directions: The authors suggest that future work should investigate attention-level mechanisms to understand why LLMs fail on long-range dependencies in graphs and explore methods like prompt compression or reducing output formatting constraints to mitigate these issues.

In summary, while LLMs are powerful for general language tasks, they currently lack the structural inductive bias required to efficiently parse complex relational graphs compared to dedicated, smaller architectures.