💬 NLP

From Global to Local: Learning Context-Aware Graph Representations for Document Classification and Summarization

This paper proposes a data-driven method that leverages dynamic sliding-window attention to construct context-aware graph representations for documents, enabling Graph Attention Networks to achieve competitive performance in classification and summarization with reduced computational costs.

Original authors: Ruangrin Ldallitsakool, Margarita Bugueño, Gerard de Melo

Published 2026-03-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruangrin Ldallitsakool, Margarita Bugueño, Gerard de Melo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a massive, 500-page novel.

The Old Way (Traditional AI):
Most computer programs read books like a person walking down a long, narrow hallway. They look at one word, then the next, then the next. If the book is huge, the computer gets tired, forgets what it read at the beginning by the time it reaches the end, and gets confused by repetitive sentences. It's like trying to remember the plot of a movie while only seeing one frame at a time.

The New Way (This Paper's Idea):
The researchers in this paper say, "Why walk in a line? Let's build a map!"

They propose turning the document into a social network or a city map.

The Nodes (Cities): Every sentence in the document becomes a "city."
The Edges (Roads): Instead of just connecting city A to city B because they are next to each other, they draw roads between cities that talk to each other, even if they are far apart. If Sentence 1 mentions "The President" and Sentence 50 mentions "The White House," they build a direct road between them, even though they are far apart in the text.

The Problem with the Old "Map Makers"

Previously, experts tried to build these maps by hand. They used rules like, "If two sentences share a noun, build a road."

The Flaw: This is like a tourist trying to draw a map of a foreign city using only a phrasebook. It's rigid, often wrong, and doesn't work well if you move to a different city (a different topic or language).

The Innovation: The "Sliding Window" Tour Guide

The authors created a smart, data-driven way to build these maps automatically. Here is the secret sauce, explained simply:

1. The Sliding Window (The "Flashlight" Method)
Imagine you are in a dark room with a flashlight.

Old Method (Full Attention): You turn on a blinding floodlight that illuminates the entire room at once. It's powerful, but it's expensive (requires a lot of electricity/computer power) and the glare makes it hard to see the details of the furniture right in front of you.
New Method (Sliding Window): You use a focused flashlight. You shine it on a small group of sentences (a "window"), see who is talking to whom, then slide the light over to the next group.
- Why it's better: It's much cheaper to run (saves energy) and it pays extra attention to the immediate context, which is where most of the meaning lives.

2. The Pruning Shears (Statistical Filtering)
When the computer draws all these roads, it sometimes gets a little crazy and draws too many. It might connect every sentence to every other sentence, creating a tangled mess of spaghetti.

The researchers use "pruning shears" (statistical filters) to cut away the weak, noisy roads. They keep only the strongest, most important connections.
Analogy: It's like cleaning up a messy room. You throw away the junk (irrelevant connections) so you can actually see the furniture (the important ideas).

What Did They Find? (The Results)

1. Classification (Sorting the Mail)
They tested this on sorting news articles (e.g., "Is this about Sports or Politics?").

Result: Their "Sliding Window Map" worked just as well as the expensive, full-room floodlight methods, but it was much faster and cheaper. It proved that you don't need to see the whole document at once to understand it; you just need to see the right local neighborhoods well.

2. Summarization (Writing a Book Report)
They tried to use their maps to write summaries (picking the best sentences).

Result: This was a bit trickier. The maps were good at finding semantically similar sentences (ideas that match), but the computer struggled a bit with the specific "gold standard" summaries humans wrote.
The Insight: The computer was actually being too smart. It picked sentences that were thematically perfect but didn't use the exact keywords the human graders were looking for. It's like a student who understands the essay perfectly but uses different vocabulary than the teacher expects.

The Big Takeaway

This paper is a victory for efficiency.

It shows that we don't need massive, expensive computers to understand long documents. By using a "sliding window" to focus on local neighborhoods and then pruning the noise, we can build smart, lightweight maps of text that are almost as good as the heavyweights, but run on a fraction of the power.

In a nutshell: They taught computers to stop reading word-by-word in a straight line and start "hopping" between related ideas using a smart, energy-efficient flashlight, making them faster and better at understanding long stories.

1. Problem Statement

Current Natural Language Processing (NLP) systems typically treat documents as linear sequences of tokens. While effective for short texts, this approach struggles with:

Long-range dependencies: Linear models often fail to capture structural relationships and dependencies across long documents.
Redundancy and Inefficiency: Language models (LMs) frequently re-encode overlapping or identical content, leading to computational inefficiencies.
Heuristic Limitations: Existing graph-based approaches often rely on manually designed (heuristic) rules for constructing nodes and edges (e.g., co-occurrence, syntactic relations). These methods lack domain transferability, depend heavily on external preprocessing (like POS tagging), and may introduce cascading errors.
Computational Cost: Previous data-driven graph construction methods (e.g., Bugueño and de Melo, 2025) utilized full self-attention, which incurs quadratic computational complexity ( $O(N^2)$ ), making them prohibitively expensive for long documents.

The paper aims to develop an automatic, data-driven graph construction method that captures local and mid-range semantic dependencies efficiently, reducing computational overhead while maintaining or improving performance on document classification and summarization tasks.

2. Methodology

The proposed framework builds upon the data-driven graph learning approach of Bugueño and de Melo (2025) but introduces a critical architectural shift: replacing full attention with Sliding-Window Attention (SWA).

A. Graph Construction Pipeline

Sentence Embedding: Documents are encoded using a pre-trained Sentence Transformer (e.g., paraphrase-MiniLM-L6-v2), mapping each sentence $s_i$ to an embedding vector $x_i$ .
Sliding-Window Attention (SWA): Instead of attending to all sentences globally, the model restricts attention to a local neighborhood (a window) around each sentence. This mimics the mechanism used in models like Longformer.
- Window Size: The window size is defined as a percentage of the document length (e.g., 30% of sentences).
- Non-linear Transformations: The raw attention logits are passed through various activation functions (Softmax, ReLU, Sigmoid) and potentially an annealing temperature schedule to sharpen the attention distribution during training.
Statistical Filtering: To ensure sparsity and remove noise, the learned attention weights are filtered using row-wise thresholds:
- Mean-bound: Retains connections above the average attention score for a sentence.
- Max-bound: Retains only the strongest connections (above a threshold slightly below the maximum score).
- Unfiltered: Retains all soft dependencies (dense graphs).
Graph Definition: The document is represented as a graph $G=(V, E)$ where nodes are sentences. An undirected weighted edge exists between nodes $v_i$ and $v_j$ if the filtered attention weight exceeds a threshold. Identical embeddings are merged to reduce redundancy.

B. Downstream Task Modeling

The constructed graphs are fed into Graph Attention Networks (GATs) to perform downstream tasks. The GATs learn to aggregate information from neighboring sentences based on the learned edge weights.

3. Key Contributions

Efficient Graph Learning: The authors extend the graph structure learning framework by replacing full attention with Sliding-Window Attention. This significantly reduces computational complexity from quadratic to linear relative to document length while preserving local and mid-range context.
Systematic Evaluation: The method is rigorously evaluated on three document classification benchmarks (BBC, Hyperpartisan News Detection, and arXiv) and one extractive summarization task (GovReport).
Exploration of Summarization: The paper provides an exploratory analysis of the method's applicability to extractive summarization, highlighting both its potential and current limitations compared to classification.
Ablation Studies: The authors conduct extensive ablation studies on window sizes, activation functions, and filtering strategies to determine optimal configurations for different document lengths.

4. Experimental Results

Document Classification

Performance: The SWA-based graphs achieved competitive or superior results compared to previous full-attention graphs and heuristic baselines.
- On the BBC dataset, SWA graphs achieved near-perfect accuracy (100.0%) and F1 scores, outperforming full-attention baselines.
- On the HND dataset, SWA graphs with max-bound filtering achieved 95.5% accuracy, surpassing the full-attention baseline (92.6%).
- On the arXiv dataset (long documents), SWA graphs achieved 92.2% accuracy, outperforming the full-attention baseline (91.9%).
Efficiency: The use of SWA and statistical filtering resulted in sparser graphs (fewer edges) and significantly lower disk/memory usage compared to full-attention graphs. For instance, mean-bound SWA graphs had ~40% fewer edges than their full-attention counterparts.
Window Size Ablation:
- A 30% window size consistently yielded the best performance for GATs on the HND dataset.
- Interestingly, unfiltered (dense) graphs often performed best on medium-length documents, suggesting that GATs can implicitly suppress noise. However, for longer documents, max-bound filtering was crucial to prevent oversmoothing and maintain efficiency.

Document Summarization (Extractive)

Performance: The results were mixed. The data-driven graphs did not show significant gains over simple heuristic baselines (like sentence-order or fixed co-occurrence windows) on the GovReport dataset.
Analysis:
- The models tended to over-predict summary-worthy sentences (predicting ~20% of sentences vs. the oracle's ~6%), likely due to class imbalance and loss weighting.
- Evaluation against "oracle" summaries (optimized for lexical overlap) may have penalized models that captured semantic coherence but lacked exact keyword matches.
- The results suggest that for extractive summarization, simple positional heuristics remain highly effective, and the added complexity of learned semantic graphs offers limited benefit under current evaluation protocols.

5. Significance and Conclusion

Scalability: The primary significance of this work is demonstrating that local and mid-range context modeling is sufficient for high-performance document classification, eliminating the need for computationally expensive global attention mechanisms.
Data-Driven vs. Heuristic: The study reinforces that automatically learned graph structures can outperform manually engineered ones, offering better generalization without reliance on external NLP tools.
Trade-offs: The paper highlights a clear trade-off: while unfiltered dense graphs offer marginal performance gains, they incur higher resource costs. Statistical filtering (specifically max-bound for long documents) provides the best balance of efficiency and performance.
Future Directions: The authors note limitations in the summarization task, suggesting that future work should focus on evaluation metrics that better capture semantic coherence rather than just lexical overlap, and investigate how context scope should dynamically adapt to varying document lengths.

In summary, the paper successfully proposes a computationally efficient, data-driven framework for constructing document graphs that rivals or exceeds state-of-the-art methods in classification while being significantly more scalable for long documents.