Text Summarization With Graph Attention Networks

1. Problem Statement

Automatic Text Summarization (ATS), particularly abstractive summarization, remains a challenging task in Natural Language Processing (NLP). While Transformer-based models (e.g., BART, T5) have achieved state-of-the-art results, they often struggle with:

• Long-range dependencies: Managing relationships across lengthy documents.
• Factual consistency: Generating summaries that adhere to the facts of the source text.
• Lack of Graph Data: There is a scarcity of large-scale datasets annotated with discourse structures (like Rhetorical Structure Theory - RST) and coreference relations, which are crucial for understanding document coherence.

The authors aim to investigate whether integrating Graph Neural Networks (GNNs), specifically Graph Attention Networks (GATs), with discourse graphs (RST and Coreference) can improve the performance of baseline summarization models. They also aim to address the data gap by creating a graph-annotated version of the XSum dataset.

2. Methodology

The authors propose a two-stage hybrid summarization model consisting of an Encoder (for extractive selection) and a Decoder (for abstractive generation).

A. Dataset and Preprocessing

• Primary Dataset: CNN/DailyMail (CNN/DM).
• Secondary Dataset: XSum. The authors annotated XSum with RST and Coreference (Coref) graphs, creating a new benchmark for graph-based summarization.
• Elementary Discourse Units (EDUs): Documents are segmented into EDUs. The model's goal is to classify EDUs as "important" or "trivial."

B. The Proposed Architecture

1. Encoder (Extraction Stage):

   • Input: EDUs are processed using pre-trained language models (BERT or Longformer).
   • Span Representation: A SelfAttentiveSpanExtractor aggregates token embeddings within an EDU into a single vector.
   • Graph Integration: The model attempts to incorporate graph information (RST relations and Coref edges) into the EDU representations.
     • Experiment 1 (GAT): The authors initially used Graph Attention Networks (GATs) to process the graph structure, following previous works like DiscoBERT.
     • Experiment 2 (MLP): Finding GATs ineffective, they replaced the GNN with a Multi-Layer Perceptron (MLP). The graph information was converted into a one-hot vector (representing RST relation types) and a scalar (Coref share percentage) and fed into the MLP alongside the EDU embeddings.
   • Classification: A classifier predicts the importance of each EDU. The authors utilized MSE loss (instead of Binary Cross Entropy) and class weighting to handle the severe class imbalance (where ~90% of EDUs are unimportant).

2. Decoder (Abstraction Stage):

   • Selected EDUs are concatenated and fed into a BART model (BART-base for CNN/DM, BART-large for XSum) to generate the final abstractive summary.

C. Training Strategy

• Fine-tuning: A two-phase approach was used: first freezing the pre-trained language model weights while training the encoder, then unfreezing the entire network.
• Optimization: Extensive hyperparameter tuning, including learning rate schedulers, dropout, and loss function adjustments, was critical for convergence.

3. Key Contributions

1. Graph-Annotated XSum Dataset: The authors created and released a version of the XSum dataset annotated with RST and Coreference graphs, addressing a significant gap in available resources for graph-based summarization research.
2. Empirical Analysis of GNNs vs. MLPs: The study challenges the prevailing assumption that GNNs (specifically GATs) are the optimal method for integrating graph information into NLP tasks. They demonstrated that a simple MLP processing graph features as one-hot vectors outperformed complex GAT architectures in their setting.
3. Hybrid Model Performance: They developed a robust hybrid model that surpasses existing baselines on the CNN/DM dataset.
4. Comprehensive Benchmarking: The paper provides detailed comparisons across extractive, abstractive, and hybrid models, including an analysis of "Oracle" limits (the theoretical maximum performance if the perfect EDUs were selected).

4. Results

The experiments were conducted on CNN/DM and XSum datasets, evaluated using ROUGE metrics (R-1, R-2, R-L).

• Graph Integration Efficacy:
  • GAT Failure: Using GATs to incorporate RST/Coref graphs did not improve performance over the "without graph" baseline. In some cases, it slightly degraded results.
  • MLP Success: Replacing GATs with a simple MLP to process graph features improved the F1 selection score and final ROUGE scores.
    • CNN/DM (BERT + MLP): Achieved 40.15 F1 (selection) and 43.42 R-1 (final summary), outperforming the baseline without graphs.
    • CNN/DM (Longformer + MLP): Achieved 38.65 F1 (selection) and 43.73 R-1 (final summary), setting a new state-of-the-art for their specific hybrid configuration.
• Dataset Differences:
  • The graph-based improvements were more pronounced on CNN/DM (longer, more extractive documents) than on XSum (highly abstractive, shorter summaries).
  • On XSum, the "Oracle" EDU selection yielded lower ROUGE scores compared to CNN/DM, suggesting that the extraction-abstraction pipeline has inherent limitations for highly abstractive datasets.
• Baseline Comparisons: The proposed "Longformer with MLP for RST & Coref" model outperformed previous baselines including DiscoBERT, BERTSUM, and standard BART on the CNN/DM test set.

5. Significance and Future Directions

• Paradigm Shift in Graph Integration: The paper suggests that for text summarization, complex GNN architectures may be overkill or misaligned with how modern Transformers (like Longformer) already handle attention. Simple feature engineering (one-hot encoding of graph types) combined with MLPs can be more effective and computationally efficient.
• Resource Creation: The release of the annotated XSum dataset provides a crucial benchmark for future research into discourse-aware summarization.
• Future Work: The authors propose moving toward end-to-end models that jointly perform extraction and generation to avoid gradient disconnection issues. They also suggest using semantic-aware metrics (like BERTScore) over syntax-based ones (ROUGE) and generating Coref graphs for more datasets.

In conclusion, the study demonstrates that while graph information is valuable for summarization, the method of incorporation is critical. A lightweight MLP approach proved superior to complex GATs in this context, and the newly created dataset facilitates further exploration of discourse structures in NLP.