Technical Summary: StrucSum

Problem Statement

While Large Language Models (LLMs) have demonstrated strong zero-shot capabilities in general summarization, they face significant challenges when processing long, structured documents. Specifically, LLMs often struggle to model document structure, identify salient information across long contexts, and maintain factual consistency due to limited context windows and a lack of inherent discourse awareness. Existing extractive summarization methods often rely on supervised training with human-written gold summaries, limiting their scalability and adaptability across domains. Furthermore, while prompting strategies like Chain-of-Thought have enhanced reasoning in other areas, the integration of structural reasoning into prompts for long-document extractive summarization remains under-explored.

Methodology: StrucSum

The authors propose StrucSum, a training-free, structure-aware prompting framework designed to enhance LLM reasoning for zero-shot extractive summarization. The core mechanism involves constructing a Text-Attributed Graph (TAG) for each document, where sentences serve as nodes and edges represent semantic or positional relationships (derived via Sentence-BERT embeddings and cosine similarity).

StrucSum injects structural signals from this graph into the LLM prompts via three distinct strategies:

1. Neighbor-Aware Prompting (NAP):

   • Mechanism: Augments each sentence in the prompt by explicitly listing its 1-hop neighbors in the TAG.
   • Goal: Provides local contextual cues to enhance short-range discourse awareness, helping the LLM understand the immediate semantic dependencies of a sentence.

2. Centrality-Aware Prompting (CAP):

   • Mechanism: Appends a normalized degree centrality score to each sentence, calculated based on the number of connections within the TAG.
   • Goal: Highlights globally important sentences (discourse hubs) to guide the LLM in distinguishing structurally prominent content from peripheral information.

3. Centrality-Guided Masking (CGM):

   • Mechanism: Selectively includes only the top-ranked sentences (based on centrality scores) in the prompt until a cumulative centrality threshold (e.g., 85%) is met, while masking the remaining low-salience sentences.
   • Goal: Reduces input length and token usage while preserving key informational content, improving reasoning focus and efficiency.

The framework operates in a fully zero-shot setting without any parameter updates or fine-tuning of the underlying LLM.

Key Contributions

• Framework Proposal: Introduction of StrucSum, a novel prompting framework that integrates graph-based structural priors (NAP, CAP, CGM) into LLM inputs for extractive summarization.
• Empirical Validation: Demonstration across three long-form benchmarks (ArXiv, PubMed, Multi-News) that structure-aware prompting consistently outperforms both standard vanilla prompting and unsupervised baselines (e.g., Lead, LexRank, TextRank) in summary quality and factual consistency.
• Strategic Insights: Identification that different structural strategies serve distinct goals:
  • CGM is most effective for maximizing overlap with reference summaries (content coverage).
  • NAP is most effective for maintaining factual consistency and reducing hallucinations.
  • CAP offers a balance between performance and input length.
  • The study notes that combining all strategies does not necessarily yield further gains, suggesting they address different aspects of the task.

Experimental Results

Experiments were conducted on ArXiv, PubMed, and Multi-News using GPT-4o-mini, GPT-4o, and LLaMA3-70B.

• Summary Quality: StrucSum consistently improved ROUGE and BERTScore metrics compared to vanilla prompting. For instance, on ArXiv with GPT-4o-mini, the Centrality-Aware Prompting (CAP) strategy achieved an R-2 score of 15.39, outperforming the vanilla baseline (14.14).
• Factual Consistency: Significant improvements were observed in faithfulness metrics. On ArXiv, NAP increased FactCC by 19.2 percentage points (from 38.44 to 57.67) and SummaC by 8.0 points, demonstrating a stronger alignment between the generated summary and source content.
• Efficiency: CGM reduced token usage by 40–50% compared to vanilla prompting while maintaining or improving performance, making it a viable option for latency-constrained scenarios.
• Human Evaluation: Human raters on the PubMed dataset confirmed that StrucSum strategies, particularly NAP, improved fluency, informativeness, and faithfulness scores compared to vanilla prompts.
• Ablation & Analysis:
  • Combining multiple strategies (NAP + CAP + CGM) did not yield clear performance gains over individual strategies, indicating they target different optimization objectives.
  • Analysis of graph-only inputs (without sentence text) showed that while structural cues are helpful, they cannot replace textual information for optimal results.
  • Correlation analysis confirmed that LLMs follow centrality cues when explicitly provided (CAP), though this does not always translate to the highest overall ROUGE scores compared to NAP or CGM.

Significance and Claims

The paper claims that StrucSum represents a promising and under-explored direction for advancing zero-shot extractive summarization. By integrating simple, symbolic graph priors into prompts, the framework guides black-box LLMs to reason over document structure without requiring additional training or supervision. The authors emphasize that this approach addresses the specific limitations of LLMs in long-document contexts—namely, the lack of structural awareness and the tendency for factual inconsistency—providing a scalable, model-agnostic solution that enhances both the quality and reliability of machine-generated summaries. The work suggests that structural reasoning is a critical component for improving LLM performance in complex, long-form tasks.