SCURank: Ranking Multiple Candidate Summaries with Summary Content Units for Enhanced Summarization

1. Problem Statement

The paper addresses two critical challenges in the field of text summarization, specifically within the context of model distillation (training smaller models like BART to mimic Large Language Models):

• Instability of LLM-based Ranking: Existing methods (e.g., GPTRank) rely on LLMs to directly compare and rank candidate summaries. Research indicates that LLMs suffer from positional bias and inconsistency when performing text comparisons, leading to unreliable supervision signals for contrastive learning.
• Limitations of Surface-Level Metrics: Traditional automatic metrics like ROUGE rely on n-gram overlap, which fails to capture semantic richness, abstractive quality, or the actual informational value of a summary.
• Single-Source Bias: Previous distillation approaches often rely on summaries generated by a single LLM, which limits the diversity of generation patterns and introduces model-specific biases.

2. Methodology: SCURank

The authors propose SCURank (Summary Content Unit Ranking), a framework designed to rank candidate summaries based on information richness and semantic importance rather than surface overlap or direct LLM comparison. The process involves three main stages:

A. SCU Extraction (Information Decomposition)

Instead of comparing full summaries, the framework decomposes each candidate summary into Summary Content Units (SCUs).

• Definition: An SCU is a simple, standalone, and unique piece of information (similar to a "Semantic GPT Unit" or SGU).
• Implementation: The authors use an LLM (specifically gpt-4o-mini) to extract SCUs from each summary. The prompt instructs the model to split sentences into distinct factual statements.

B. SCU Aggregation (Clustering)

To determine the importance of specific information, the framework aggregates SCUs across all candidate summaries generated for a single document.

• Embedding: SCUs are converted into vector representations using a sentence encoder (all-mpnet-base-v2).
• Clustering: The vectors are clustered using HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise).
  • HDBSCAN is chosen because it can automatically determine the number of clusters and handle varying densities without requiring a pre-defined cluster count.
  • Each cluster represents a distinct semantic concept found across the candidate summaries.
• Consensus Logic: The size of a cluster (the number of SCUs belonging to it) reflects the consensus among the diverse LLMs. A larger cluster implies that multiple models independently identified that piece of information as important.

C. Scoring and Ranking

Each summary is assigned a score based on the SCUs it contains:

1. Base Score: The score of an individual SCU is defined as the size of the cluster it belongs to ($\phi(u_{i,j}) = \|C_k\|$).
2. Summary Score: The total score for a summary is the sum of the scores of its constituent SCUs ($\Phi(s_i)$).
3. Length Penalty: To prevent longer summaries from receiving artificially high scores, the total score is normalized by the square root of the summary's token count:
   $$\Phi_{lp}(s_i) = \frac{\Phi(s_i)}{\sqrt{\|s_i\|}}$$
4. Ranking: Summaries are ranked based on these adjusted scores. These ranked summaries are then used as positive/negative pairs for contrastive learning in the BRIO framework.

3. Key Contributions

1. Novel Ranking Framework: Introduction of SCURank, which shifts the evaluation focus from direct LLM comparison to information-centric ranking via SCUs. This bypasses the instability of LLM-based pairwise comparisons.
2. Multi-LLM Distillation Strategy: The paper investigates and validates the use of multiple diverse LLMs (9 different models) to generate candidate summaries. This approach enhances the diversity of training data and improves the abstractive capabilities of the distilled model.
3. Integration with Contrastive Learning: The authors successfully integrate SCURank into the BRIO (BART with Ranking for Information Ordering) framework, replacing ROUGE-based ranking with SCURank to provide more stable supervision.
4. Comprehensive Evaluation: The study provides extensive empirical evidence across multiple datasets (CNN/DailyMail, XSum) and evaluation dimensions (automatic metrics, human evaluation, and LLM-based judges).

4. Experimental Results

The authors evaluated SCURank against baselines including ROUGE, BERTScore, BLANC, and GPTRank, using both single-LLM (BASE) and multi-LLM (LLMs-9) datasets.

• Performance on Automatic Metrics:
  • SCURank consistently outperformed GPTRank and traditional metrics (ROUGE, BERTScore) across ROUGE-1/2/L, BLEURT, and BERTScore on both CNN/DailyMail and XSum datasets.
  • On the XSum dataset (highly abstractive), SCURank achieved the best scores in ROUGE-1, ROUGE-2, and BLEURT.
• Stability Analysis:
  • GPTRank showed high sensitivity to input order (positional bias); its correlation dropped significantly when summary order was shuffled.
  • SCURank demonstrated high stability (Krippendorff's $\alpha$ = 0.846), remaining robust regardless of input order because it relies on clustering consensus rather than direct LLM ranking.
• Human Evaluation:
  • In pairwise comparisons via Amazon Mechanical Turk, summaries trained with SCURank were preferred over GPTRank in terms of Completeness, Conciseness, and Faithfulness.
  • SCURank showed statistically significant improvements ($p < 0.01$) in Completeness, indicating better retention of key information.
• Abstractive Quality:
  • Models distilled from multiple LLMs using SCURank showed lower Coverage and Density scores compared to single-LLM distillation, indicating a shift toward more abstractive (paraphrased) rather than extractive summaries, closer to human writing styles.
• Comparison with LLMs:
  • The distilled BART model trained with SCURank achieved performance comparable to, and in some metrics exceeding, the average of the top 9 LLMs used for generation.

5. Significance and Conclusion

The paper establishes that information-centric ranking is a superior alternative to direct LLM comparison for training summarization models. By leveraging the consensus of multiple LLMs through Summary Content Units, SCURank provides a stable, robust, and semantically meaningful supervision signal.

Key Takeaways:

• Stability: SCURank solves the positional bias and inconsistency issues inherent in LLM-based ranking.
• Diversity: Utilizing multiple LLMs for candidate generation significantly boosts the abstractive quality of the distilled model.
• Practicality: The framework allows small language models (SLMs) to achieve performance levels comparable to large LLMs in summarization tasks, making high-quality summarization more resource-efficient.

The code for SCURank is publicly available, facilitating further research into information-based evaluation and multi-LLM distillation.