ThreadSumm: Summarization of Nested Discourse Threads Using Tree of Thoughts

1. Problem Statement

The paper addresses the challenge of summarizing deeply nested discussion threads (e.g., from Reddit, StackOverflow, or specialized forums). Unlike linear documents, these threads present unique difficulties:

• Structural Complexity: Content is organized as a tree with interleaved replies, quotes, and sub-threads, making linear summarization insufficient.
• Topic Interleaving: Multiple topics often run in parallel, with off-topic replies burying main content.
• Viewpoint Overlap: In multi-party conversations, speakers interrupt or respond to specific nodes, making it difficult to infer target responses using simple linear adjacency or timestamp-based ordering.
• Limitations of Existing Methods: Traditional extractive/abstractive models often fail to capture the hierarchical structure, leading to summaries that lack logical flow, miss minority viewpoints, or suffer from positional bias (over-representing early posts).

2. Methodology: ThreadSumm

The authors propose ThreadSumm, a multi-stage Large Language Model (LLM) framework that treats thread summarization as a hierarchical reasoning problem. Instead of generating a summary in a single pass, it decomposes the task into structured modules and employs a Tree of Thoughts (ToT) search strategy to optimize for coherence and coverage.

The framework consists of the following stages:

A. Content Planning & Decomposition

1. Aspect Extraction: The LLM identifies unique "aspects" (Who, What, Where, specific entities, or topics) across the entire document collection. This ensures the summary covers diverse facets rather than just the most frequent topics.
2. Atomic Content Unit (ACU) Generation: For each aspect, the LLM extracts Atomic Content Units—self-contained semantic statements that cannot be broken down further without losing meaning. This creates a fine-grained, interpretable representation of the source information.
3. Sentence Ordering: The generated ACUs are reordered to form a coherent narrative flow. The authors use a zero-shot LLM prompt for this, arguing that simple heuristics (like chronological sorting) fail in nested threads where important content appears deep in reply chains.

B. Tree of Thoughts (ToT) Search

This is the core innovation. Instead of generating one summary, the system iteratively explores multiple paths:

• Proposal Generation: The LLM generates multiple candidate paragraphs based on the reordered ACUs.
• Evaluation: Each candidate is scored by an LLM based on two metrics:
  • Coherence: Logical flow and connectedness of ideas.
  • Coverage: The extent to which the summary includes important information from the source (specifically targeting aspect retention and opinion diversity).
• Iterative Refinement: The highest-scoring paragraph candidates are selected to guide the next iteration of the search, allowing the model to explore diverse compositional paths and converge on a high-quality summary.

3. Key Contributions

• ThreadSumm Framework: A novel multi-stage pipeline specifically tailored for nested discourse, moving beyond standard linear summarization.
• Explicit Reasoning over Structure: The integration of Aspect Extraction and ACUs provides an interpretable intermediate layer that ensures balanced coverage of topics.
• ToT Integration: The application of Tree of Thoughts to thread summarization allows for the joint optimization of coherence and coverage through iterative refinement and multi-proposal search.
• Metadata Agnosticism: The method does not rely on metadata (like timestamps or thread IDs) for ordering, making it robust for real-world scenarios where such data may be missing or unreliable.

4. Experimental Setup & Results

Datasets:

• Reddit: Community discussion forums (complex, branching structures).
• StackOverflow: Community Q&A (multi-party, argumentative).
• Bitcoin Forum: A case study with deep hierarchical trees (visualized in the paper).

Baselines:

• Vanilla: Direct LLM prompting.
• arg-graph: Argument mining with graph construction.
• CHRONOS: Retrieval-based timeline summarization.
• mRedditSumm: Clustering-based summarization.

Evaluation Metrics:

• QAGS: Factual consistency (detects hallucinations).
• SummaC: NLI-based factual consistency.
• ROUGE-1: Lexical overlap (Recall).
• Custom Metrics: Aspect retention, opinion cluster coverage, and positional bias analysis.
• Human Evaluation: Ratings for Coherence and Coverage (1-5 scale).

Key Results:

• Performance: ThreadSumm consistently outperformed all baselines across three different LLMs (GPT-4, Claude, LLaMA 3) and all datasets.
  • On QAGS (factual consistency), ThreadSumm showed gains of +10 to +20 points over baselines.
  • On ROUGE-1, it achieved significantly higher recall, indicating better coverage of source content.
• Aspect & Opinion Coverage: ThreadSumm achieved the highest aspect overlap scores (e.g., 0.80 on Bitcoin with Claude vs. 0.60 for baselines) and better preserved opinion diversity, ensuring minority viewpoints were not lost.
• Positional Bias: Analysis showed ThreadSumm did not favor early-position sentences; omitted sentences were scattered throughout the thread and were semantically less significant.
• Ablation Study: Removing Tree of Thoughts caused the most significant drop in performance (especially in ROUGE and SummaC), confirming it as the most critical component. Removing Aspect Extraction or Sentence Reordering also degraded factual consistency and coherence.

5. Significance and Conclusion

The paper demonstrates that treating thread summarization as a hierarchical reasoning problem rather than a simple text reduction task yields superior results.

• Robustness: The method generalizes well across different domains and LLM sizes (even performing competitively with open-source LLaMA 3).
• Quality: By explicitly optimizing for coherence and coverage via ToT, the system produces summaries that are logically structured and comprehensive, avoiding the "distorted view" caused by positional bias.
• Future Work: The authors plan to extend the ToT approach to optimize other quality dimensions (e.g., length control, cost-efficiency) and apply the framework to multimodal data.

In summary, ThreadSumm represents a significant step forward in handling the complexity of real-world online discussions, providing a reliable mechanism to synthesize multi-voice, nested conversations into coherent, factually consistent, and inclusive summaries.