DIAL-SUMMER: A Structured Evaluation Framework of Hierarchical Errors in Dialogue Summaries

Technical Summary: DIAL-SUMMER

DIAL-SUMMER: A Structured Evaluation Framework of Hierarchical Errors in Dialogue Summaries presents a novel framework designed to address the unique complexities of evaluating dialogue summarization. Unlike news summarization, dialogues involve multi-speaker interactions, scattered information across turns, and a shift in narration viewpoint (from first/second person to third person).

1. The Problem

Existing evaluation frameworks for dialogue summarization often fail to capture the structural and perspectival nuances specific to conversational data. Current metrics and taxonomies frequently suffer from:

• Structural Neglect: They do not account for the hierarchical nature of dialogues (speaker/turn level vs. within-turn semantic level).
• Perspective Shifts: They overlook errors arising when a summary transitions from the speakers' direct narration to a third-person narrator (e.g., misrepresenting a speaker's opinion as an objective fact).
• Taxonomic Ambiguity: Prior error categories (like "contradiction") often overlap with others (like "entity error"), making evaluation inconsistent and uninformative.

2. Methodology

The authors propose the DIAL-SUMMER framework, which is built upon two pillars: a hierarchical taxonomy and a human-annotated dataset.

A. Hierarchical Error Taxonomy:
The framework categorizes errors into two distinct levels to mirror the structure of a conversation:

• Dialogue-Level (Macro): Focuses on the structural integrity of the conversation. Categories include Wrong Turn Sequence, Missed Turn, Speaker Misattribution, Speaker Identity Bias, and Viewpoint Distortion (where the summary presents a speaker's subjective statement as an objective fact).
• Within-Turn-Level (Micro): Focuses on semantic discrepancies within specific utterances. Categories include Wrong Linking, Changed Meaning, Extrinsic Conversation, Extrinsic Context, and Missed Conversation.

B. Dataset Creation:
The authors curated a dataset of 192 multi-turn dialogues sourced from the ANTHROPIC-TEST dataset. Summaries were generated using LLAMA-3.3-70B-INSTRUCT. A team of expert human annotators performed fine-grained labeling, identifying specific error types, the exact sentences where they occurred, and providing qualitative reasoning for each annotation.

3. Key Contributions

• A Novel Taxonomy: A comprehensive, mutually exclusive, and hierarchical error taxonomy specifically tailored for the nuances of dialogue.
• A High-Quality Dataset: A human-annotated inference dataset that provides ground-truth labels for both sentence-level and summary-level errors, facilitating the training and testing of error-detection models.
• Empirical Insights: A detailed analysis of how and where errors typically manifest in LLM-generated summaries.

4. Results and Empirical Analysis

A. Error Trends:

• Positional Bias: The study found that Viewpoint Distortion tends to occur at the beginning of summaries, while Extrinsic Hallucinations (Extrinsic Conversation/Context) frequently occur at the end. Missed Turns are most common in the middle of the dialogue.
• Frequency: Incompleteness is a major issue; approximately 73% of summaries exhibited Missed Conversation errors (a very harsh metric), and 29.7% exhibited Missed Turn errors.

B. LLM-as-a-Judge Performance:
The authors tested several LLMs (GPT-5, Claude-Sonnet-4.5, Llama-3.1-70B, and Qwen2.5-32B) to see if they could act as automated judges.

• Taxonomy Impact: Providing the DIAL-SUMMER taxonomy in the prompt significantly improved the LLMs' ability to detect hallucinations compared to vanilla or prior taxonomies (like TofuEval).
• Capability Gap: Despite improvements, LLM performance remained "moderately good" (60-70% accuracy), highlighting that current LLMs still struggle with the fine-grained nuances of dialogue faithfulness.
• Model Variance: GPT-5 and Claude-Sonnet-4.5 generally outperformed open-weight models, which showed near-random performance in some categories.

5. Significance

DIAL-SUMMER moves the field of dialogue summarization evaluation from coarse, binary "is it faithful?" metrics toward a more explainable and informative paradigm. By providing a structured way to identify why a summary failed (e.g., was it a sequence error or a viewpoint error?), the framework enables researchers to develop more targeted error-correction methods and more robust summarization models. It serves as a benchmark for the next generation of "LLM-Judges" in the conversational AI domain.