Technical Summary: Argument Representation and Coverage (ARC)

Problem Statement

While Large Language Models (LLMs) have achieved fluency in abstractive summarization, their reliability in high-stakes domains (such as law and science) is compromised by frequent hallucinations and the omission of key content. Existing evaluation metrics often fail to distinguish between fluency and factual completeness, particularly in long-document summarization where critical arguments are sparsely distributed. Current methods struggle to provide interpretable diagnostics regarding what specific information is missing or misrepresented, and whether LLMs inherently preserve salient argumentative structures without explicit supervision.

Methodology: The ARC Framework

The authors introduce Argument Representation Coverage (ARC), a bottom-up evaluation framework designed to assess how well summaries preserve salient arguments. ARC operates hierarchically through three levels of granularity:

1. Decomposition to Atomic Facts:

   • Salient argument roles (e.g., Issues, Reasons, Conclusions in legal texts; Own Claims, Background Claims, Data in scientific texts) are decomposed into sets of atomic facts ($F_r$) using a strong LLM (GPT-4o).
   • An entailment check filters out over-generated facts to ensure they are strictly derived from the original argument.

2. Fact-Level Evaluation ($\delta$):

   • Each atomic fact is compared against the generated summary using an LLM judge.
   • The judge assigns a binary status (supported or not) and categorizes the error type: Missing (omission) or Non-factual (contradiction/misrepresentation). This separation allows for diagnosing whether errors stem from poor content selection or generation inaccuracies.

3. Hierarchical Aggregation:

   • $ARC_{role}$: The coverage score for a specific argument role is the average of its atomic fact coverage.
   • $ARC_{score}$: The overall summary score is the average of all role-level scores, providing a holistic measure of argument preservation.

Datasets and Models:
The framework is applied to two domains:

• CANLII: 1,049 long legal opinions where argument roles (Issue, Reason, Conclusion) constitute only ~7.66% of the input text but ~66% of the reference summary, creating a "needle in a haystack" challenge.
• DRI (DR. INVENTOR): 40 scientific articles with dense argumentative content (74.14% of input).

Eight open-weight instruction-following LLMs (ranging from 1B to 14B parameters, including LLaMA, Mistral, and Qwen families) were evaluated on these datasets.

Key Results

1. Coverage Limitations and Error Types

• Incomplete Coverage: Even the best-performing models fail to achieve perfect argument coverage. In legal domains (CANLII), the best models plateau around an $ARC_{score}$ of 0.72, significantly below the human reference (0.98).
• Omission vs. Hallucination: Fact-level analysis reveals that missing facts are the dominant source of error across all models and domains, far exceeding factual errors (hallucinations). This suggests the primary challenge is retrieving and prioritizing sparse salient content rather than generating incorrect facts.
• Model Scaling: Larger models generally achieve higher coverage, but the gap between model performance and human references remains substantial, particularly in legal texts.

2. Positional Bias (RQ2)

• U-Shaped Bias: Consistent with prior work, models exhibit a U-shaped positional bias, favoring content at the beginning and end of the context window.
• Impact on Legal Summaries: In CANLII, where reference summaries do not follow fixed positional trends, this bias negatively correlates with coverage ($ARC_{score}$). Arguments located in the middle of long legal opinions are systematically underrepresented.
• Scientific Documents: In DRI, the correlation is weaker, as the reference summaries align more closely with the LLMs' natural positional biases.

3. Role-Specific Bias (RQ3)

• Systematic Preferences: LLMs exhibit systematic preferences for certain argument roles.
  • Legal (CANLII): Models disproportionately favor Conclusions over Issues and Reasons. The bias score ($\beta$) is consistently lowest for conclusions, indicating they are better preserved.
  • Scientific (DRI): Background Claims tend to be covered less effectively than Own Claims, even when controlling for argument length.
• Model Size Effect: Smaller models (<3B parameters) exhibit higher bias scores, indicating less balanced coverage across different argument roles.

4. Metric Validation

• Correlation with Experts: $ARC_{score}$ achieves the highest correlation with human expert judgments (Kendall's $\tau$ = 0.465, Pearson's $\rho$ = 0.593) compared to standard metrics (ROUGE, BERTScore) and decomposition-based baselines like FactScore.
• Interpretability: Unlike FactScore, which provides only fact-level coverage, ARC offers multi-level analysis (role-level and error-type differentiation), making it a superior diagnostic tool.
• Judge Robustness: The framework performs well with open-weight judges (e.g., DeepSeek-R1-Distill-Qwen-14B), which achieved correlations even higher than proprietary GPT-4o in some settings.

Significance and Claims

The paper claims that ARC provides a principled and interpretable diagnostic tool for evaluating structured argument coverage in long-context summarization. Its primary contributions are:

1. Granular Diagnostics: By distinguishing between omissions and factual errors, and by analyzing coverage at the role level, ARC reveals why summaries fail (e.g., missing middle-ground arguments or ignoring specific role types) rather than just that they fail.
2. Evidence of Systematic Bias: The study uncovers that current instruction-following LLMs have inherent biases toward specific argument roles (favoring conclusions) and document positions (favoring ends), which hinders their reliability in high-stakes domains where comprehensive argumentation is required.
3. Need for Alignment: The results underscore that fluency alone is insufficient. The persistent gap between model performance and human references, particularly in sparse legal contexts, indicates a need for further alignment strategies (e.g., explicit argument structure in training or prompting) to improve saliency coverage.

The authors conclude that while ARC improves the evaluation of coverage, it currently emphasizes recall (capturing salient facts) over precision. Future work should aim to jointly assess both dimensions and integrate salience detection directly into the summarization pipeline.