ArgBench: Benchmarking LLMs on Computational Argumentation Tasks

1. Problem Statement

Large Language Models (LLMs) possess significant potential for computational argumentation tasks, such as countering hate speech, detecting misinformation, and facilitating collaborative debate. However, existing benchmarks for LLMs (e.g., MMLU, ARC) largely focus on tasks with single correct answers (mathematics, common sense) or overlook the complexity of argumentation, which involves conflicting perspectives, divergent reasoning paths, and open-ended generation.

Current argumentation-specific benchmarks are fragmented, often focusing on single tasks (e.g., only argument mining) or specific genres, and rarely evaluate LLMs' ability to generalize across a broad spectrum of argumentation skills. There is a lack of a unified, standardized framework to assess LLMs on the full lifecycle of argumentation: mining, assessing, reasoning, and generating arguments.

2. Methodology

The ArgBench Benchmark

The authors introduce ArgBench, the first unified benchmark designed to standardize the evaluation of LLMs on computational argumentation.

• Scope: It encompasses 33 datasets and 46 distinct tasks derived from pre-LLM era research, unified into a consistent prompt format.
• Skill Taxonomy: The 46 tasks are categorized into five core argumentation skills:
  1. Argument Mining: Extracting arguments, premises, conclusions, and relations from text (e.g., unit segmentation, relation detection).
  2. Argument Perspective Assessment: Identifying stances, frames, key points, and argument similarity.
  3. Argument Quality Assessment: Ranking arguments, rating quality, detecting suboptimal claims, and scoring controversy/reasonableness.
  4. Argument Reasoning: Detecting fallacies (e.g., ad-hominem), identifying argumentation schemes, and warrant identification.
  5. Argument Generation: Generating arguments, counterarguments, warrants, and optimizing claims.
• Task Types: Tasks are modeled as Classification, Segmentation, or Generation, with inputs and outputs standardized as prompts.

Evaluation Setups

The benchmark employs two distinct evaluation protocols to assess different capabilities:

1. Prompting (Zero-shot, Few-shot, Chain-of-Thought): Evaluates the model's "out-of-the-box" capabilities without fine-tuning.
   • Zero-shot: Task definition + instance.
   • Few-shot: Task definition + 4 examples (one per label on average).
   • Chain-of-Thought (CoT): Task definition + instruction to "Think step by step" before outputting.
2. Leave-One-Task-Out (LOTO): Evaluates generalization and transfer learning.
   • Models are fine-tuned on the training sets of 45 tasks and evaluated on the 46th (unseen) task.
   • This setup tests whether learning one argumentation skill transfers to another.

Experimental Setup

• Models: The study evaluates 11 LLMs across 5 families (Mistral, Qwen3, Llama-3.1/3.3, DeepSeek-R1, Phi-3.5) ranging from 4B to 70B parameters, plus the closed-source GPT-4.1.
• Metrics:
  • Classification/Segmentation: Macro F1-score.
  • Generation: BertScore (semantic similarity) for automated evaluation, supplemented by manual human evaluation for counterargument generation (assessing argumentativeness, countering, and relevance).
• Fine-tuning: Used LoRA (Low-Rank Adaptation) with 4-bit precision.

3. Key Contributions

1. Unified Benchmark: Creation of ArgBench, covering 46 tasks across 5 skills and 33 datasets, providing the most comprehensive evaluation protocol for computational argumentation to date.
2. Systematic Analysis: A quantitative analysis of how model size, prompting techniques (Zero/Few-shot/CoT), and training strategies affect performance across diverse argumentation tasks.
3. Generalization Insights: Empirical evidence regarding the limits of LLMs in transferring skills between different argumentation tasks (e.g., mining vs. generation).
4. Error Analysis: Identification of specific failure modes, such as label bias (preferring positive/high labels), difficulties with emotional language in reasoning tasks, and incomplete reasoning chains in CoT.

4. Key Results

Prompting Performance

• Model Size: Larger models generally outperform smaller ones, particularly in Argument Mining, Perspective Assessment, and Reasoning.
• Prompting Techniques:
  • Few-shot prompting is generally the most effective for large models (32B+), especially for reasoning and generation.
  • Chain-of-Thought (CoT) improves performance on Argument Quality Assessment for most models but surprisingly underperforms or yields mixed results on Argument Reasoning tasks for Llama, Mistral, and Qwen3 families.
• Skill Difficulty: Argument Quality Assessment is identified as the most challenging skill for LLMs, showing the lowest performance gains from scaling or prompting. Models struggle with subjective quality ratings and ranking.

Leave-One-Task-Out (Generalization)

• Limited Transfer: LLMs show limited ability to generalize from one task to another. Performance on unseen tasks is often significantly lower than in-task fine-tuning.
• Skill Transfer Patterns:
  • Argument Mining and Perspective Assessment tasks serve as effective training grounds for other tasks (e.g., mining helps with counterargument generation).
  • Quality Assessment and Reasoning tasks show better cross-task generalization than generation or extraction tasks.
  • Models like Mistral-22b and DeepSeek-R1-32b show the best generalization, particularly in ad-hominem detection and argument rating, but still struggle with unit segmentation and similarity tasks when trained on other skills.

Manual Evaluation

• In the counterargument generation task, task-specific fine-tuning yielded significantly higher quality than training on other tasks.
• Without specific training, models were better at generating opposing stances (countering) than at grounding those stances in well-reasoned arguments (argumentativeness).

5. Significance and Limitations

Significance:

• ArgBench fills a critical gap in NLP research by providing a standardized, multi-skill benchmark for argumentation, moving beyond single-task evaluations.
• It highlights that while LLMs are improving, they still lack robust reasoning capabilities for complex, subjective argumentation tasks, particularly regarding quality assessment and fallacy detection.
• The findings suggest that "one-size-fits-all" fine-tuning is insufficient; optimal performance requires task-specific training or carefully selected transfer skills.

Limitations:

• Unified Modeling: Some tasks required simplification (e.g., reducing label sets or reframing inputs) to fit a unified prompt format, potentially altering original task nuances.
• Evaluation Metrics: Automated metrics (BertScore) for generation tasks are imperfect; the study relied on a limited manual evaluation subset for generation quality.
• Model Coverage: Due to computational constraints, the study did not include models larger than 70B parameters or all available model families.
• Data Restrictions: Some existing datasets were excluded due to licensing or specific application constraints.

In conclusion, ArgBench establishes a rigorous foundation for advancing LLMs in computational argumentation, revealing that while scaling and prompting help, significant challenges remain in subjective quality assessment and cross-task generalization.