💬 NLP

Argument Quality Assessment with Large Language Models: A Pairwise Bradley-Terry Approach

This paper evaluates the ability of 12 large language models to assess argument quality across logical, rhetorical, and dialectic dimensions by using pairwise comparisons within a Bradley-Terry framework, finding that while models like Llama-70B show moderate alignment with human experts, they offer a stable and partially complementary approach to automated argument evaluation.

Original authors: Nicolás Benjamín Ocampo, Agnes Paullate Nyiranziza, Davide Ceolin

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nicolás Benjamín Ocampo, Agnes Paullate Nyiranziza, Davide Ceolin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hosting a massive debate tournament. You have thousands of arguments on topics like politics, science, and ethics. To decide who wins, you need to know which arguments are the strongest. Traditionally, you'd hire a panel of expert judges to read every single argument and rank them. But that takes forever, costs a fortune, and different judges might disagree with each other.

This paper asks a simple question: Can we replace those expensive human judges with Large Language Models (LLMs)—the same kind of AI that powers chatbots?

Here is the breakdown of their experiment, using some everyday analogies:

The Setup: The "Taste Test" Approach

Instead of asking the AI to give every argument a score from 1 to 10 (which is hard and inconsistent), the researchers used a "Taste Test" method.

The Method: They showed the AI two arguments at a time (Argument A vs. Argument B) and asked, "Which one is better?"
The Dimensions: They didn't just ask "which is better?" generally. They asked the AI to judge based on three specific "flavors" of quality:
1. Logical: Does the math and reasoning add up? (Is the recipe actually edible?)
2. Rhetorical: Is it persuasive and well-written? (Is the food presented beautifully?)
3. Dialectical: Does it handle counter-arguments well? (Does it stand up to a food critic?)
The Scoreboard: Once the AI made thousands of these "A vs. B" choices, the researchers used a statistical formula (called the Bradley-Terry model) to turn those pairwise choices into a final leaderboard, just like how sports leagues calculate rankings based on win/loss records.

The Experiment: The "Taste Test" Panel

The researchers didn't just test one AI. They gathered a panel of 12 different AI models of varying sizes (from small "compact" models to massive "super-computer" models) from five different families (like Llama, Mistral, Qwen, etc.).

They tested these AIs in three different "modes":

Zero-Shot: "Here are two arguments. Pick the winner." (No hints).
Few-Shot: "Here are two arguments. Also, here are three examples of how a human judge picked winners in the past. Now pick the winner." (Learning by example).
Chain-of-Thought: "Think step-by-step about the logic, the style, and the counter-arguments before you pick a winner." (Reasoning out loud).

The Results: Who Won the Tournament?

The researchers compared the AI's final leaderboard against a "Gold Standard" leaderboard created by actual human experts.

The Star Player: The Llama-70B model (a very large AI) using the "Few-Shot" method (learning by example) was the clear winner. It didn't perfectly match the humans, but it was the closest. Think of it as a student who got a B+ on a test where the human experts got an A. It wasn't perfect, but it was surprisingly good.
The "Good Enough" Players: Other large models (like Qwen-72B) also did well, showing that big brains generally understand the task better.
The Surprises: Some smaller models did okay, but some medium-sized models actually performed worse than their smaller siblings. It wasn't just about "bigger is always better."
The Consistency Check: The researchers ran the tests three times to see if the AI would change its mind. The results were very stable. The AI only changed its "winner" choice in less than 8% of cases. This is like a judge who, if asked the same question three times, gives the same answer almost every time.

The Key Takeaways

AI is a Promising Assistant, Not a Replacement: The AI models can mimic human experts to a "moderate" degree. They aren't perfect judges yet, but they are good enough to help scale up the process.
Examples Matter: Giving the AI a few examples of how to judge (Few-Shot) helped the biggest models perform better, acting like a "cheat sheet" that clarified the rules.
Different Models See Different Things: Some models agreed with the human experts, while others agreed with the best AI model but not the humans. This suggests that different AIs might have slightly different "perspectives" on what makes an argument good, and mixing them together might be even better than relying on just one.

The Caveats (The "Fine Print")

The authors are careful to point out that this isn't a magic bullet yet:

The Dataset: The arguments tested were short paragraphs from online debates. We don't know if these AIs would work as well on long, complex essays or legal briefs.
The "Gold Standard" Flaw: The human experts who created the "correct" answers might have had their own biases or disagreements. If the humans were wrong or inconsistent, the AI might just be copying those inconsistencies.
The "Black Box" Problem: Sometimes an AI might pick an argument because it sounds confident, even if the logic is flawed. It's hard to know exactly why the AI made a choice without digging deep into its code.

In summary: This paper shows that AI can be a helpful "assistant judge" for sorting through arguments. It's not quite ready to replace the human jury, but it's a powerful tool that can help us evaluate thousands of arguments quickly and consistently.

Technical Summary: Argument Quality Assessment with Large Language Models: A Pairwise Bradley-Terry Approach

Problem Statement

The automatic assessment of argument quality is critical for effective persuasion, decision-making, and discourse analysis in fields ranging from law to science. However, manually evaluating argument quality at scale remains a bottleneck due to high costs and annotation inconsistency. While Large Language Models (LLMs) have demonstrated capabilities in reasoning and judgment, it remains an open question whether they can reliably replicate expert-level judgments in argument quality assessment. Existing research offers mixed results regarding LLMs' ability to assess specific quality aspects or rank arguments, often relying on limited model sets or single-dimensional evaluations (e.g., convincingness) rather than multidimensional frameworks.

This study addresses the research question: How well can LLM pairwise comparisons reproduce expert rankings of argument quality? It specifically investigates the performance of LLMs across different sizes, families, and prompting strategies against human expert annotations.

Methodology

The study utilizes the Webis-ArgQuality-20 corpus, a gold-standard dataset comprising 1,610 texts across 20 debate topics. The dataset includes 13,952 pairwise comparisons annotated by experts (via crowdsourcing) across three distinct dimensions of argument quality defined by Wachsmuth et al. (2017b):

Logical Quality: Focuses on cogency, including local acceptability, relevance, and sufficiency of premises.
Rhetorical Quality: Measures effectiveness and persuasion, covering credibility, emotional appeal, clarity, appropriateness, and arrangement.
Dialectical Quality: Assesses reasonableness and the ability to resolve disagreement, including acceptability, relevance, and sufficiency (rebuttal).

Experimental Setup

The authors evaluated 12 open-weight LLMs from five model families (Mistral, Llama 3, Qwen2.5, Olmo2, and Command-R), spanning parameter scales from 7B to 104B. The evaluation followed a strict pairwise comparison protocol where models were instructed to select the better argument (A, B, or Tie) for a specific quality dimension.

Three prompting strategies were employed:

Zero-shot: Inputting the dimension and the argument pair.
Chain-of-Thought (CoT): Instructing the model to reason internally over sub-dimensions (e.g., evaluating local relevance before determining logical quality) before outputting the final label.
Few-shot: Providing three annotated examples per topic (one for A, one for B, one for Tie) selected based on ground-truth score differences to ensure clarity.

Aggregation and Metrics

To derive continuous quality scores from discrete pairwise judgments, the study applied the Bradley-Terry (BT) model. This statistical approach estimates the probability that one argument is preferred over another ( $P(d_i \succ d_j) = \frac{\gamma_i}{\gamma_i + \gamma_j}$ ) to generate latent strength scores and a final ranking.

Performance was evaluated using:

Correlation Metrics: Pearson, Spearman, and Kendall coefficients comparing LLM-derived BT rankings against expert-derived BT rankings.
Error Metrics: Mean Absolute Error (MAE) and Root Mean Square Error (RMSE).
Agreement Metrics: Cohen's $\kappa$ and percent agreement for direct pairwise label comparison.
Stability: Krippendorff's $\alpha$ and variability analysis across three independent runs to measure prediction consistency.

Key Results

1. Model Performance vs. Human Experts

The study found that LLMs achieve moderate correlation with human expert judgments, with no model reaching perfect alignment.

Best Performer: Llama-70B with few-shot prompting achieved the highest alignment. It reached a Cohen's $\kappa$ of 0.493 and correlations with expert BT scores of 0.473 (Pearson), 0.477 (Spearman), and 0.327 (Kendall).
Runner-up: Qwen2.5-72B (zero-shot) followed closely with correlations of 0.463–0.466.
Model Size Trends: Generally, larger models (70B+) outperformed smaller variants (7B–32B) in correlation with experts. However, the Mistral family was an exception, where the 7B variant outperformed the 22B variant in correlation, though the 22B variant showed higher inter-annotator agreement (IAA).
Prompting Strategies: There was no universal preference for a single strategy. Few-shot prompting significantly benefited large models (Llama-70B, Command-R-104B) but was often detrimental or neutral for medium-sized models. Chain-of-thought did not consistently improve performance and sometimes reduced correlation scores compared to zero-shot.

2. Cross-LLM Alignment

When comparing other models against the best-performing configuration (Llama-70B with few-shot):

Qwen2.5-72B and Olmo2-32B showed strong alignment with Llama-70B.
Qwen2.5-7B (the small variant) surprisingly aligned well with its larger counterpart and the best configuration.
Mistral-22B showed near-zero correlation with Llama-70B, suggesting a divergent understanding of the quality dimensions despite both models achieving moderate agreement with human experts. This indicates that different models may capture complementary perspectives on argument quality.

3. Stability and Variability

LLM predictions were generally stable across three independent trial runs.

Small Models: Exhibited high stability (often 100% agreement across runs) but lower accuracy relative to experts.
Large Models: Showed slightly higher variability but benefited from few-shot prompting, which reduced the occurrence of "all unequal" (conflicting) labels.
Worst Case: Mistral-22B with few-shot prompting yielded the highest variability, with 7.75% of cases resulting in different labels across all three runs.
Conclusion: Aggregation strategies like majority voting are effective for handling the remaining variability.

Significance and Contributions

The paper claims three primary contributions:

Systematic Evaluation: It provides the first extensive evaluation of 12 open-weight LLMs across five families and three prompting strategies for multidimensional argument quality assessment.
Bradley-Terry Application: It is, to the authors' knowledge, the first application of the Bradley-Terry model to aggregate LLM-based comparative judgments in argument mining, moving beyond simple Likert-scale evaluations.
Insights on Model Behavior: The study reveals that while LLMs do not yet match expert-level performance, they offer a "promising but moderate" approximation. Crucially, it highlights that different models exhibit partial but complementary understanding of quality dimensions. This suggests that ensembling diverse models could potentially outperform reliance on a single best-performing model.

The authors maintain a modest stance, acknowledging that LLMs are not yet a replacement for human experts but serve as a scalable tool for approximating expert rankings. They emphasize that the moderate correlation (up to $\kappa \approx 0.49$ ) highlights both the potential and current limitations of LLMs as evaluators.

Limitations

The study notes several limitations regarding validity:

Internal Validity: Stability was tested at temperature 1; lower temperatures were not ablated. Prompt formulation differences across model families may influence interpretation.
External Validity: Results are based on short, online debate arguments and may not generalize to complex texts like essays. The ground truth is limited to the specific annotator pool of the Webis-ArgQuality-20 dataset. The study did not include fine-tuning, which might yield different performance characteristics.
Construct Validity: There is a risk that LLMs evaluate based on surface features (e.g., confident phrasing) rather than deep logical validity. The separation of dimensions in prompts may not fully prevent cross-dimensional contamination in model reasoning.
Conclusion Validity: The dataset size (13,952 comparisons) is smaller than some prior works, potentially limiting the statistical robustness of the Bradley-Terry rankings.