💬 NLP

A comprehensive study of LLM-based argument classification: from Llama through DeepSeek to GPT-5.2

This paper presents a comprehensive evaluation of state-of-the-art large language models, including GPT-5.2, Llama 4, and DeepSeek, on argument classification tasks, demonstrating that advanced prompting strategies significantly boost performance while revealing systematic failure modes in handling complex argumentative structures.

Original authors: Marcin Pietroń, Filip Gampel, Jakub Gomułka, Andrzej Tomski, Rafał Olszowski

Published 2026-03-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Marcin Pietroń, Filip Gampel, Jakub Gomułka, Andrzej Tomski, Rafał Olszowski

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a group of very smart, but sometimes overly literal, robots how to understand human arguments. That's essentially what this paper is about.

The researchers wanted to see if the latest "super-brains" of Artificial Intelligence (called Large Language Models or LLMs) are good at Argument Mining. In plain English, this means: Can the AI read a text, figure out who is arguing for what, who is against it, and who is just saying something that doesn't really count as an argument?

Here is a breakdown of their study using simple analogies:

1. The Contestants: The AI "Olympians"

The researchers pitted several different AI models against each other. Think of them as athletes with different training backgrounds:

The Heavyweights (GPT-5.2, Llama 4, DeepSeek): These are the massive, expensive, or highly complex models. They are like Olympic gold medalists with years of training.
The Underdogs (Smaller Open Models): These are smaller, free-to-use models. The researchers wanted to see if a "local gym" athlete could beat the "Olympic" one.

2. The Arena: The Debate Datasets

To test the AIs, they used two giant libraries of real human debates (called UKP and Args.me).

UKP: Like a collection of heated comments on controversial topics (abortion, gun control, death penalty) from the internet.
Args.me: Like transcripts from formal debate clubs.
The Task: The AI had to read a sentence and decide: "Is this person For the idea? Against it? Or are they just Not Arguing at all?"

3. The Coaching Strategies (Prompting)

The researchers didn't just ask the AI, "What do you think?" They tried different ways of asking, like a coach trying different drills to get the best performance:

Chain-of-Thought: They told the AI, "Don't just give me the answer. Show your work first. Explain why you think this is an argument before you pick a side." It's like asking a student to show their math steps.
Rephrasing (mRAR): They asked the AI to rewrite the question in its own words before answering. This is like a lawyer restating a client's story to make sure they understand it before going to court.
The Voting Panel: Instead of asking one AI once, they asked the same AI four different times with slightly different questions. Then, they held a "vote" to see what the majority said. This is like asking four judges to score a gymnast and taking the average score to avoid one bad call.

4. The Results: Who Won?

The Champion: GPT-5.2 (the most advanced proprietary model) won the gold medal. It got about 78% accuracy on the internet comments and 92% on the formal debates.
The Surprise: A smaller, open-source model called gpt-oss-120b did almost as well as the champion! This is huge news because it means you don't need a billion-dollar supercomputer to do good argument analysis; a smaller, cheaper model can get you 95% of the way there.
The Power of the Panel: Using the "Voting Panel" strategy boosted everyone's scores by about 2% to 8%. It proved that asking the AI multiple times and averaging the answers makes it much smarter.

5. Where They Stumbled (The "Hallucinations")

Even the best AIs made mistakes, and the researchers found some funny (and frustrating) patterns:

The "Literal Robot" Problem: If a sentence had numbers or facts (e.g., "A study showed X"), the AI often thought, "Oh, this is just a fact, not an argument!" and ignored it. But humans know that facts are often used to support an argument.
The "But" Confusion: Humans use words like "but" or "however" to flip the meaning of a sentence. The AIs sometimes got confused by the first part of the sentence and forgot the main point came after the "but."
The "Topic Bias": If the topic was "Death Penalty," the AI sometimes assumed any comment was against it, just because that's what it saw in its training data. It was guessing based on the subject rather than reading the actual words.
The "Borderline" Problem: Sometimes, the humans who labeled the data were confused too! The researchers found that in about half the cases where the AI "failed," the human label was actually ambiguous. The AI might have been right, but the human label was wrong.

6. The Big Takeaway

This study tells us three main things:

AI is getting really good at understanding arguments, but it's not perfect yet. It still struggles with sarcasm, complex sentence structures, and subtle hints.
You don't need the biggest model to win. Smaller, smarter models, when coached well (using voting and rephrasing), can do almost as well as the expensive giants.
The data isn't perfect. Sometimes the "ground truth" (the correct answer) is fuzzy. We need better ways to teach AI how to handle the gray areas of human debate, not just the black-and-white facts.

In a nutshell: We taught a bunch of AIs to be debate judges. The smartest one won, but the smaller ones did surprisingly well when we gave them a team of judges to vote with. However, they still get tripped up by sarcasm, complex "but" sentences, and sometimes they just guess based on the topic. We need to keep teaching them how to read between the lines!

1. Problem Statement

Argument Mining (AM) is the task of automatically identifying and classifying argumentative components (claims, premises) and their relationships (support, attack, neutrality) within natural language text. While Large Language Models (LLMs) have shown promise in various NLP tasks, their application to structured argument classification remains under-explored.

The Gap: Existing benchmarks often focus on multiple-choice reasoning or code generation. Argument classification requires nuanced understanding of implicit premises, pragmatic context, and complex semantic relations (e.g., distinguishing between a rhetorical question and a direct claim).
The Challenge: There is a lack of systematic comparative evaluation of state-of-the-art LLMs (including open-weight and proprietary models) across multiple datasets. Furthermore, there is limited understanding of how these models fail (error patterns) and whether advanced prompting strategies (like Chain-of-Thought) effectively improve performance in this specific domain.

2. Methodology

2.1 Datasets

The study utilized two established, manually annotated corpora:

UKP Corpus: Contains ~25,000 instances from online comments on eight controversial topics (e.g., abortion, gun control). Labels: For, Against, No Argument.
Args.me Corpus: Contains ~48,000 arguments from debate portals. Labels: PRO (For) and CON (Against).

Data Processing: The authors created "full" sets and "trimmed" sets (2,000 samples each) to manage computational costs for proprietary models.

2.2 Models Evaluated

The study compared a wide spectrum of models:

Llama Family: Llama 3.1 (1B, 3B, 8B, 70B), Llama 3.3 (70B), and Llama 4 Scout (70B).
DeepSeek: DeepSeek R1 (70B).
GPT Family: GPT-5.2 (proprietary), GPT oss-20B, and GPT oss-120B (open-weight).

2.3 Prompting Strategies

The researchers tested several prompting techniques to optimize performance:

mRAR (Manual Rephrase and Respond): Four distinct prompt variations differing in response format (symbolic vs. verbal) and verbosity (concise vs. detailed).
Chain-of-Thought (CoT): Explicitly encouraging step-by-step reasoning. This was applied primarily to Llama models, as proprietary "reasoning models" (like GPT-5.2) have built-in reasoning capabilities.
Certainty Self-Assessment: Models were asked to assign a confidence score (0–100%) to their classification.

2.4 Ensemble Voting Algorithms

To aggregate predictions from multiple prompts ( $N=4$ ), three voting strategies were implemented:

Simple Vote: Majority vote (ignoring confidence).
Tiebreak Vote: Majority vote; ties are broken using the model's self-assessed confidence.
Weighted Vote: Sum of confidence scores for each class; the highest sum wins.

3. Key Contributions

First Comprehensive Benchmark: A systematic evaluation of modern LLMs (including GPT-5.2, Llama 4, and DeepSeek R1) on argument mining tasks across two major datasets.
Prompt Engineering Analysis: Demonstrated that prompt formulation (verbosity and format) significantly impacts performance, with natural language formats outperforming symbolic ones.
Ensemble Effectiveness: Showed that multi-prompt voting strategies (specifically Tiebreak and Simple voting) consistently improve Accuracy and F1 scores by 2% to 8% over single-shot prompting.
Qualitative Error Analysis: Provided a deep dive into failure modes, identifying that models struggle with:
- Contrastive discourse structures (e.g., "but," "however").
- Implicit criticism and rhetorical questions.
- Referential alignment (resolving pronouns like "that" or "this").
- Distinguishing between factual descriptions and argumentative stances.
Dataset Critique: Through a pilot re-annotation study, the authors found that 46.4% of model "errors" on the UKP abortion dataset were actually borderline cases where the original human annotation was ambiguous. In 43.9% of these borderline cases, the re-annotation aligned with the model's prediction, suggesting current benchmarks may underestimate model performance.

4. Key Results

4.1 Quantitative Performance

Top Performer: GPT-5.2 achieved the highest accuracy: 78.0% on UKP and 91.9% on Args.me.
Open-Weight Competitors: The GPT oss-120B model performed remarkably well, achieving 82.1% (UKP) and 89.6% (Args.me) with voting, nearly matching the proprietary GPT-5.2.
Model Size vs. Performance:
- Small models (e.g., Llama 1B) performed near random guessing.
- Interestingly, GPT oss-20B outperformed larger Llama 70B models, suggesting that architectural tuning and reasoning capabilities matter more than parameter count alone.
Impact of Voting: Voting algorithms provided statistically significant improvements. For example, on UKP, voting boosted GPT oss-120B from ~77.2% to 82.1%.

4.2 Error Analysis Findings

Systematic Bias: Models tend to associate specific topics with specific stances regardless of the text (e.g., misclassifying neutral statements about the death penalty as "Against").
Structural Failures: The most common error type was misinterpreting contrastive structures (e.g., failing to see that the clause after "but" is the main argument).
Pragmatic Inference: Models struggled with implicit intent, often treating rhetorical questions or counterfactuals as neutral facts.
Model Differences:
- GPT oss-120b: Struggled with literal interpretation of facts and implicit intent.
- GPT-5.2: Strong on pragmatic inference but vulnerable to complex structural reasoning.
- DeepSeek (ds70b): Intermediate performance but highly sensitive to emotional language.

5. Significance and Implications

Cost-Efficiency: The study proves that smaller, open-weight models (like GPT oss-120B), when combined with advanced prompting and voting strategies, can achieve performance parity with massive proprietary models. This makes high-quality argument mining accessible and scalable without reliance on closed APIs.
Benchmark Limitations: The findings suggest that current argument mining datasets contain significant annotation noise and ambiguity. Future benchmarks must account for "borderline" cases and explicitly test discourse-level reasoning (e.g., handling contrastive clauses and rhetorical devices) rather than just aggregate accuracy.
Practical Deployment: For real-world applications (legal tech, policy analysis), the study recommends using hybrid human-AI workflows or ensemble methods to mitigate the systematic biases and structural failures identified in the models.

In conclusion, this paper establishes that while LLMs have revolutionized argument mining, their reasoning capabilities are not yet perfect. Success depends heavily on prompt engineering, ensemble techniques, and a critical understanding of the limitations of current datasets.