💬 NLP

Beyond Logical Forms: LLM-Extracted Patterns for Fallacy Classification

This paper proposes a framework that leverages Large Language Models to inductively extract data-driven patterns combining abstract logical structures and linguistic cues, significantly improving the accuracy and generalization of fallacy classification compared to zero-shot baselines and existing methods.

Original authors: Eleni Papadopulos, Firoj Alam, Giovanni Da San Martino

Published 2026-06-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eleni Papadopulos, Firoj Alam, Giovanni Da San Martino

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to catch people who are lying or tricking you with bad logic. These tricks are called logical fallacies. Sometimes, the trick is obvious, like saying, "If it rains, the ground gets wet. It is raining, so the ground is wet." That's good logic. But a fallacy is like a magician's sleight of hand: the argument looks smart on the surface, but if you look closer, the logic is broken.

For a long time, computers trying to catch these tricks have been like detectives who only look at the blueprint of a building. They check the structure (the "logical form") but ignore the paint, the furniture, and the smell of the room. The problem is, many bad arguments are tricky because of how they are said, not just their structure. They use specific words, emotional tones, or rhetorical tricks that a blueprint alone can't see.

The New Approach: Teaching the Computer to "Read Between the Lines"

This paper introduces a new way to teach computers to spot these tricks. Instead of just giving the computer a rigid rulebook written by humans, the authors let a super-smart AI (a Large Language Model, or LLM) act like a student intern.

Here is how they trained this intern:

The Lesson (Explanation Generation): They showed the AI thousands of examples of bad arguments and asked it to write an explanation of why they were bad. This is like asking a student to explain a math mistake in their own words.
The Pattern Hunt (Pattern Extraction): Once the AI understood the mistakes, the researchers asked it to find the "recipe" for each type of trick. The AI didn't just write a rule; it created a template.
- Analogy: Imagine the AI is a chef. Instead of just saying "Don't burn the toast," it writes a recipe card that says: "If you see someone saying 'X happened, then Y happened, so X caused Y' (even if they didn't prove it), that's a 'False Cause' fallacy."
- Crucially, these recipe cards (patterns) kept the specific "flavor" of the language (like emotional words or specific phrases) while still keeping the logical structure clear.

The Experiment: Putting the Patterns to the Test

The researchers tested this new "recipe card" system against other methods using a dataset called LOGIC (a collection of logical puzzles).

The Old Way (Zero-Shot): Just asking the AI, "Is this a fallacy?" without any help. It was like asking a detective to solve a case with no clues.
The "Blueprint" Way: Giving the AI only the abstract logical structure (like the building blueprint).
The New Way (Patterns): Giving the AI the "recipe cards" the intern created.

The Results:
The new method was a huge success.

When the AI used the pattern-based recipes, it got about 73-74% of the answers right.
This was much better than the "Zero-Shot" method (which got about 62% right) and better than using just the abstract blueprints.
It even beat other advanced methods that relied on human-written rules.

The paper also tested these "recipes" on different types of conversations, like Reddit comments (casual, messy internet talk) and presidential debate transcripts (formal, political talk). The patterns worked well in these new environments too, proving they aren't just memorizing the training data but actually understanding the logic.

Why Some Tricks Were Harder to Catch

The researchers noticed something interesting:

Group 1 (The Easy Ones): Fallacies with very clear structures, like "Circular Reasoning" (saying the same thing twice in different words), were caught almost perfectly. These are like a house with a crooked roof; you can see the mistake from the outside.
Group 2 (The Hard Ones): Fallacies that rely on subtle context or emotional manipulation, like "Red Herring" (distracting you with an irrelevant topic), were harder to catch. These are like a house where the walls are straight, but the furniture is arranged to hide a secret door. The AI struggled a bit more here because it needed to understand the nuance of the conversation, not just the shape of the argument.

The Bottom Line

This paper shows that to catch a logical trick, you can't just look at the skeleton of the argument; you have to look at the whole body, including the clothes it's wearing (the language) and the context it's in.

By letting AI generate its own "detective guides" (patterns) based on real examples, the researchers created a system that is much better at spotting bad reasoning than previous methods. It's a data-driven approach that doesn't require humans to write every single rule, making it flexible enough to handle the messy, nuanced way people actually talk and argue.

Important Note: The authors explicitly state they are only testing these methods on datasets of arguments. They do not claim this technology is ready for clinical use, legal judgment, or real-time social media moderation, nor do they suggest it can be used to manipulate people. It is currently a research tool for better understanding how to detect fallacies.

Technical Summary: Beyond Logical Forms: LLM-Extracted Patterns for Fallacy Classification

Problem Statement

Logical fallacies, defined as defective reasoning patterns, contribute significantly to information disorder. While traditional classification distinguishes between formal fallacies (violations of logical structure) and informal fallacies (context-dependent errors in everyday language), automated detection remains challenging. Existing approaches face two primary limitations:

Over-reliance on Abstract Logic: Methods that map arguments to single, abstract logical forms often fail to capture the full spectrum of how fallacies manifest in natural discourse, particularly for informal fallacies where nuance and context are critical.
Rigidity of Hand-Crafted Schemas: Prior work utilizing manually designed templates (e.g., based on Walton's argumentation schemes) is limited in scope, failing to represent the diversity of fallacy types and intra-class variations.
Computational Cost: Many supervised approaches require heavy fine-tuning of Large Language Models (LLMs), which is resource-intensive.

The paper posits that purely abstract representations are insufficient and that a more comprehensive characterization requires integrating logical structure with context-level linguistic cues (lexical markers, rhetorical devices).

Methodology

The authors propose a data-driven framework that inductively extracts "structural patterns" from fallacious examples and their explanations using LLMs. This approach avoids fine-tuning, relying instead on inference-only prompting.

1. Pattern Generation

The process involves two steps using specific LLMs:

Step 1: Explanation Generation: Using llama-3.3-70B-Instruct, the system generates explanations for training examples that justify why a specific sentence contains a designated fallacy. This step aims to make the reasoning process explicit.
Step 2: Pattern Extraction: Using OpenAI's reasoning model o4-mini, the system extracts patterns from the sentences and their generated explanations. The model is instructed to:
- Preserve function words (prepositions, adverbs).
- Abstract content words into placeholders (e.g., X, Y, Z) while retaining the original reasoning form.
- Capture recurring lexical schemes and syntactic particles that convey fallacious intent.
- Derive new fallacy definitions based on these patterns.

The resulting patterns combine logical structure (variables) with concrete linguistic features (specific phrases, loaded terms).

2. Experimental Setup

The framework was evaluated on three datasets:

LOGIC: 2,449 examples across 13 fallacy types (educational dialogues).
REDDIT: Fallacious comments from various subreddits.
ELECDEBATE: Televised US presidential debate transcripts.

Baselines and Configurations:
The study compared the proposed method against several baselines and prompting strategies:

Zero-Shot: Providing only fallacy names.
Definitions (DEF): Using manually curated or LLM-generated definitions.
Logical Forms: Using expert-crafted logical forms from logicallyfallacious.com.
LLM-Derived Patterns: Using the extracted structural patterns (PATTERNS) and definitions (NEW DEF).
One-Shot Variants: Including static examples, examples with explanations, and dynamic retrieval based on semantic or syntactic similarity.
Multi-Step Classification: A chain-of-thought approach where the model generates a logical form, matches it to a pattern, and then classifies.

Experiments were conducted across six different LLMs (including gpt-4o, o4-mini, deepseek-r1, llama-3.3-70B, etc.) to assess robustness and the impact of reasoning capabilities.

Key Contributions

Automated Pattern Extraction: A novel framework leveraging LLMs to inductively extract structural patterns from fallacious examples and explanations, moving beyond hand-crafted templates.
Inference-Only Classification: A method that achieves high performance without fine-tuning, utilizing generated patterns and prompting strategies (including one-shot and dynamic retrieval).
Cross-Dataset Validation: Demonstration that patterns extracted from one domain (LOGIC) generalize effectively to diverse domains (REDDIT, ELECDEBATE) and argumentative styles.

Results

The experiments on the LOGIC dataset yielded the following key findings:

Performance Gains: The proposed approach significantly outperformed zero-shot baselines and competing unsupervised methods.
- gpt-4o with PATTERNS achieved 73.5% accuracy.
- o4-mini with DYNAMIC + EXP + PATTERNS achieved 74.2% accuracy.
- These results represent an 8.2% improvement over expert-crafted logical forms and a 10.7% average improvement over manually designed schemes (Robbani et al., 2024).
Reasoning Models: Models with explicit reasoning capabilities (e.g., o4-mini, gpt-4o) consistently demonstrated superior performance compared to non-reasoning models.
Dynamic Prompting: Dynamic one-shot prompting (retrieving similar examples) provided significant gains (approx. 8.87% increase) over static one-shot approaches.
Generalization:
- On REDDIT, LOGIC-derived patterns outperformed competing approaches, achieving results comparable to patterns extracted directly from the dataset.
- On ELECDEBATE, dynamic retrieval performed best, though LOGIC-derived patterns still showed robust generalization.
- Cross-dataset tests confirmed that patterns generated from REDDIT and ELECDEBATE could effectively classify fallacies in LOGIC, validating the transferability of the method.

Error Analysis

Group 1 vs. Group 2: The models performed significantly better on fallacies with clear structural characteristics (Group 1, e.g., Circular Reasoning, Ad Hominem) compared to those relying heavily on nuanced semantic understanding (Group 2, e.g., Red Herring, Emotional Language).
Ambiguity: Some arguments conform to multiple structural patterns, creating ambiguity. The study noted that while pattern matching is effective, it requires context-aware application to resolve cases where structural alignment alone is insufficient.
Multi-Step Limitations: The multi-step classification approach (generate form $\to$ match pattern $\to$ classify) underperformed compared to direct pattern matching, suggesting that current LLMs struggle to bridge the gap between abstract logical forms and content-dependent manifestations in a single chain of thought.

Significance and Claims

The paper claims that data-driven pattern extraction is an effective method for generating generalizable logical representations for fallacy detection. The authors argue that:

Merging abstract logical structures with context-level linguistic cues is beneficial for classification, particularly for informal fallacies.
LLMs can successfully inductively extract these representations without the need for expensive fine-tuning.
The resulting patterns are not bound to a fixed set of fallacies and can adapt to intra-class variations, offering a more flexible alternative to hand-crafted schemas.

The study concludes that while purely abstract representations are insufficient, the proposed framework successfully combines logical structure with linguistic cues to significantly enhance automated fallacy detection performance across diverse domains.