💬 NLP

CAF-Gen: A Multi-Agent System for Enriching Argumentation Structures

This paper introduces CAF-Gen, a multi-agent framework that utilizes an iterative Creator-Reviewer pipeline to automatically transform shallow argument structures into rich, formal Carneades Argumentation Framework (CAF) models, effectively overcoming the structural limitations of single-pass generation methods.

Original authors: Jakub Bąba, Jarosław Chudziak

Published 2026-06-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jakub Bąba, Jarosław Chudziak

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a complex debate, like a heated discussion in a town hall meeting. Currently, computers are pretty good at spotting the "who said what" and "who is attacking whom." They can tell you that Person A made a claim and Person B offered a reason to support it. This is like having a simple map of the debate with just dots (people) and lines (connections).

However, the authors of this paper argue that this simple map isn't enough for deep, serious reasoning. To really understand the debate, you need to know how the arguments work. Is that reason a fact? Is it an assumption? Is it an expert opinion? What kind of proof is needed to believe it? This is like upgrading from a basic street map to a detailed 3D model that shows the terrain, the building materials, and the structural integrity of every house.

This paper introduces a new system called CAF-Gen that does exactly this upgrade, but automatically.

The Problem: The "One-Shot" Mistake

The authors tried using a single, super-smart AI (a Large Language Model) to do this upgrade all at once. Think of this like asking a talented but overworked architect to draw a perfect, legally binding building plan in a single breath. The AI is smart, but it often makes mistakes, gets confused, or "hallucinates" (makes up details) because the rules for formal logic are very strict. It's like trying to bake a perfect soufflé without checking the oven once; if you get it wrong, the whole thing collapses.

The Solution: The "Creator and Critic" Team

To fix this, the authors built a Multi-Agent System. Instead of one AI doing everything, they created a team of two specialized AI agents who work together in a loop:

The Creator: This AI acts like a draftsman. It takes the simple "dots and lines" map of the debate and tries to build the complex 3D model. It adds labels like "Expert Opinion" or "Proof Standard" to every part of the argument.
The Reviewer: This AI acts like a strict building inspector. It looks at the Creator's draft and checks for errors. Did the Creator mix up a fact with an assumption? Did they use the wrong type of proof? Is the structure even valid?

How They Work Together (The Iterative Loop)

Here is the magic part: They don't just work once. They work in a cycle, like a writer and an editor.

Round 1: The Creator makes a draft. The Reviewer finds 10 mistakes and sends it back with a list of corrections.
Round 2: The Creator fixes the mistakes and tries again. The Reviewer checks again.
Round 3: If there are still small issues, they go back and forth.

The paper found that this "tug-of-war" between the Creator and the Reviewer is crucial. When the AI tried to do it alone (one pass), it only got it right about 35% of the time. But with this team loop, the success rate jumped to over 91%. It's like how a team of writers and editors produces a much better book than a single author working alone.

The Results

The team tested this system on a large collection of persuasive essays (402 essays). They wanted to see two things:

Did the system get better with feedback? Yes. The "Reviewer" caught common errors like picking the wrong "argument scheme" (e.g., calling something an "Expert Opinion" when it was actually just an "Example") or making structural mistakes.
Did it stay true to the original text? Yes. The system was very careful not to invent new arguments out of thin air. It kept the original structure of the essays almost perfectly (99% accuracy) while adding the rich, detailed layers on top.

The Bottom Line

The paper concludes that by using a team of AI agents that critique and refine each other's work, we can automatically turn simple text arguments into complex, formal models that are ready for serious logical analysis. It proves that you don't need a human to check every single step; a smart, self-correcting AI team can do the heavy lifting of building these complex argument structures reliably.

Technical Summary: CAF-Gen

Problem Statement
The paper addresses a critical limitation in current Argument Mining (AM) techniques: while they effectively identify basic claims, premises, and simple support/attack relations, they struggle to capture the richer structural information required by advanced formal argumentation frameworks. Specifically, standard AM outputs lack the semantic depth needed for frameworks like the Carneades Argumentation Framework (CAF), which requires distinct premise types (e.g., ordinary premises, assumptions, exceptions), proof standards, and explicit argument schemes (e.g., Walton's taxonomy). Furthermore, the application of such frameworks is hindered by a lack of annotated corpora capable of supporting these complex structures. While Large Language Models (LLMs) show promise in natural language processing, single-pass generation often fails to adhere to the strict logical constraints of formal argumentation, leading to hallucinations and structural instability.

Methodology: CAF-Gen Framework
To bridge the gap between shallow argument annotations and rigorous CAF-compliant models, the authors introduce CAF-Gen, an automated multi-agent system. The framework operates on the UKP Argument Annotated Essays v2 corpus, transforming basic argument graphs into enriched CAF structures through an iterative Creator-Reviewer pipeline:

Task Formulation: The system treats the problem as a structure-preserving graph transformation. It maps a source graph ( $G_{src}$ ) containing basic components (Major Claims, Claims, Premises) and relations (Support, Attack) to a target CAF graph ( $G_{CAF}$ ). The target graph enriches nodes with statement types and proof standards, and edges with argument schemes and polarity.
The Creator Agent: This generative agent takes a unit from the corpus and constructs a candidate CAF entry. Its subtasks include:
- Mapping components to CAF statements (e.g., mapping Major Claims to conclusions).
- Classifying statement types (ordinary premise, presumption, exception).
- Assigning proof standards (e.g., Scintilla of Evidence, Preponderance of Evidence).
- Identifying argument schemes from Walton's taxonomy.
The Reviewer Agent: Acting as a critic, this agent evaluates the Creator's output against a checklist of integrity, textual coherence, and completeness. It flags issues such as structural inconsistencies, incorrect scheme selections, or missing fields.
Iterative Refinement: If the Reviewer rejects the model (due to Errors or Warnings), it provides specific feedback, and the Creator revises the model. This cycle continues until the model is accepted or a predefined iteration limit (five cycles) is reached. The system utilizes Google's Gemini 2.5 Pro model with a low temperature (0.1) to ensure deterministic adherence to the schema.

Key Contributions

Automated Enrichment: The introduction of a multi-agent framework capable of automatically enriching shallow argument structures into complex, CAF-compliant models without requiring pre-existing annotated corpora for the target schema.
Iterative Quality Assurance: The validation of an iterative Creator-Reviewer pipeline specifically adapted for formal argumentation. This approach mitigates the structural instability typical of single-pass LLM generation by simulating collective intelligence to refine outputs.
Grounded Transformation: A methodology that ensures the enriched models remain grounded in the original input structure, preventing the introduction of ungrounded nodes or edges while adding necessary logical attributes.

Experimental Results
The system was evaluated on the full UKP Argument Annotated Essays v2 corpus (402 essays).

Refinement Effectiveness (Q1): The iterative process proved essential. The First-Pass Acceptance Rate was only 34.6%, but after refinement, the Final Acceptance Rate rose to 91.3%. The average number of iterations per accepted essay was 2.35. The most common issues flagged by the Reviewer were Structural Inconsistencies (44.7%) and Inappropriate Argument Scheme Selection (38.9%).
Structural Fidelity (Q2): To measure how well the enriched models preserved the original argumentative structure, the output was projected back to the source schema.
- Component Identification: Achieved a Precision of 99.8%, Recall of 98.8%, and F1-score of 99.3%.
- Relation Identification: Achieved a Recall of 99.1% but a lower Precision of 67.1% (F1-score: 80.0%). The authors note that the lower precision indicates the system not only preserves original relations but also proactively adds new connections that were implicit in the source text.

Significance and Claims
The paper claims that CAF-Gen demonstrates a robust methodology for the automated modeling of formal argumentation. The primary significance lies in proving that multi-agent systems can overcome the limitations of single-pass generation, significantly improving output quality through iterative feedback loops. The authors assert that their system successfully bridges the gap between basic argument annotations and the detailed structures demanded by the Carneades Argumentation Framework.

The work suggests that such multi-agent collaboration is crucial for ensuring the structural integrity and semantic richness of generated models, a task previously believed to require human-level judgment. By connecting natural language argumentation with formal reasoning schemas, the research contributes to the development of intelligent systems for domains requiring precision, such as legal reasoning and academic debate. The authors remain modest regarding the absolute correctness of the newly assigned CAF features (due to the lack of ground truth for such complex structures) but emphasize that the system reliably respects the underlying argumentative structures without destructive behavior.