An LLM-Based System for Argument Reconstruction

Technical Summary: An LLM-Based System for Argument Reconstruction

Problem Statement
Computational argumentation faces a fundamental dichotomy. On one hand, formal Argumentation Frameworks (AFs) provide rigorous, graph-based structures for evaluating argument acceptability but struggle to extract arguments from raw natural language, often failing to handle implicit premises, pragmatic context, and the variability of real-world discourse. On the other hand, Large Language Models (LLMs) possess rich contextual knowledge and can identify implicit connections but lack systematic, theoretically grounded structures for representing arguments. Off-the-shelf LLM generations often result in inconsistent outputs that are difficult to compare or aggregate, while traditional argument mining approaches frequently rely on surface-level features that miss deeper logical relations. The core challenge is to bridge these perspectives: creating a system that leverages the fluency and common-sense reasoning of LLMs while adhering to the structural rigor of formal argumentation frameworks.

Methodology
The authors propose an end-to-end LLM-based system that reconstructs arguments from natural language text into Directed Acyclic Graphs (DAGs). The system employs a multi-stage pipeline designed to incrementally build argumentative structures, ensuring well-formed outputs through explicit constraints. The pipeline consists of the following stages:

1. Argumentative Component Identification: The LLM identifies and enumerates claims and premises, filtering out irrelevant content. Unlike rule-based systems, it leverages pre-trained contextual understanding to distinguish argumentative spans.
2. Merging and Rewriting (Optional): Redundant or closely connected components are merged to ensure logical unity. Components are rewritten to resolve context-dependent expressions (e.g., pronouns), making them self-contained.
3. Conclusion and Relation Identification:
   • Conclusion Identification: The main conclusion is extracted.
   • Premise Relation (Recursive Strategy): Using a breadth-first traversal, the system recursively identifies premises supporting or attacking a target conclusion. This decomposition reduces complexity and enforces an acyclic structure by restricting candidate premises to unvisited components.
4. Advanced Structural Refinement (Optional):
   • Unvisited Premise Check: Ensures disconnected components are attached or merged if they form cycles.
   • Linked vs. Convergent Premises: Distinguishes between premises that jointly justify a conclusion (linked) and those providing independent reasons (convergent), using intermediate nodes for linked relations.
   • Implicit Premise Recovery: Identifies and generates missing premises (enthymemes) necessary for the argument's validity.
   • Rebuttal and Undercut Distinction: Differentiates between attacks on a premise (rebuttal) and attacks on the inference link itself (undercut), inserting intermediate nodes for the latter.
5. Graph Construction and Visualization: The final output is a DAG where nodes represent components and directed edges represent support or attack relations. The system includes a transitive reduction step to remove redundant edges and a diagram builder for visual inspection.

Key Contributions

• Unified Framework: The paper presents a novel system that integrates LLM-driven extraction with formal argumentation constraints, effectively bridging the gap between natural language processing and formal logic.
• End-to-End Pipeline: A modular, multi-stage pipeline that not only extracts components but also infers complex logical relations (undercuts, linked premises, implicit assumptions) and enforces a DAG structure.
• New Dataset: The authors constructed a small, annotated dataset of 42 arguments derived from an argumentation theory textbook (Introduction to Argumentative Analysis by Marcus Sacrini), featuring diagrams that align with their proposed representation scheme.

Experimental Results
The system was evaluated through two complementary experiments:

1. Internal Evaluation (Textbook Dataset):

   • Component Level: The system achieved 92.50% accuracy in conclusion detection. Span detection yielded an F1-score of 75.70%, and relation detection accuracy was 80.57%.
   • Structure Level: Human annotators rated the system highly on a 1–3 scale for Implicit Premises (2.86), Undercuts (2.90), and Convergent/Linked Premises (2.85), with high inter-annotator agreement.
   • Global Level: The generated graphs were rated highly for Completeness (2.96) and Faithfulness (2.92), indicating the system captures main structures without hallucinating content.

2. External Evaluation (Benchmark Datasets: AAEC and AbstRCT):

   • When evaluated on standard argument mining benchmarks, the full pipeline (including span detection) showed lower performance (e.g., F1 of 27.64 on AAEC for component classification), primarily due to difficulties in exact span matching compared to ground truth.
   • However, when provided with gold-standard components (bypassing span detection), the system's performance improved significantly. On the AbstRCT dataset, it achieved a Link Prediction F1 of 51.51 and Relation Detection F1 of 46.21, outperforming the state-of-the-art baseline (Morio et al., 2022) by +11.77 and +7.5 respectively. This suggests the model effectively captures relational structure once components are known.

Significance and Claims
The paper claims that the proposed system demonstrates the potential of LLM-based pipelines for scalable argument reconstruction. It successfully recovers argumentative structures that align with argumentation theory standards, including nuanced phenomena like undercuts and implicit premises. The authors conclude that while the system adequately reconstructs textbook-style arguments and achieves reasonable performance on benchmarks when adapted to different annotation schemes, challenges remain in component identification (span detection). The work highlights that combining the contextual strengths of LLMs with the structural rigor of formal frameworks is a viable path forward, though future work is needed to refine prompting strategies and evaluate the system on more complex, realistic datasets.