From Judgments to Issues: Structured Extraction of Legal Reasoning with Citation-Hallucination Control

Technical Summary: From Judgments to Issues: Structured Extraction of Legal Reasoning with Citation-Hallucination Control

Problem Statement
Modern judicial systems, particularly in Italy, generate court decisions at a scale that exceeds the capacity of legal professionals to read them individually. The Italian tax-court database alone contains nearly one million judgments, growing by hundreds of thousands annually. While this volume holds significant value for secondary uses—such as precedent search, citation-network analysis, and training legal NLP systems—the unstructured nature of the original PDFs hinders these applications. Existing extraction methods often treat the entire judgment as a single unit, introducing noise from procedural boilerplate and failing to isolate specific legal questions. Furthermore, Large Language Models (LLMs) are prone to "hallucinating" legal citations, fabricating plausible but non-existent references to legislation and case law, which undermines the reliability of automated legal reasoning pipelines.

Methodology
The authors propose an automated pipeline designed to decompose Italian tax-court judgments into individual legal issues and extract a structured XML representation for each. The methodology is built upon three core components:

1. Issue-Based Decomposition and Schema Design:
   Rather than processing judgments as monolithic blocks, the pipeline treats the legal issue (questione giuridica) as the unit of analysis. This aligns with judicial justification theories (e.g., MacCormick, 1994) where adjudication is viewed as the resolution of specific legal questions. The extracted structure is grounded in the IRAC framework (Issue, Rule, Application, Conclusion) and the legal syllogism. Each issue is encoded in XML with specific fields:

   • <text>: A "Whether + clause" formulation of the legal question.
   • <issue_outcome>: The decision, classified by beneficiary (taxpayer, tax authority, other).
   • <factual_premises>: Relevant facts and events.
   • <legal_references>: A list of citations (legislation, case law, principles, administrative practice) with types.
   • <citation_reason>: Explanations for why specific references were cited.
   • <judge_reasoning>: The application of law to facts.
   • <summary>: An abstractive summary of the issue.

2. Extraction Pipeline and Model Selection:
   The pipeline processes a corpus of approximately 330,000 first- and second-instance Italian tax-court judgments. To balance capability with the cost of processing hundreds of thousands of documents, the authors selected DeepSeek V3, a general-purpose model offering a favorable cost-performance ratio (approx. $0.0035 per judgment). The extraction employs a two-step prompting strategy for longer judgments:

   • Step 1: Identify issues and extract the <text>, <summary>, and <issue_outcome>.
   • Step 2: For each identified issue, extract the detailed reasoning, facts, and citations.
     A post-extraction correction step uses few-shot prompting to ensure the <text> field adheres strictly to the "Whether + clause" format.

3. Hallucination Control via Linkoln:
   To address the unreliability of LLM citations, the pipeline integrates a verification layer using the Linkoln library (Bacci et al., 2019). This step:

   • Parses legal references in both the original judgment text and the LLM output.
   • Normalizes them to standard identifiers (URN-NIR, ECLI, CELEX).
   • Compares the two sets. Any reference present in the LLM output but not found in the source text is flagged as a hallucination and removed.
   • For legal principles (which lack standard identifiers), a whitelist of 182 canonical principles derived by experts is used to validate matches.

Key Contributions
The paper makes three primary contributions:

1. Structured Schema: A novel, issue-level XML schema grounded in IRAC and legal syllogism, designed to be machine-readable, useful for human consultation, and reliably extractable by mid-range LLMs.
2. Cost-Efficient Pipeline with Hallucination Control: A scalable pipeline utilizing DeepSeek V3 combined with an automatic, parser-based hallucination filter that normalizes and validates citations against the source text.
3. Expert-Validated Evaluation: A rigorous validation protocol on 50 judgments annotated by two PhD-level tax law experts, reporting inter-annotator agreement (IAA), LLM-vs-expert agreement, and the performance of the hallucination filter.

Results
The evaluation on the 50-judgment subsample yields the following findings:

• Issue Extraction: The LLM demonstrates high precision (93–94%) but moderate recall (70–82%) compared to human experts. The dominant error mode is under-extraction (missing issues) rather than hallucination. Only 4 of 78 extracted issues were marked as absent by experts.
• Citation Extraction: The LLM achieves an F1 score of approximately 74% against expert annotations. While the model extracts a similar number of references as experts, it sometimes includes references not deemed relevant to the specific issue.
• Hallucination Filter Performance: The filter is highly effective. Before filtering, the hallucination rate was 11.7%. After filtering, it dropped to 0.9%. The filter removed 29 of 31 hallucinated references while incorrectly removing only 7 valid citations (3.0% false positive rate). The filter was least effective on "principles of law," which had a 69% hallucination rate before filtering.
• Qualitative Scores: Free-text fields (reasoning, summary, outcome) received high average scores (4.6–4.9/5) from experts. Inter-annotator agreement on these qualitative dimensions was very high (Gwet's AC2 > 0.93).
• Stochasticity: The authors note that even with fixed seeds, the LLM exhibits non-determinism. Run-to-run agreement (F1 ~78% for issues, ~73% for citations) is lower than inter-annotator agreement but comparable to LLM-vs-human agreement, suggesting that model stochasticity accounts for a significant portion of the performance gap.

Significance and Claims
The authors claim this work provides the first issue-level, expert-validated structured extraction pipeline with hallucination control specifically for Italian tax-court decisions. They position the pipeline as a concrete starting point for downstream applications, including:

• Issue-level retrieval: Isolating single legal questions to improve search precision.
• Citation-network analysis: Constructing graphs where edges reflect references actually used to decide disputes, rather than incidental mentions.
• Dataset construction: Creating large-scale, structured datasets of legal reasoning for training and benchmarking legal NLP systems.

The paper maintains a modest tone, acknowledging limitations such as the small validation sample (50 judgments), the reliance on a single model and jurisdiction, and the inherent subjectivity in segmenting judgments into issues. The authors emphasize that the proposed format serves as a methodological template for structured extraction that is independent of specific jurisdictions or models, demonstrating that current LLMs can reliably instantiate complex legal reasoning structures when paired with appropriate verification layers.