LAMUS: A Large-Scale Corpus for Legal Argument Mining from U.S. Caselaw using LLMs

This paper introduces LAMUS, a large-scale, high-quality sentence-level legal argument mining corpus for U.S. caselaw constructed via an LLM-driven pipeline with human refinement, which demonstrates that chain-of-thought prompting and LLM-assisted verification significantly enhance annotation quality and model performance for future legal NLP research.

The Gist

Imagine you are trying to teach a robot how to read a legal court decision. But here's the catch: a court decision isn't just a story; it's a complex machine made of different gears. Some gears are the facts (what happened), some are the rules (the laws), some are the analysis (how the judge connects the facts to the rules), and some are the conclusion (the final verdict).

If you want a robot to understand the law, it needs to know which part of the text is which gear. This is called Legal Argument Mining.

The problem? We didn't have a big enough "training manual" for the robot, especially for American state courts. Most existing manuals were either too small or written for courts in other countries.

Enter LAMUS, a new project by researchers at the University of North Texas. Think of LAMUS as a massive, high-tech library they built from scratch to teach robots how to read American law.

Here is the story of how they built it and what they learned, explained simply:

1. The Problem: The "Missing Library"

Imagine trying to learn to drive a car, but you only have a manual for a bicycle. That's what legal researchers faced. They had small datasets, but nothing big enough to train powerful AI models on U.S. Supreme Court cases or Texas criminal cases. Without a big library, the AI models were guessing in the dark.

2. The Solution: Building the LAMUS Library

The researchers built a massive dataset called LAMUS.

• The Source: They gathered over 2.9 million sentences from U.S. Supreme Court decisions (dating back to 1921) and Texas criminal court cases.
• The Job: They needed to label every single sentence. Is this sentence a Fact? Is it a Rule? Is it the Conclusion?
• The Method: They didn't hire 1,000 lawyers to do this manually (too expensive!). Instead, they used Large Language Models (LLMs)—super-smart AI chatbots—to do the heavy lifting.

3. The "Human-in-the-Loop" Safety Net

You might think, "If the AI labels the data, how do we know it's right?"
The researchers used a clever trick called "LLM-as-a-Judge."

• Step 1: The AI labeled the sentences.
• Step 2: A second, different AI checked the work. If the second AI thought, "Wait, that sentence looks like a Rule, not a Fact," it flagged it.
• Step 3: Human experts stepped in to fix the flagged sentences.
• The Result: This process corrected nearly 20% of the errors the AI made initially. It's like having a spell-checker that actually understands the context, not just the spelling.

4. The Experiment: How to Talk to the AI

The researchers wanted to know: What is the best way to ask the AI to do this job? They tried three different "languages" (prompting strategies):

• Zero-Shot (The "Cold Call"): They just told the AI, "Here is a sentence. Tell me what it is." No examples.
• Few-Shot (The "Show and Tell"): They gave the AI a few examples first. "Here is a Fact. Here is a Rule. Now you do this one."
• Chain-of-Thought (The "Think Aloud"): They asked the AI to explain its reasoning step-by-step before giving the answer. "First, I see a date. That looks like a fact. Then I see a law citation..."

The Surprising Findings:

• The "Show and Tell" (Few-Shot) Backfired: Surprisingly, giving the AI examples actually made it worse at the job. It seems the AI got confused by the specific examples and tried to copy them too closely, rather than understanding the general rule. It's like trying to learn to cook by memorizing one specific recipe; you fail when the ingredients change.
• The "Think Aloud" (Chain-of-Thought) Won: When the AI was forced to explain its logic step-by-step, it got much smarter. It was like asking a student to show their math work; they got the right answer more often.
• Fine-Tuning is King: The absolute best method wasn't just asking questions; it was teaching the AI. They took a model and trained it specifically on this legal data. This boosted the accuracy from about 76% to 85%. It's the difference between giving a student a textbook (prompting) and hiring a tutor for a semester (fine-tuning).

5. Why This Matters

The LAMUS project is a game-changer for two reasons:

1. The Data: They released a massive, high-quality library of labeled legal text. Now, other researchers can build better legal AI tools without starting from zero.
2. The Guidebook: They figured out the best way to train these AI models. They proved that for legal tasks, you shouldn't just "ask" the AI nicely; you need to either teach it specifically (fine-tuning) or make it think through its steps (Chain-of-Thought).

The Bottom Line

The researchers built a giant, high-quality library of legal arguments and figured out that the best way to teach a robot to read it is to make it "show its work" or give it a dedicated "tutor." This paves the way for future AI tools that can help lawyers find precedents faster, summarize complex cases, and even help judges make more consistent decisions.

In short: They built the map, figured out the best compass, and showed us how to navigate the tricky world of legal AI.

Technical Summary

Here is a detailed technical summary of the paper "LAMUS: A Large-Scale Corpus for Legal Argument Mining from U.S. Caselaw using LLMs."

1. Problem Statement

Legal Argument Mining (LAM) aims to automatically identify and classify functional components of judicial reasoning (e.g., Facts, Issues, Rules, Analysis, Conclusions). Despite the potential for AI to revolutionize legal research and decision support, progress in the U.S. context is severely constrained by:

• Data Scarcity: A lack of large-scale, high-quality, annotated datasets for U.S. caselaw, particularly at the state level. Existing resources focus on non-U.S. jurisdictions (e.g., ECHR) or Chinese law.
• Annotation Complexity: Legal texts involve hierarchical reasoning, specialized vocabulary, and complex references to precedent, making manual annotation expensive and prone to inconsistency.
• Evaluation Gaps: There is insufficient systematic evaluation of Large Language Models (LLMs) for sentence-level legal argument classification under different prompting strategies (zero-shot, few-shot, chain-of-thought) and domain specializations.

2. Methodology

The authors propose a data-centric pipeline to construct the LAMUS corpus and evaluate LLM performance. The methodology consists of four main stages:

A. Corpus Construction & Data Sources

• Sources: The dataset combines U.S. Supreme Court (SCOTUS) decisions (1921–2025) and Texas criminal appellate opinions (1840–2018).
• Preprocessing: Raw texts are cleaned, normalized, and segmented into sentences using legal-domain tools (LexNLP). Invalid segments (headers, citations) are filtered.
• Annotation Schema: Sentences are classified into six categories based on the FIRAC framework:
  1. Fact: Pertinent events or records.
  2. Issue: Legal questions the court must decide.
  3. Rule/Law/Holding: References to statutes, rules, or prior holdings.
  4. Analysis: Synthesis of facts and law to reach a decision.
  5. Conclusion: Final resolutions or opinions.
  6. Other: Non-informative text or structural elements.

B. Automated Annotation & Quality Control (The "LLM-as-Judge" Approach)

• Seed Data: A manually annotated subset of Texas cases (approx. 4,000 sentences) served as the ground truth.
• LLM Verification: To improve label consistency, the authors employed a GPT-based verification step. The model reviewed human-annotated sentences; if the model's prediction disagreed with the human label (or showed high confidence in an alternative), the sentence was flagged.
• Human-in-the-Loop: Flagged sentences underwent manual review. This process corrected ~19.4% of the initial labels, significantly reducing noise and improving inter-annotator agreement.
• Scaling: A high-performing model (LLaMA-3-70B) was then used to automatically label the entire SCOTUS corpus (2.9 million sentences).

C. Model Evaluation Framework

The study evaluated 7 LLMs (ranging from 8B to 54B parameters, including general-purpose and legal-domain models like LegalBERT and SaulLM) under three prompting strategies:

1. Zero-Shot: Task instructions and label definitions only.
2. Few-Shot: Inclusion of 1–100 labeled examples.
3. Chain-of-Thought (CoT): Requiring the model to generate explicit justifications and confidence scores before assigning a label.
4. Fine-Tuning: Supervised fine-tuning (using QLoRA) on the curated Texas dataset.

D. Metrics

Performance was measured using Accuracy, Precision, Recall, and F1-score. Stability was verified via 10 independent runs.

3. Key Contributions

1. LAMUS Corpus: The creation of a massive, sentence-level legal argument mining corpus containing 2,900,083 labeled sentences from U.S. Supreme Court decisions (1921–2025) and Texas criminal cases. This addresses the critical gap in U.S.-specific legal datasets.
2. Scalable Annotation Pipeline: A novel workflow integrating LLM-based automatic labeling with targeted human verification. This hybrid approach improved label consistency and demonstrated a method for scaling corpus construction in complex domains.
3. Systematic Empirical Evaluation: A comprehensive benchmark of general vs. legal-domain LLMs across different prompting strategies and model scales.
4. Critical Insights on Prompting: The discovery that Few-Shot prompting often degrades performance in legal classification tasks, while Chain-of-Thought significantly boosts large models.

4. Key Results & Findings

A. Prompting Strategies

• Chain-of-Thought (CoT): Substantially improved performance for large, general-purpose models. LLaMA-3-8B with CoT achieved 75.89% accuracy, outperforming its zero-shot baseline (65.38%).
• Few-Shot Failure: Contrary to expectations, Few-Shot prompting consistently reduced accuracy compared to Zero-Shot baselines. For LLaMA-3-8B, accuracy dropped from 65.38% (Zero-Shot) to as low as 45.75% (Few-Shot). The authors attribute this to domain mismatch and noise introduced by fixed examples.
• Zero-Shot Stability: Domain-specialized models (e.g., SaulLM-54B) exhibited more stable Zero-Shot behavior but did not benefit as much from CoT as general models.

B. Fine-Tuning Superiority

• Best Performance: Supervised fine-tuning yielded the highest results. Fine-tuned LLaMA-3-8B achieved 85.32% accuracy, a 23.34% improvement over the majority-class baseline and a 9.43% improvement over the best prompting method (CoT).
• LegalBERT: Also performed well (81.30%) after fine-tuning, confirming the value of domain adaptation.

C. Model Scale & Domain Specialization

• General vs. Legal Models: Surprisingly, general-purpose models (LLaMA-3-8B) outperformed specialized legal models (SaulLM-54B, Law-LLM) in this specific sentence-level classification task. The authors suggest general models have better instruction-following capabilities and broader semantic coverage.
• Capacity Matters: CoT prompting was only effective for models with sufficient capacity (≥8B parameters). Smaller models (e.g., SaulLM-7B) performed worse with CoT, likely due to the cognitive load of generating reasoning steps.

D. Data Quality

• Human Verification: The final SCOTUS annotations achieved a Cohen's Kappa of 0.85 (almost perfect agreement) between human experts, validating the LLM-assisted pipeline.
• Domain Shift: The paper highlights a significant distribution shift between Texas criminal cases (dominated by "Facts") and SCOTUS cases (more balanced distribution of "Analysis" and "Rule/Law/Holding").

5. Significance

• Resource Availability: LAMUS provides the first large-scale, structured resource for U.S. Supreme Court argument mining, enabling research into the evolution of judicial reasoning over a century.
• Practical Guidance: The study offers actionable recommendations for legal AI practitioners:
  • Avoid Few-Shot prompting for legal sentence classification.
  • Use Chain-of-Thought for large general models if fine-tuning is not feasible.
  • Prioritize Fine-Tuning for high-stakes, high-accuracy applications.
• Methodological Impact: The paper demonstrates that "LLM-as-Judge" verification is a viable strategy for cleaning and scaling legal datasets, reducing the reliance on prohibitively expensive manual annotation while maintaining high quality.

The code and datasets are publicly available on GitHub to ensure reproducibility.