💬 NLP

Mining Legal Arguments to Study Judicial Formalism

This study introduces the MADON dataset and a three-stage NLP pipeline to automatically classify judicial reasoning and formalism in Czech Supreme Court decisions, challenging prevailing narratives about formalism in Central and Eastern Europe while demonstrating the potential of legal argument mining for computational legal studies.

Original authors: Tomáš Koref, Lena Held, Mahammad Namazov, Harun Kumru, Yassine Thlija, Ivan Habernal

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tomáš Koref, Lena Held, Mahammad Namazov, Harun Kumru, Yassine Thlija, Ivan Habernal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a giant library where every book is a court decision. For decades, legal scholars have whispered a rumor about the libraries in Central and Eastern Europe (CEE): "The judges here are robots."

The rumor goes like this: These judges don't think about fairness, justice, or the "spirit" of the law. Instead, they are "formalists"—mechanical robots who only look at the exact words on the page. If a law says "file by day 15," and you file on day 16 because your lawyer was in the hospital, a "robot judge" throws your case out. A "human judge" would say, "Wait, let's look at the bigger picture."

The Problem:
Until now, no one could prove if this rumor was true or just gossip. Reading 300,000 court decisions by hand would take a human team of lawyers a lifetime. It's like trying to find a specific needle in a haystack the size of a mountain.

The Solution: The "Legal Detective" AI
This paper introduces a team of AI detectives (using advanced Large Language Models) to solve the mystery. They built a system to read Czech court decisions and answer three big questions:

Can the AI spot where the judge is actually making an argument?
Can the AI tell if the judge is using a "robot" argument (strict text) or a "human" argument (values, fairness, consequences)?
Can the AI decide if the whole decision was "robotic" or "human"?

The Toolkit: How They Did It

1. The "MADON" Dataset (The Training Camp)
Before the AI could go to work, the researchers had to teach it. They gathered 272 real court decisions and hired legal experts to act as "coaches."

The Analogy: Imagine teaching a dog to fetch. You don't just throw a ball; you show it the ball, say "fetch," and reward it.
The Process: The experts labeled every paragraph of the 272 decisions. They marked which paragraphs contained arguments and what kind of argument it was (e.g., "Linguistic," "Historical," "Practical Consequences"). They also gave each whole decision a "Holistic Label": Formalistic (Robot) or Non-Formalistic (Human).

2. The Three-Stage Pipeline (The Assembly Line)
Instead of throwing the whole text at a giant, expensive AI brain, they built a smart assembly line to save money and make the process clearer:

Stage 1 (The Filter): A small, fast AI scans the document and throws away 85% of the text (like boring procedural lists or repetitive phrases) that doesn't contain real arguments. It's like a bouncer at a club checking IDs and letting only the relevant people in.
Stage 2 (The Classifier): A powerful AI reads the remaining "argumentative" text and sorts the arguments into 8 different buckets (e.g., "This is about the text of the law," "This is about fairness").
Stage 3 (The Verdict): A final calculator looks at the mix of arguments. If the judge used mostly "fairness" and "consequences" arguments, the AI says, "This is a Human Judge." If they only used "text" arguments, it says, "This is a Robot Judge."

The Big Surprise: The Rumor Was Wrong!

When the AI finished reading the data, the results were shocking. The "Robot Judge" narrative was mostly a myth.

The Findings:

Text is Rare: The judges rarely relied on the strict, literal wording of the law (only about 5.7% of their arguments). They weren't reading the dictionary; they were thinking.
Values Rule: The most common arguments were about Principles, Values, and Teleology (the "purpose" of the law). They were asking, "What is the right thing to do?" and "What are the consequences?"
Precedent is King: Instead of ignoring past cases (as the "robot" rumor suggested), these judges relied heavily on Case Law (previous decisions). They were actually very good at following the "human" tradition of looking at how similar cases were decided before.

The Twist:
The study did find a difference between the two highest courts in Czechia:

The Supreme Court (older, inherited from the communist era) was slightly more "robotic" in the past.
The Supreme Administrative Court (newer, created in 2003) has become increasingly "human" and open-minded over time.

Why This Matters

This study is like a lie detector test for legal history.

For Scholars: It proves that you can use AI to study philosophy and judicial behavior at a massive scale, something that was previously impossible.
For the Public: It challenges the idea that Eastern European judges are cold, mechanical robots. It shows they are actually engaging in complex, value-based reasoning, just like judges in the US or Western Europe.

In a Nutshell:
The researchers built a high-tech "argument miner" to dig through thousands of court cases. They expected to find a mountain of robotic, text-only decisions. Instead, they found a garden full of judges arguing about fairness, values, and the greater good. The "robot judges" were mostly a myth, and AI helped us finally see the truth.

1. Problem Statement

The paper addresses the difficulty of systematically analyzing judicial reasoning at scale, particularly regarding judicial formalism in Central and Eastern Europe (CEE).

The Narrative: CEE judiciaries (specifically in Czechia) are widely alleged to be "formalistic," characterized by "hyperpositivism," "mechanical jurisprudence," and a rigid reliance on statutory text while ignoring broader principles, purpose, or external standards.
The Gap: These claims are largely anecdotal or based on small, manually coded samples. There is a lack of large-scale empirical evidence to validate or refute these narratives.
Technical Challenges:
- Conceptual Ambiguity: Legal concepts like "formalism" are abstract and disputed.
- Data Scarcity & Imbalance: High-quality annotated legal datasets are rare, and argument types are often imbalanced (e.g., some arguments appear very rarely).
- Language: Most Legal NLP (LNLP) research focuses on English; Czech legal texts present specific linguistic and domain challenges.
- Explainability: Black-box models often lack the interpretability required for legal scholarship.

2. Methodology

The authors propose a three-stage pipeline combining legal theory with state-of-the-art Natural Language Processing (NLP).

A. Dataset Creation: MADON

Source: 272 decisions from the Czech Supreme Court (SC) and Supreme Administrative Court (SAC), spanning 1997–2024.
Annotation: 9,183 paragraphs were annotated by legal experts.
Labels:
1. Argument Presence: Binary (Argumentative vs. Non-argumentative).
2. Argument Types: Multi-label classification into 8 categories based on CEE formalism tenets:
  - Formalistic: Linguistic Interpretation (LIN), Systemic Interpretation (SI), Case Law (CL), Doctrine (D).
  - Non-Formalistic: Historical Interpretation (HI), Principles of Law/Values (PL), Teleological Interpretation (TI), Practical Consequences (PC).
3. Holistic Formalism: A binary label (Formalistic vs. Non-Formalistic) for the entire decision, based on a complex assessment of the five core tenets of CEE formalism.
Reliability: Inter-annotator agreement was measured (Cohen's $\kappa$ = 0.65 for holistic; Krippendorff's $\alpha$ varied by argument type). Disagreements were resolved by a senior legal expert.

B. Corpus and Pretraining

A large corpus of 300,511 anonymized Czech court decisions was compiled for Continued Pretraining (CPT) of Large Language Models (LLMs) to adapt them to the Czech legal domain.

C. Model Architecture & Tasks

The study decomposes the problem into three tasks:

Task 1 (Argument Detection): Classify paragraphs as argumentative or not.
Task 2 (Argument Classification): Multi-label classification of argument types.
Task 3 (Holistic Formalism Classification): Classify the entire decision.

Models Tested:

Llama 3.1 8B: Decoder-only, multilingual. Tested with full fine-tuning and Parameter-Efficient Fine-Tuning (PEFT/LoRA).
ModernBERT: Encoder-only, long-context support. Tested with full fine-tuning and CPT.
MLP (Multi-Layer Perceptron): Used as a feature-based classifier for Task 3, using argument counts as input.

Loss Functions:

Addressed class imbalance using Asymmetric Loss (which emphasizes minority positive instances) compared to standard Binary Cross-Entropy (BCE).

Evaluation Metric:

Balanced F1 (Bal-F1): A macro-average of Positive F1 ( $F1^+$ ) and Negative F1 ( $F1^-$ ). This is crucial because standard F1 ignores true negatives, which are vital in legal analysis to avoid over-predicting rare argument types.

D. The Three-Stage Pipeline

To improve explainability and reduce computational costs, the authors designed a pipeline:

Filtering: ModernBERT identifies and removes non-argumentative paragraphs (filtering out ~85% of text).
Classification: Llama 3.1 classifies the remaining text into argument types.
Decision: An MLP uses the distribution of these predicted arguments to classify the decision as formalistic or non-formalistic.

3. Key Contributions

MADON Dataset: The first large-scale, expert-annotated dataset for Czech judicial reasoning, including 272 decisions, 9,183 paragraphs, and 8 argument types.
Operationalization of Formalism: A rigorous definition of CEE formalism translated into concrete annotation guidelines and 8 distinct argument types.
Methodological Innovation:
- Demonstrated that Asymmetric Loss significantly outperforms BCE in imbalanced legal multi-label tasks.
- Showed that full fine-tuning is superior to PEFT for these specific legal tasks.
- Proposed a hybrid pipeline (Filter $\to$ Classify $\to$ Predict) that balances performance, explainability (via SHAP values), and compute efficiency.
Empirical Evidence: The first large-scale empirical test of the "CEE formalism" narrative using NLP.

4. Results

Task 1: Argument Presence Detection

Best Model: ModernBERT-large-Czech-Legal.
Performance: 82.6% Bal-F1.
Finding: Continued pretraining (CPT) provided negligible gains over the base model. PEFT (LoRA) caused a severe performance drop, indicating full fine-tuning is necessary for this task.

Task 2: Argument Type Classification

Best Model: Llama 3.1 8B (Full Fine-tuning) + Asymmetric Loss.
Performance: 77.5% Bal-F1 (Macro).
Finding:
- Asymmetric Loss boosted performance significantly (up to 9 points).
- PEFT models failed to outperform baselines.
- Difficulty: Argument types with low inter-annotator agreement (Linguistic, Systemic, Practical Consequences) remained the hardest to classify.

Task 3: Holistic Formalism Classification

Best End-to-End Model: ModernBERT-large-Czech-Legal (82.2% Bal-F1).
Best Pipeline Approach: The 3-stage pipeline (Filter $\to$ Llama $\to$ MLP) achieved 83.8% Bal-F1.
Upper Bound: Using gold-standard (human) argument labels as features for an MLP yielded 91.7% Bal-F1, proving that argument distribution is a strong predictor of formalism.
Explainability: SHAP analysis revealed that Principles of Law (PL) and Teleological Interpretation (TI) are the strongest predictors of non-formalism, while Systemic Interpretation (SI) had low predictive value despite being theoretically "formalistic."

5. Significance and Empirical Findings

The study challenges prevailing legal narratives about the Czech and CEE judiciary:

Textualism is Rare: Contrary to the "formalism" narrative, Linguistic Interpretation accounts for only 5.7% of all arguments.
Precedent is Dominant: Case Law (CL) is the most frequent argument type (37.4%), contradicting claims that CEE courts are precedent-averse.
Purpose over Text: Non-formalistic arguments (Principles, Teleological) make up 32.3% of arguments, suggesting courts frequently engage in purposive reasoning.
Court Divergence: While the Supreme Court (SC) was historically viewed as formalistic and the Supreme Administrative Court (SAC) as discursive, the data shows their argumentation profiles are similar. However, the SAC has trended more toward non-formalistic reasoning in recent years (post-2014).

Conclusion: The paper demonstrates that legal argument mining can effectively quantify judicial philosophies. It provides a reproducible, explainable pipeline that can be applied across jurisdictions to move legal scholarship from anecdotal claims to data-driven evidence. All datasets, models, and code are open-sourced.