Validate Your Authority: Benchmarking LLMs on Multi-Label Precedent Treatment Classification

Technical Summary: Validating Authority in Legal Precedent Treatment Classification

Problem Statement
In common law jurisdictions, the doctrine of stare decisis requires legal practitioners to determine if a judicial decision remains "good law" or if its authority has been weakened or nullified by subsequent cases. Cases can be "negatively treated" in various nuanced ways, such as being overruled, criticized, distinguished, or questioned. Misclassifying these treatments carries significant risk, potentially leading to unfavorable judgments and professional repercussions. While commercial citator services (e.g., Shepard's, KeyCite) employ human editors to analyze these treatments, documented human errors have spurred interest in automated solutions. However, previous research indicates that classifying citation treatment is intrinsically difficult for machine learning models, particularly due to the semantic subtleties of legal reasoning and the scarcity of high-quality, expert-annotated data. This paper addresses the central question: How accurately can contemporary Large Language Models (LLMs) replicate the sophisticated legal reasoning required to classify the various ways a judicial precedent is treated in a subsequent case?

Methodology
The authors conducted a systematic benchmarking of modern LLMs on a new, expert-annotated dataset derived from Hellyer (2018), which originally evaluated commercial citator services.

• Dataset Curation: The authors digitized and structured a corpus of 329 citing relationships, filtering out ambiguous cases to create a high-confidence set. They retrieved full-text opinions for 239 citing cases via the CourtListener API to provide models with full-document context, rather than isolated snippets.
• Ground Truth Construction: The ground truth was established using a priority-based logic. Explicit corrections made by the original annotator were prioritized. For entries without explicit corrections, labels deemed "acceptable" were adopted. Crucially, the dataset is multi-label, acknowledging that multiple distinct treatments (e.g., "Criticized by" and "Questioned by") can be simultaneously valid for a single relationship.
• Classification Schemas: The study utilizes a two-tiered schema:
  1. Fine-Grained: 16 distinct categories (e.g., Overruled, Distinguished by, Reversed by).
  2. High-Level: 5 conceptually distinct groups (e.g., INVALIDATED, CRITICIZED OR QUESTIONED, LIMITED OR DISTINGUISHED) designed to mitigate the extreme semantic overlap of fine-grained labels.
• Experimental Setup: The study evaluated four models: Google's Gemini 2.5 Pro, Gemini 2.5 Flash, OpenAI's GPT-5-mini, and the open-source Qwen3 (30B). Experiments were conducted using zero-shot and few-shot prompting strategies. Models were provided with the seed case citation and the full text of the citing case, specifically the paragraphs analyzing the cited case.
• Evaluation Metrics: Beyond standard Instance-Based Accuracy, the authors introduced the Average Severity Error and Median Severity Error. These metrics map labels to a severity scale (1 for Neutral to 5 for Invalidated) to quantify the practical impact of misclassifications, distinguishing between minor errors (confusing two negative labels) and critical errors (confusing a negative treatment with a neutral one).

Key Results
The experiments revealed a performance split between the models depending on the complexity of the schema:

• High-Level Schema: Gemini 2.5 Flash achieved the highest accuracy (79.1% in zero-shot). It performed exceptionally well on the most frequent label, LIMITED OR DISTINGUISHED (Precision: 0.947), but struggled with underrepresented classes like NEUTRAL CITATION (F1: 0.400).
• Fine-Grained Schema: GPT-5-mini was the top performer (67.7% in zero-shot), though performance dropped significantly for rare labels. Seven of the fifteen fine-grained labels resulted in zero successful predictions (F1: 0.000), including legally significant treatments like Reversed by and Questioned by.
• Severity Analysis: In terms of Average Severity Error, Gemini 2.5 Flash again led, achieving the lowest error scores (0.3933 for high-level, 0.3755 for fine-grained). The Median Severity Error for top models was 0.0, indicating that for over half of the predictions, the models produced the perfectly correct label.
• Qualitative Findings: Error analysis revealed that models often struggle with the "target of judicial action" (misattributing whether a court is correcting a party's argument vs. the precedent itself) and face inherent ambiguity where distinct labels (e.g., Criticized vs. Not Followed) are semantically overlapping and context-dependent.

Key Contributions

1. New Dataset: The paper introduces a publicly available, expert-annotated dataset of 239 real-world legal citations with full-text context and multi-label ground truth, addressing the scarcity of high-quality data for this specific task.
2. Robust Evaluation Framework: The authors propose the Average Severity Error metric, arguing that standard accuracy is insufficient for legal tasks where the cost of misclassification varies drastically based on the severity of the error.
3. Systematic Benchmark: This work establishes a crucial baseline for the performance of modern LLMs on negative precedent treatment classification, demonstrating that while proprietary models show significant potential, they remain constrained by class imbalance and the intrinsic complexity of legal semantics.

Significance and Claims
The paper claims to be the first academic study to systematically benchmark modern LLMs on the fine-grained, multi-label classification of negative citation treatments. The authors assert that their work provides a necessary baseline for a "critical yet nuanced" NLP task where misclassification carries high risk. They conclude that while leading proprietary models have the potential to automate this aspect of legal research, current performance is limited by data scarcity and the semantic complexity of legal reasoning. The study emphasizes that standard accuracy metrics are inadequate for this domain and that the proposed severity-based evaluation offers a more realistic assessment of model fitness for high-stakes legal applications. The authors remain modest about the definitive nature of the results, noting that the inherent subjectivity of legal interpretation suggests a ceiling on achievable accuracy and that the results should be viewed as an exploration of current capabilities rather than a final judgment on legal reasoning.