AR-BENCH: Benchmarking Legal Reasoning with Judgment Error Detection, Classification and Correction

Technical Summary of AR-BENCH: Benchmarking Legal Reasoning with Judgment Error Detection, Classification and Correction

Problem Statement
Legal judgments are prone to errors due to the complexity of case circumstances and the abstract nature of legal concepts. While judicial systems employ appellate review mechanisms to correct these errors, these processes face significant efficiency pressures due to surging case volumes. Existing legal AI research has predominantly focused on legal judgment prediction (predicting outcomes from facts) and legal document generation. However, these tasks fundamentally differ from appellate review, which occurs post-judgment and centers on anomaly detection—identifying, classifying, and correcting errors in finalized documents—rather than prediction or generation. Currently, there is no standardized evaluation benchmark for this critical diagnostic reasoning task, leaving a gap in assessing the reliability of AI in legal practice.

Methodology
To address this gap, the authors introduce a novel task named APPELLATE REVIEW and construct a corresponding benchmark dataset, AR-BENCH.

1. Task Decomposition: The APPELLATE REVIEW task is decomposed into three sequential subtasks:

   • Error Detection: Determining whether a given judgment document contains legal errors (binary classification).
   • Error Classification: Identifying the specific type of error present (e.g., charge, prison term, or fine errors).
   • Error Correction: Generating a legally valid revision to correct the identified error.

2. Dataset Construction (AR-BENCH):

   • Source: Derived from the JuDGE dataset (originally from China Judgments Online), comprising 8,700 finely annotated decisions and 34,617 supplementary corpora.
   • Annotation: The dataset includes 20 fine-grained annotations covering reasoning processes, law articles, and sentencing factors.
   • Error Taxonomy: The authors manually constructed six specific error scenarios relevant to real-world appellate review:
     • Charge Errors: Erroneous Conviction (E1) and Omission of Charges (E2).
     • Prison Term Errors: Sentencing Outside Limits (E3) and Failure to Consider Sentencing Factors (E4).
     • Fine Errors: Fixed-Amount Fine Outside Limits (E5) and Percentage-based Fine Outside Limits (E6).
   • Data Processing: The dataset was cleaned to remove duplicates and multi-defendant cases, with sensitive information anonymized.

3. Evaluation Framework:

   • The authors evaluated 14 Large Language Models (LLMs), categorized into General-Domain, Reasoning-Enhanced, and Domain-Specific models.
   • Metrics: Accuracy, Macro-Precision, Macro-Recall, and Macro-F1 were used for detection and classification. For correction, specific metrics including ImpScore and Acc@0.1 were employed for numerical outputs (prison terms and fines).
   • Input Settings: Experiments tested various input configurations, ranging from basic case facts to settings including reasoning processes, cited law articles, and erroneous legal knowledge.

Key Results

• Task Difficulty: Error detection was found to be the most tractable task, while error correction proved substantially more challenging, with a significant performance drop across all models.
• Model Performance:
  • General-Domain LLMs consistently achieved the strongest performance in error detection and classification.
  • Reasoning-Enhanced LLMs outperformed general-domain models specifically in the error correction task, suggesting that explicit or implicit reasoning mechanisms (Chain-of-Thought) are beneficial for inferring and generating corrected outputs.
  • Domain-Specific LLMs lagged significantly behind both general and reasoning-enhanced models across all tasks, even when compared to general models of similar parameter scales.
• Error Type Sensitivity: Models performed relatively well on charge and prison term errors but struggled significantly with fine prediction, particularly proportional fine errors (E6), indicating limitations in numerical understanding.
• Human vs. AI: Human annotators substantially outperformed all evaluated models (e.g., Human MaF1 for correction was 70.10% vs. 53.81% for the best LLM), highlighting significant room for improvement.
• Input Impact: Introducing explicit reasoning processes improved performance, whereas naively incorporating full legal statutes sometimes degraded performance, suggesting potential noise introduction.

Significance and Contributions
The paper claims three primary contributions:

1. Novel Task Definition: The proposal of APPELLATE REVIEW as a distinct task that directly assesses models' diagnostic reasoning capabilities in legal practice, moving beyond the prediction and generation paradigms.
2. Structured Framework: The innovative decomposition of the review process into three progressive subtasks (detection, classification, correction), providing a clear framework for systematic evaluation.
3. Benchmark and Empirical Evidence: The construction of AR-BENCH, a large-scale benchmark with fine-grained annotations. The comprehensive evaluation of mainstream LLMs reveals critical limitations in handling complex legal logic and provides empirical evidence for future improvements in legal AI.

The authors emphasize that while AI-assisted review may help standardize adjudication criteria, current models cannot replace human prosecutors or judges due to the complexity of legal cases and the risks of overlooking unique circumstances. The work aims to clarify the capabilities and boundaries of existing AI models for legal practitioners.