LEXam: Benchmarking Legal Reasoning on 340 Law Exams

1. Problem Statement

Despite recent advances in test-time scaling and reasoning capabilities of Large Language Models (LLMs), long-form legal reasoning remains a significant challenge. Existing legal benchmarks often suffer from three critical limitations:

• Outcome-only Evaluation: Most benchmarks focus solely on the correctness of the final answer (e.g., multiple-choice selection), ignoring the intermediate reasoning steps required to reach a legally sound conclusion.
• Lack of Complexity: Many datasets rely on narrow, task-specific formulations (e.g., statute retrieval or simple classification) rather than holistic, exam-style reasoning that requires issue spotting, rule application, and handling ill-defined problems.
• Evaluation Reliability: Evaluating open-ended legal responses is difficult. Traditional metrics (BLEU, ROUGE) fail to capture legal nuance, while "LLM-as-a-Judge" approaches often suffer from self-bias, family-bias, and a lack of alignment with human expert standards.
• Multilingual Gaps: Legal reasoning varies significantly between Common Law and Civil Law systems, and existing benchmarks often lack robust multilingual (specifically English/German) coverage to test cultural and linguistic biases.

2. Methodology

A. Dataset Construction (LEXAM)

The authors constructed LEXAM, a comprehensive benchmark derived from 340 real law exams spanning 116 courses at the University of Zurich (Bachelor's and Master's levels).

• Scale & Composition: The dataset contains 7,537 questions in both English and German.
  • 2,841 Open-ended Questions: Long-form problems requiring structured legal arguments.
  • 4,696 Multiple-Choice Questions (MCQs): Including standard MCQs and True/False questions converted into MCQs with varying numbers of options (4 to 32) to test robustness.
• Metadata & Diversity: Questions cover 78 legal subdomains across Private, Public, Criminal, and Interdisciplinary law. The dataset includes Swiss, International, and generic jurisdictions.
• Reference Standards: Open-ended questions are paired with reference answers written by leading law professors, including explicit guidance on the expected reasoning chain (e.g., issue spotting, rule recall, application).

B. Evaluation Framework

The paper introduces a rigorous evaluation paradigm moving beyond simple accuracy:

• Ensemble LLM-as-a-Judge: To evaluate open-ended questions, the authors employ an ensemble of three LLMs (GPT-4o, DeepSeek-V3, and Qwen3-32B).
  • Scoring Mechanism: The ensemble uses a minimum-score strategy (taking the lowest score assigned by any of the three judges) to mitigate self-bias and family-bias, ensuring robustness.
  • Validation: The ensemble was validated against human experts using the Alternative Annotator Test (Alt-test). The ensemble achieved a winning rate of 1.00 against human experts, statistically proving it can serve as a reliable proxy for human evaluation.
• Robustness Checks:
  • Perturbation Analysis: MCQs were tested with varying numbers of distractors (4, 8, 16, 32) to determine if models rely on genuine reasoning or pattern matching.
  • Multilingual Consistency: Performance was analyzed across English and German to detect linguistic biases.

C. Experimental Setup

• Models Evaluated: A wide range of 35+ SOTA LLMs, categorized into Reasoning models (e.g., GPT-5, DeepSeek-R1, Claude-3.7), Large conventional models (e.g., Llama-3.1-405B), and Small models.
• Metrics:
  • Open Questions: Judge Score (0.0–1.0) based on alignment with reference answers.
  • MCQs: Accuracy, with standard error calculated via bootstrapping.

3. Key Contributions

1. Novel Benchmark: Introduction of LEXAM, the first general legal reasoning dataset featuring challenging, long-form ground-truth answers derived from real university exams, covering 78 subdomains.
2. Process-Based Evaluation: A scalable framework for evaluating the reasoning process (not just the outcome) using an expert-validated, ensemble LLM-as-a-Judge system.
3. Comprehensive Baselines: Extensive benchmarking of 35+ models, establishing a new state-of-the-art baseline for legal reasoning and highlighting specific failure modes.
4. Statistical Validation: Rigorous proof (via Alt-test) that carefully constructed LLM ensembles can replace human experts for evaluating open-ended legal reasoning, addressing the scalability bottleneck in legal AI research.

4. Key Results

A. Performance on Open-Ended Questions

• Reasoning Models Dominate: Specialized reasoning models (e.g., GPT-5 with 70.20, Gemini-2.5-Pro with 67.40) significantly outperform conventional large models and small models.
• Struggle with Structure: Even top models struggle with complex, multi-step legal reasoning. They frequently fail to correctly identify the applicable legal chapter (e.g., confusing PILA Chapter 12 with other chapters) or miss critical jurisdictional nuances (e.g., domicile of parties).
• Language Gap: All model groups perform better on English questions than German ones, with the gap widening for smaller models.

B. Performance on Multiple-Choice Questions

• Distractor Sensitivity: Model accuracy drops significantly as the number of answer choices increases.
  • Example: Gemini-2.5-Pro drops from 68.6% (4 choices) to 35.6% (32 choices).
  • This suggests many models rely on heuristics or "guessing" rather than deep reasoning when faced with complex distractors.
• Negative Phrasing: Models exhibit a sharp performance decline on negatively phrased questions (e.g., "Which statements are incorrect?"), indicating difficulty with logical negation in legal contexts.

C. Expert Validation & Failure Modes

• Human Alignment: The ensemble judge correlates strongly with human experts (Pearson $r \approx 0.70$).
• Common Failure Modes:
  • Hallucinations: Citing non-existent statutes or case law.
  • Reasoning Flaws: Incorrect application of rules (e.g., applying the wrong chapter of the Private International Law Act).
  • Language Issues: Smaller models often produce incoherent outputs or mix languages in German contexts.
  • Overconfidence: Models often provide authoritative-sounding but legally shallow or incorrect conclusions.

5. Significance and Impact

• Real-World Relevance: By using actual law school exams, LEXAM provides a high-fidelity testbed for assessing whether LLMs can handle the high-stakes, nuanced reasoning required in legal practice, moving beyond toy datasets.
• Safety & Reliability: The study highlights that current LLMs are not yet reliable for autonomous legal decision-making due to hallucinations and reasoning gaps. It emphasizes the need for human oversight.
• Evaluation Standard: The proposed Ensemble LLM-as-a-Judge methodology offers a reproducible, cost-effective, and statistically validated alternative to expensive human annotation for open-ended tasks in specialized domains.
• Future Direction: The paper identifies the need for broader multilingual and cross-jurisdictional (Common vs. Civil law) datasets to fully understand and mitigate cultural biases in legal AI.

Project Page: https://lexam-benchmark.github.io/