Technical Summary: Magis-Bench

Problem Statement

Existing benchmarks for legal AI predominantly focus on advocacy-oriented tasks, such as drafting legal arguments or providing legal advice. However, a well-functioning legal system relies equally on the capacity to judge those arguments—weighing competing claims, applying doctrine to facts, and rendering reasoned, impartial decisions. While current evaluations test persuasive writing, they often fail to assess the specific competencies required for magistrate-level reasoning, which demands comprehensive analysis and authoritative resolution grounded in law. There is a lack of benchmarks that evaluate Large Language Models (LLMs) against official certification criteria used to select human judges.

Methodology

Dataset Construction (Magis-Bench)

The authors constructed Magis-Bench, a dataset derived from recent Brazilian competitive examinations for judicial positions (substitute judges). These examinations represent the second phase of judicial selection, designed to assess candidates' ability to apply legal knowledge in sophisticated writing tasks.

• Scope: The benchmark comprises 74 questions from eight examinations conducted between 2023 and 2025 (covering federal and state courts).
• Task Types:
  • Discursive Legal Analysis (58 questions): Multi-turn questions presenting factual scenarios requiring legal analysis.
  • Sentence Drafting (16 questions): Practical exercises requiring the composition of complete civil (8) and criminal (8) judicial sentences following Brazilian procedural law.
• Evaluation Criteria: Each question is accompanied by official evaluation rubrics produced by the examination boards, specifying expected legal concepts, analytical steps, and structural elements.

Evaluation Protocol

The authors evaluated 23 state-of-the-art LLMs (including proprietary models from OpenAI, Anthropic, Google, Maritaca AI, and various open-source families) using an LLM-as-a-judge methodology.

• Judges: Four independent frontier models served as evaluators: GPT-5.1, Gemini-2.5-Pro, Gemini-3-Pro-Preview, and Claude-4.5-Opus.
• Scoring Process: For each response, the judge received the original question, the official rubric, and the model's generated output. Judges assigned a score from 0 to 10 with a brief justification.
• Blind Evaluation: Judges were blinded to the identity of the candidate model to mitigate stylistic biases.
• Consistency Checks: To ensure robustness, the authors measured inter-judge agreement using Kendall's $W$ (overall concordance) and Kendall's $\tau$ (pairwise rank correlations). Temperature was set to 0 for judges to maximize reproducibility.

Key Contributions

1. Magis-Bench Dataset: A benchmark of 74 magistrate-level legal tasks derived from real judicial selection exams, complete with official evaluation rubrics.
2. Multi-Judge Evaluation Framework: A methodology employing four independent frontier models as judges, demonstrating high inter-judge agreement and providing a robust mechanism for evaluating open-ended legal writing at scale.
3. Comprehensive Performance Baselines: A systematic evaluation of 23 LLMs, establishing performance hierarchies for judicial writing tasks and revealing the current limitations of AI in this domain.

Results

Performance Hierarchy

• Top Performers: Google's Gemini-3-Pro-Preview achieved the highest average score (6.97/10), followed by Gemini-3-Flash-Preview (6.67) and Claude-4.5-Opus (6.46).
• Overall Difficulty: Even the best-performing models scored below 70% of the maximum possible score. This indicates that magistrate-level legal reasoning and writing remain a significant challenge for current LLMs.
• Task Difficulty: Discursive questions were the easiest (mean score 4.68), followed by criminal sentence drafting (4.39), with civil sentence drafting being the most difficult (3.89).
• Model Tiers: A clear hierarchy emerged, with frontier models occupying the top five positions, mid-tier models (including reasoning-enhanced models) in ranks 6–14, and smaller open-source models (primarily Qwen family) in the lower tier.

Inter-Judge Agreement

• High Consensus: The four judge models exhibited remarkably high agreement, with a Kendall's $W$ of 0.984. Pairwise correlations ($\tau$) ranged from 0.897 to 0.945, all statistically significant ($p < 10^{-13}$).
• Calibration: When tested against "oracle" responses (generated by a model with access to the rubrics), all judges assigned near-perfect scores (mean 9.957), confirming their ability to distinguish high-quality outputs from typical model responses.
• Robustness: Leave-one-judge-out analysis showed that removing any single judge did not significantly alter the overall ranking (no model moved more than two positions).

Significance and Claims

The paper claims that Magis-Bench addresses a critical gap in legal AI evaluation by shifting focus from advocacy to adjudicative reasoning. The authors emphasize that while LLMs are increasingly considered for legal research and decision support, their capacity to function as impartial judges remains unproven.

The study demonstrates that:

1. LLM-as-a-judge is viable: The high inter-judge agreement suggests that automated evaluation can reliably capture genuine quality differences in legal writing without human intervention for every item.
2. Current models are insufficient for judicial roles: The fact that top models score below 70% highlights that judicial-level reasoning is a distinct and difficult capability not yet mastered by state-of-the-art LLMs.
3. Standardization is possible: The use of official rubrics allows for systematic, criteria-grounded assessment, moving beyond subjective preferences.

The authors conclude that while the benchmark provides a rigorous way to study magistrate-level writing, it does not advocate for the immediate deployment of LLMs in real-world adjudication. Instead, it serves as a diagnostic tool to track progress in legal AI capabilities.