💬 NLP

AppellateGen: A Benchmark for Appellate Legal Judgment Generation

This paper introduces AppellateGen, a benchmark comprising 7,351 case pairs for second-instance legal judgment generation, and proposes a judicial SOP-based multi-agent system (SLMAS) to address the dialectical complexities of appellate review, revealing that while the system improves logical consistency, current LLMs still struggle with the inherent challenges of appellate reasoning.

Original authors: Hongkun Yang, Lionel Z. Wang, Wei Fan, Yiran Hu, Lixu Wang, Chenyu Liu, Yu Zeng, Shenghong Fu, Lei Gong, Zhengxin Zhang, Haoyang Li, Jiexin Zheng, Xin Xu

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hongkun Yang, Lionel Z. Wang, Wei Fan, Yiran Hu, Lixu Wang, Chenyu Liu, Yu Zeng, Shenghong Fu, Lei Gong, Zhengxin Zhang, Haoyang Li, Jiexin Zheng, Xin Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge. In a normal trial (the First Instance), you listen to two people argue, look at the evidence they bring to the table, and make a decision. It's like a referee blowing the whistle at the end of a soccer match based on what happened during the game.

But what happens when one of the players screams, "Wait! I found a new video camera footage that proves I didn't do it!" or "The referee missed a rule!"? They ask for a Second Look. This is the Appellate (Second-Instance) Review.

This paper, AppellateGen, is about teaching computers how to be that "Second Look" judge. Here is the breakdown in simple terms:

1. The Problem: The Computer is Too "Linear"

Right now, most AI legal tools are like a student who only knows how to write a story from start to finish. They look at the facts and write a verdict.

The Flaw: They don't understand the drama of an appeal. An appeal isn't just a new story; it's a debate. You have to compare the old verdict with the new evidence and figure out where the first judge went wrong.
The Analogy: Imagine a computer trying to edit a movie. It's great at writing a script from scratch, but if you ask it to "fix the ending because the actor forgot their lines in the middle scene," it gets confused. It doesn't know how to look back, find the mistake, and rewrite the logic.

2. The Solution: A New "Gym" for AI (The Benchmark)

The researchers built a new training ground called AppellateGen.

What's inside? They gathered 7,351 real-life legal cases where a first decision was made, and then a second decision was made later.
The Twist: They didn't just give the AI the final answer. They gave it the "Before" (First Verdict), the "New Evidence" (like a new witness or a hidden photo), and the "After" (Second Verdict).
The Goal: The AI has to act like a detective. It must look at the old verdict, see the new evidence, and write a brand new legal document that explains why the old decision was right or wrong.

3. The Method: The "Assembly Line" of Judges (SLMAS)

The researchers realized that asking one giant AI brain to do this whole complex job at once makes it hallucinate (make things up). So, they built a Legal Multi-Agent System (SLMAS).

Think of this not as one super-genius judge, but as a team of four specialized clerks working on an assembly line, following a strict rulebook (Standard Operating Procedure):

The Detective (Issue Identification): "Okay, what is the actual argument here? Is it about the money, the speed of the car, or the law?"
The Librarian (Legal Retrieval): "Now that we know the argument, let's find the exact law book pages that apply to this specific situation."
The Forecaster (Reversal Prediction): "Based on the new evidence and the law, should we keep the old verdict or throw it out?"
The Scribe (Drafting): "Okay, now write the official document explaining our decision clearly."

By breaking the big, scary task into four small, manageable steps, the AI makes fewer mistakes and thinks more logically.

4. The Results: Who Wins?

The researchers tested this new system against:

General AI: The "smart" models you hear about in the news (like Qwen or Gemini).
Specialized Legal AI: Models trained only on legal texts.

The Surprise: The General AIs (with the help of the "Assembly Line" team) actually did better than the specialized Legal AIs!

Why? Because appeals aren't about memorizing laws; they are about logic and reasoning. It's like a chess game. A specialized model might know all the chess pieces, but a general model with a good strategy (the assembly line) knows how to think through the moves.

5. The Takeaway

This paper is a big step forward. It tells us that to make AI truly useful in law, we can't just ask it to "predict the winner." We have to teach it how to argue, compare, and correct itself, just like a human judge does when reviewing a case.

In short: They built a new test to see if AI can handle the messy, complicated business of "appealing a decision," and they found that if you give AI a good team structure, it can actually do a pretty good job of thinking like a judge.

1. Problem Definition

The paper addresses a critical gap in Legal Natural Language Processing (Legal NLP): the lack of benchmarks and models capable of handling Second-Instance (Appellate) Legal Judgment Generation.

Limitations of Current Research: Existing Legal Judgment Prediction (LJP) and generation tasks predominantly focus on First-Instance trials. These tasks typically model a static "fact-to-verdict" mapping, where a model generates a judgment based solely on initial facts.
The Appellate Challenge: Appellate review is inherently dialectical. It requires a comparative analysis between the original judgment and new evidence or grievances raised by the appellant. In Civil Law jurisdictions (like China), appellate courts can re-examine facts and admit new evidence, leading to complex reasoning tasks where the model must:
- Identify discrepancies between the first and second instances.
- Process dynamic factual updates (new evidence).
- Determine whether to affirm, reverse, or remand the initial ruling based on legal errors (fact-finding, law application, or procedure).
Current Shortcomings: Existing benchmarks either focus on discriminative classification (multiple-choice) or generative tasks based on static facts. They fail to capture the procedural interdependency and causal reasoning required for appellate decisions.

2. Methodology

The authors propose a two-pronged approach: the creation of a new dataset (AppellateGen) and a novel framework for solving the task (SLMAS).

A. The AppellateGen Benchmark

Data Collection: The authors collected 4.7 million raw judgment documents from China Judgments Online (2021–2024).
Matching Pipeline: They developed a multi-stage pipeline to link first-instance judgments with their corresponding second-instance outcomes. This involved:
- Regex Extraction: Using regular expressions to extract case identifiers from second-instance documents to find first-instance counterparts.
- Semantic Validation: Using an LLM to verify that the parties, cause of action, and facts match between the two documents.
- Quality Control: Manual expert review of a 5% sample achieved a 96% consistency rate.
Dataset Statistics: The final curated dataset contains 7,351 paired cases (First-instance + Second-instance).
- Distribution: 89.3% Affirmations, 10.7% Reversals.
- Fine-Grained Annotation: Unlike standard datasets, AppellateGen includes annotations for:
  - Judgment Result: Affirm vs. Reverse.
  - Rationale for Reversal: Specific legal triggers (e.g., erroneous fact-finding, misapplication of law).
  - New Evidence: Facts introduced only in the second instance.
  - Statutory Citations: Specific articles controlling the appellate decision.

B. The SLMAS Framework (Standard Operating Procedure-based Legal Multi-Agent System)

To address the complexity of appellate reasoning and mitigate hallucinations, the authors propose SLMAS, which mimics the human judicial workflow by decomposing the generation task into four sequential agents:

Disputed Issues Identification Agent (Analysis): Analyzes the discrepancy between the first-instance judgment and new evidence to distill core points of contention ( $z_{issue}$ ).
Legal Retrieval Agent (Search): Queries a legal knowledge base using the identified issues to retrieve relevant statutes and judicial interpretations ( $z_{law}$ ).
Reversal Prediction Agent (Predict): Acts as a judge's internal deliberation, integrating context and laws to predict the outcome (Affirm/Reverse) and generate a rationale ( $z_{verdict}$ ).
Document Generation Agent (Write): Synthesizes the entire reasoning chain ( $z = \{z_{issue}, z_{law}, z_{verdict}\}$ ) to draft the final second-instance judgment.

This approach enforces a Directed Acyclic Graph (DAG) of reasoning, ensuring the final output is grounded in explicit legal logic rather than statistical correlations.

3. Key Contributions

Pioneering Task Definition: Shifts the paradigm from static first-instance prediction to Second-Instance Legal Judgment Generation, requiring dynamic dialectical reasoning.
AppellateGen Benchmark: The first large-scale dataset (7,351 pairs) explicitly linking trial proceedings with appellate outcomes, featuring fine-grained annotations for reversal rationales and evidentiary updates.
Methodological Framework (SLMAS): A multi-agent system that decomposes the judicial workflow. It demonstrates that structuring the generation process into discrete reasoning stages significantly improves logical consistency and reduces hallucinations.
Empirical Insight: The study reveals that general-purpose LLMs with strong reasoning capabilities (e.g., Qwen3, Gemini) outperform traditional domain-specific legal models (e.g., DISC-LawLLM) on this task. This suggests that cognitive flexibility and comparative reasoning are more critical than static domain pre-training for appellate tasks.

4. Experimental Results

The authors evaluated SLMAS against various baselines, including single-agent LLMs (with and without self-reflection/self-consistency), commercial models (Gemini 2.5), and domain-specific models.

Performance Metrics: Evaluated using ROUGE-L, BERTScore, and an LLM-as-a-Judge framework (scoring Verdict Consistency, Fact Consistency, Legal Application, and Logical Reasoning on a 0–5 scale).
Key Findings:
- SLMAS Superiority: Using Qwen3-235B, SLMAS improved the average generation quality score from 2.257 (single agent) to 2.536 and increased Reversal Prediction Accuracy from 61.42% to 67.24%.
- Outperforming Commercial Models: SLMAS (Qwen3-235B) outperformed the proprietary Gemini 2.5 Flash (66.53% accuracy).
- Domain vs. General Models: General LLMs achieved >60% accuracy, while domain-specific models lagged significantly (35–37%), highlighting the need for reasoning over static knowledge retrieval.
- Ablation Study: Removing the "Disputed Issues Identification" agent caused a drop in performance, confirming that explicit issue definition is crucial for guiding legal retrieval and preventing error propagation.
- Metric Divergence: SLMAS showed lower ROUGE-L scores than some baselines (which tended to verbatim copy input) but significantly higher Logical Reasoning and Verdict Consistency scores, proving that semantic evaluation is superior to n-gram overlap for this task.

5. Significance and Future Work

Theoretical Impact: The paper establishes that legal AI must evolve from simple text generation to causal dependency modeling and dialectical reasoning. It highlights that the "black box" end-to-end generation is insufficient for complex legal workflows.
Practical Application: The SLMAS framework offers a blueprint for building reliable legal AI assistants that can simulate judicial workflows, potentially aiding judges in drafting appellate opinions and reducing the burden of manual review.
Limitations: The dataset is currently limited to Mainland China (Civil Law system) and the Chinese language. The authors note that while SLMAS improves performance, the absolute accuracy (67.24%) and logical reasoning scores (2.6/5) indicate that current models still struggle with the intricate nuances of high-stakes judicial writing.

In conclusion, AppellateGen provides a rigorous testbed for evaluating long-context reasoning in legal AI, and SLMAS demonstrates that decomposing complex legal tasks into structured, multi-agent workflows is a viable path toward more robust and logically consistent legal judgment generation.