💬 NLP

Legal-DC: Benchmarking Retrieval-Augmented Generation for Legal Documents

This paper introduces Legal-DC, a specialized benchmark dataset and the LegRAG framework featuring legal adaptive indexing and dual-path self-reflection, to address limitations in Chinese legal Retrieval-Augmented Generation and demonstrate superior performance over existing state-of-the-art methods.

Original authors: Yaocong Li, Qiang Lan, Leihan Zhang, Le Zhang

Published 2026-03-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yaocong Li, Qiang Lan, Leihan Zhang, Le Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex legal puzzle, like figuring out if a drone owner needs to register their device. You ask a super-smart AI assistant for the answer.

In the past, this AI might have "hallucinated"—making up a confident-sounding but completely wrong answer because it was relying on its memory, which might be outdated or fuzzy.

To fix this, researchers built a system called RAG (Retrieval-Augmented Generation). Think of RAG not as a student taking a test from memory, but as a student who is allowed to use their textbook while answering. The AI first looks up the relevant pages in the law books (Retrieval) and then writes the answer based only on what it found (Generation).

However, the researchers in this paper found two big problems with how this was currently being done in China:

The Test Was Flawed: There was no good way to grade the AI. Did it find the right page in the textbook? Did it read it correctly? Existing tests only checked the final answer, not the research process.
The Textbooks Were Messy: Legal documents are structured like a building with floors, rooms, and hallways. Standard AI tools often chop these documents into random, messy chunks, breaking the logical flow of the law.

Here is how the authors fixed these problems, explained simply:

1. The New "Exam Hall": Legal-DC

The authors created a brand-new, specialized test called Legal-DC.

The Library: They gathered 480 real Chinese legal documents (covering things like business rules, contracts, and food safety).
The Questions: They created 2,475 specific questions and answers.
The Secret Sauce: Every single answer is tagged with the exact paragraph in the law book that supports it.
Why it matters: This is like giving a teacher an answer key that says, "The student got this right because they cited Page 42, Paragraph 3." Now, we can grade not just what the AI said, but how well it found the information.

2. The New "Librarian": LegRAG Framework

The authors built a new system called LegRAG to act as the AI's librarian. They realized that if you ask a librarian for a rule about "drone registration," they shouldn't just hand you a random paragraph; they need to understand the structure of the law.

LegRAG uses a Dual-Path Strategy (a two-lane highway for information):

Lane A (The Chunk): It breaks the text into small, semantic paragraphs (like reading a single scene in a movie). This helps find specific details.
Lane B (The Article): It keeps entire legal articles together (like keeping a whole chapter intact). This ensures the AI doesn't miss the context or the "big picture" of a rule.
The Result: By using both lanes at once, the system finds the right information much faster and more accurately than systems that only use one method.

3. The "Self-Check" Mechanism

Even with a great librarian, the AI might still get confused. So, LegRAG includes a Self-Reflection step.

The Metaphor: Imagine the AI writes a draft answer. Before it hands it to you, it stops and asks itself: "Wait, did I actually find the rule that says this? Or am I just guessing?"
If the AI realizes it's missing a key piece of evidence, it goes back to the library, finds the missing page, and rewrites the answer. This acts like a second pair of eyes checking the work before it's submitted.

4. The Results

When they tested this new system against the old ones:

Better Search: It found the right legal pages 2–5% more often than the best existing systems.
Fewer Mistakes: The answers were more accurate because the AI was forced to stick to the evidence.
The Verdict: The system proved that for legal questions, you can't just rely on a smart brain (the AI); you need a smart way to organize the library (the indexing) and a strict editor (the self-reflection).

Summary

Think of this paper as the architects who built a specialized law library (Legal-DC) and a super-efficient librarian (LegRAG) who knows how to read the building's blueprints. They proved that if you want an AI to give you legal advice, you can't just ask it to "guess"; you have to give it the right tools to find the exact rule and a strict process to double-check its work.

They even made their tools and library open for everyone to use, so other developers can build better legal AI systems in the future.

1. Problem Statement

The application of Retrieval-Augmented Generation (RAG) in Chinese legal document consultation faces two critical limitations:

Lack of Specialized Benchmarks: Existing benchmarks primarily focus on answer generation and lack dedicated evaluation for retrievers. They often miss fine-grained annotations that explicitly link questions to specific clauses within legal documents, making joint evaluation of retrieval and generation difficult.
Structural Mismatch: Mainstream RAG systems often fail to accommodate the structured nature of legal provisions. Legal texts require strict adherence to specific articles and clauses; standard chunking methods often break the integrity of legal logic, leading to hallucinations or incorrect citations.
Reliability Issues: Legal scenarios demand high reliability. Current systems struggle with "hallucinations" (generating facts not in the text) and "outdated corpora," and existing evaluation metrics (like BLEU/ROUGE) often fail to capture semantic consistency and factual accuracy in legal contexts.

2. Methodology

A. The Legal-DC Benchmark Dataset

The authors constructed a specialized dataset named Legal-DC to address the evaluation gap:

Scale: Comprises 480 legal documents (laws, administrative regulations, departmental rules) covering market supervision, contract management, food safety, and UAV management.
Content: Contains 2,475 refined question-answer (QA) pairs.
Annotation Process:
- Generation: Used ERNIE-4.0 to generate candidate QA pairs based on strict constraints (answers must rely only on the document).
- Human-in-the-Loop: Five annotators (law students and legal informatics experts) performed a two-round process:
  1. Selection & Annotation: Filtering for "Significance" (high-frequency business needs) and "Quality" (clarity, no hallucinations). Annotators tagged Question Types (Concept, Summary, Logic) and Reference Content (specific article numbers and clauses).
  2. Cross-Proofreading: A "blind review" by new annotators to ensure consistency.
- Quality Control: Achieved high inter-annotator consistency (Cohen's Kappa $\ge$ 0.83).
Structure: Each entry is a 5-tuple: (Document, Question, Answer, Question Type, Reference Content).

B. The LegRAG Framework

The authors propose LegRAG, a framework designed to handle legal document structures and ensure accuracy through a Dual-Path Self-Reflection mechanism:

Legal Adaptive Indexing (Splitter Module):
- Instead of simple semantic chunking, the framework splits documents into two granularities simultaneously:
  - Chunk-based: Semantic paragraphs for context.
  - Article-based: Legal article numbers (e.g., "Article 19") to ensure provision integrity.
Dual-Path Retrieval:
- 1st Retrieval: Hybrid retrieval (Vector Embedding + Keyword matching) to recall top 20 candidates.
- 2nd Retrieval (Reranking): A reranking model filters the top 20 down to the top 5 ( $K=5$ ) most relevant chunks.
Generation with Self-Reflection:
- Initial Generation: The LLM generates an answer based on the top 5 retrieved chunks.
- Self-Verification: A closed-loop mechanism checks the initial answer against the retrieved chunks for:
  - Evidence: Is the answer fully supported by the retrieved text?
  - Consistency: Do legal conclusions match the provisions?
  - Completeness: Are key points omitted?
- If verification fails, the system triggers supplementary retrieval or re-generation.

C. Automated Evaluation Protocol

To overcome the limitations of n-gram metrics (BLEU/ROUGE), the authors introduced an LLM-based automated evaluator (using Qwen3-max):

Dimensions: Relevance (30%), Accuracy (40%), Completeness (20%), and Fluency (10%).
Weighting: Heavily prioritizes Accuracy and Relevance over fluency, reflecting the high-risk nature of legal advice.
Validation: The automated scores showed a strong correlation ( $\rho = 0.87$ ) with human expert ratings.

3. Key Contributions

Legal-DC Benchmark: The first comprehensive Chinese legal RAG benchmark enabling joint evaluation of retrievers and generators with clause-level reference annotations.
LegRAG Framework: A novel architecture integrating dual-path indexing (chunk + article) and self-reflection, specifically tailored to maintain legal clause integrity.
Evaluation Methodology: A robust, automated evaluation protocol that aligns with legal professional standards, moving beyond simple text overlap metrics.
Toolkit: An open-source toolkit facilitating the combination and testing of different RAG components.

4. Experimental Results

The study compared LegRAG against state-of-the-art baselines (LightRAG, QAnything, Weknora) using various LLMs (Qwen3-max, DeepSeek-V3.2).

Retrieval Performance:
- LegRAG achieved 78.02% Recall and 50.23% MRR@5, outperforming the strongest baseline (LightRAG) by 2.16% and 2.67% respectively.
- Dual-Path Strategy: Combining chunk-based and article-based processing yielded the best results (Recall 85.23% with Qwen3-max), proving that structural awareness is critical.
Generation Performance:
- LegRAG + Qwen3-max achieved the highest LLM-eval score (81.25%) and F1 score (68.30%).
- Self-Reflection Impact: Adding the self-reflection module improved the F1 score by 3.52% and LLM-eval by 3.72% for the best configuration.
Task Difficulty Analysis:
- Conceptual questions were easiest to answer.
- Logic questions (requiring multi-hop inference) were the most difficult, highlighting a current bottleneck in legal reasoning.
- Entity Augmentation (adding entity recognition) significantly improved retrieval metrics.

5. Significance

Bridging the Gap: This work fills a critical void in Chinese legal AI by providing a dataset and framework that treats retrieval and generation as equally important, rather than just focusing on the generator.
Practical Applicability: The LegRAG framework offers a practical solution for legal consultation systems, ensuring that answers are not only fluent but legally accurate and traceable to specific clauses.
Standardization: By establishing a rigorous evaluation protocol with weighted dimensions (prioritizing accuracy), the paper sets a new standard for evaluating legal RAG systems, moving the field away from generic NLP metrics toward domain-specific reliability.
Future Direction: The findings highlight that while retrieval and generation are improving, logical reasoning remains a challenge, suggesting future work should focus on legal logic rule bases and multi-hop inference.

Availability: The code and data are available at https://github.com/legal-dc/Legal-DC.